Measurement of inter-rater agreement for transient events using Monte Carlo sampled permutations

Statistics in Medicine
Robert G Norman, Marc A Scott

Abstract

In this paper we demonstrate the adverse effect of serially observed data sequences containing transient events on the calculation of Cohen's kappa as an index of inter-rater agreement in the detection of these events. We develop and use a Monte-Carlo-based permutation technique to produce an empiric distribution of kappa in the presence of serial dependence. We find that the empiric confidence intervals for kappa tend to be wider than parametrically derived intervals and in the case of longer event lengths, are markedly so. We evaluate the effect of number and length of events, and further, describe and evaluate three permutation methods which match specific rating situations. Finally, we apply these techniques to the measurement of inter-rater agreement for sleep disordered breathing events, a transient event identified during nocturnal polysomnography, for which traditionally computed confidence intervals for kappa are incorrect.

References

Jan 1, 1990·Journal of Clinical Epidemiology·A R Feinstein, D V Cicchetti
Nov 1, 1984·Journal of Neuroscience Methods·D BliwiseW Dement
Aug 12, 1998·American Journal of Respiratory and Critical Care Medicine·M J DrinnanG J Gibson
Apr 6, 2002·American Journal of Respiratory and Critical Care Medicine·Holger SteltnerJ Christian Virchow
Jul 12, 2002·Statistics in Medicine·Helena Chmura KraemerArt Noda
Aug 3, 2002·American Journal of Respiratory and Critical Care Medicine·Steven J HeitmanW Ward Flemons

❮ Previous
Next ❯

Citations

Oct 29, 2011·Archives of Physical Medicine and Rehabilitation·Janneke AnnegarnKenneth Meijer
Apr 17, 2010·Clinical Neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology·Kirsi PalmuSampsa Vanhatalo
Dec 8, 2014·Clinical Neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology·J J HalfordS M LaRoche

❮ Previous
Next ❯

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.