Abstract
Evaluation of expert assessment of exposure depends, in the absence of a validation measurement, upon measures of agreement among the expert raters. Agreement is typically measured using Cohen's Kappa statistic, however, there are some well-known limitations to this approach. We demonstrate an alternate method that uses log-linear models designed to model agreement. These models contain parameters that distinguish between exact agreement (diagonals of agreement matrix) and non-exact associations (off-diagonals). In addition, they can incorporate covariates to examine whether agreement differs across strata. We applied these models to evaluate agreement among expert ratings of exposure to sensitizers (none, likely, high) in a study of occupational asthma. Traditional analyses using weighted kappa suggested potential differences in agreement by blue/white collar jobs and office/non-office jobs, but not case/control status. However, the evaluation of the covariates and their interaction terms in log-linear models found no differences in agreement with these covariates and provided evidence that the differences observed using kappa were the result of marginal differences in the distribution of ratings rather than differences in agr...Continue Reading
References
Jan 1, 1990·Journal of Clinical Epidemiology·D V Cicchetti, A R Feinstein
Sep 1, 1990·Statistics in Medicine·K L PosnerF W Cheney
Jan 1, 1990·Journal of Clinical Epidemiology·A R Feinstein, D V Cicchetti
Dec 1, 1989·American Journal of Epidemiology·J SiemiatyckiL Richardson
Aug 1, 1987·American Journal of Epidemiology·M Maclure, W C Willett
Oct 1, 1986·British Journal of Industrial Medicine·M S GoldbergM Gérin
Sep 1, 1994·American Journal of Industrial Medicine·W F Stewart, P A Stewart
Sep 1, 1994·American Journal of Industrial Medicine·P A Stewart, W F Stewart
Sep 1, 1996·American Journal of Epidemiology·L FritschiL Richardson
Mar 1, 1997·American Journal of Industrial Medicine·J SiemiatyckiM Gérin
Jun 1, 1997·International Journal of Epidemiology·G BenkeM Salzberg
May 22, 1998·Statistics in Medicine·I Guggenmoos-Holzmann, R Vonk
Aug 17, 1999·Occupational and Environmental Medicine·E TielemansK Hartog
Nov 25, 2000·Teratology·J B CarlinL B Holmes
Feb 24, 2001·Statistical Methods in Medical Research·J C Nelson, M S Pepe
Apr 21, 2001·Biometrics·H Zhang, D Zelterman
Dec 11, 2002·The British Journal of Mathematical and Statistical Psychology·Christof Schuster
Apr 22, 2003·American Journal of Industrial Medicine·Lin FritschiJack Siemiatycki
Sep 4, 2003·Environmental Health : a Global Access Science Source·Susan R SamaDonald K Milton
Sep 23, 2003·Epidemiology·Andrea 't MannetjePaolo Boffetta
Oct 18, 2006·Birth Defects Research. Part A, Clinical and Molecular Teratology·Adolfo CorreaLeila W Jackson
Apr 23, 2011·The Annals of Occupational Hygiene·Melissa C FriesenPatricia A Stewart
Nov 16, 2012·PloS One·Svetlana SolovievaEira Viikari-Juntura
Nov 28, 2012·The Annals of Occupational Hygiene·Melissa C FriesenHormuzd A Katki
Sep 30, 2014·Epidemiology·Maria VutcoviciSamy Suissa
Jun 23, 2015·Epidemiology·Sohaib A Virk, Guy D Eslick