Abstract
Evaluation of computational intelligence (CI) systems designed to improve the performance of a human operator is complicated by the need to include the effect of human variability. In this paper we consider human (reader) variability in the context of medical imaging computer-assisted diagnosis (CAD) systems, and we outline how to compare the detection performance of readers with and without the CAD. An effective and statistically powerful comparison can be accomplished with a receiver operating characteristic (ROC) experiment, summarized by the reader-averaged area under the ROC curve (AUC). The comparison requires sophisticated yet well-developed methods for multi-reader multi-case (MRMC) variance analysis. MRMC variance analysis accounts for random readers, random cases, and correlations in the experiment. In this paper, we extend the methods available for estimating this variability. Specifically, we present a method that can treat arbitrary study designs. Most methods treat only the fully-crossed study design, where every reader reads every case in two experimental conditions. We demonstrate our method with a computer simulation, and we assess the statistical power of a variety of study designs.
References
Sep 1, 1992·Investigative Radiology·D D DorfmanC E Metz
Sep 1, 1986·Investigative Radiology·C E Metz
Jul 1, 1988·Medical Decision Making : an International Journal of the Society for Medical Decision Making·A N Tosteson, C B Begg
Sep 1, 1983·Radiology·J A Hanley, B J McNeil
Apr 1, 1982·Radiology·J A Hanley, B J McNeil
Aug 15, 1996·Statistics in Medicine·X H Zhou, C A Gatsonis
Aug 1, 1995·Academic Radiology·N A Obuchowski
Sep 29, 1998·Academic Radiology·D D DorfmanB A Donaghy
Mar 9, 1999·Journal of the Optical Society of America. A, Optics, Image Science, and Vision·A E Burgess
Jan 5, 2000·Medical Physics·H P ChanN Petrick
May 10, 2000·Academic Radiology·S V BeidenG Campbell
Mar 27, 2001·Journal of the Optical Society of America. A, Optics, Image Science, and Vision·C K Abbey, H H Barrett
Aug 23, 2003·Biostatistics·Todd A Alonzo, Margaret Sullivan Pepe
Mar 18, 2005·Biostatistics·Xiao Song, Xiao-Hua Zhou
Apr 9, 2005·The New England Journal of Medicine·Robert Tibshirani
Sep 15, 2005·Statistics in Medicine·Timothy D Johnson, Valen E Johnson
Feb 21, 2006·Academic Radiology·Brandon D Gallas
Oct 27, 2006·IEEE Transactions on Pattern Analysis and Machine Intelligence·Waleed A YousefMurray H Loew
Oct 31, 2006·Academic Radiology·Eric ClarksonHarrison H Barrett
May 16, 2007·Academic Radiology·Robert F WagnerGregory Campbell
Dec 7, 2007·Journal of the Optical Society of America. A, Optics, Image Science, and Vision·Brandon D GallasKyle J Myers
Citations
Aug 10, 2013·Medical Physics·Nicholas PetrickHeang-Ping Chan
Dec 11, 2013·Medical Physics·Lucretiu M Popescu, Kyle J Myers
Nov 6, 2012·Academic Radiology·Nancy A ObuchowskiStephen L Hillis
Jun 22, 2012·Academic Radiology·Weijie ChenBerkman Sahiner
Feb 7, 2012·Academic Radiology·Brandon D GallasMargarita L Zuley
May 27, 2017·Medical Physics·Dev P ChakrabortyColin G Orton
Mar 6, 2019·Journal of the National Cancer Institute·Alejandro Rodriguez-RuizIoannis Sechopoulos
Jan 11, 2012·Revista Da Sociedade Brasileira De Medicina Tropical·Leonardo Ponce da MottaMarcelo Rosandiski Lyra
Oct 26, 2018·Frontiers in Medicine·Tina M MorrisonEdward Margerrison
Feb 5, 2019·Journal of Medical Imaging·Brandon D GallasKyle J Myers
Apr 18, 2019·European Radiology·Alejandro Rodriguez-RuizRitse M Mann
Sep 2, 2021·Statistical Methods in Medical Research·Erich P Huang, Joanna H Shih