Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Behavior Research Methods
Eyal PeerAlessandro Acquisti

Abstract

Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.

References

Oct 15, 1979·The Biochemical Journal·L M Russell, H Rosenberg
Jun 1, 1984·Journal of Personality Assessment·John T CacioppoC F Kao
Sep 27, 1974·Science·A Tversky, D Kahneman
Oct 12, 2012·Behavior Research Methods·Frederik AustJochen Musch
Jul 10, 2013·Behavior Research Methods·Jesse ChandlerGabriele Paolacci
Jan 1, 2011·Perspectives on Psychological Science : a Journal of the Association for Psychological Science·Michael D BuhrmesterSamuel D Gosling

Citations

Sep 30, 2014·Journal of Behavior Therapy and Experimental Psychiatry·Thomas A Fergus, Wade C Rowatt
Mar 11, 2016·Annals of Behavioral Medicine : a Publication of the Society of Behavioral Medicine·Rebecca A FerrerPaschal Sheeran
Mar 10, 2016·Journal of Health Communication·Elizabeth M GlowackiRobert A Bell
Feb 24, 2015·Aging & Mental Health·Maggie L Syme, Tracy J Cohn
Jan 17, 2016·Annual Review of Clinical Psychology·Jesse Chandler, Danielle Shapiro
Apr 29, 2014·Journal of Pediatric Urology·Jessica C LloydJonathan C Routh
Apr 14, 2016·Behavior Research Methods·Leib LitmanTzvi Abberbock
Jun 10, 2015·Journal of Medical Internet Research·David J McIverJohn S Brownstein
Apr 27, 2016·Behavioral Sleep Medicine·Thomas A Fergus, Michael K Scullin
May 18, 2016·JMIR Mental Health·Scott R BraithwaiteCarl Lee Hanson
Jun 24, 2016·Annals of Behavioral Medicine : a Publication of the Society of Behavioral Medicine·Marissa G HallNoel T Brewer
Oct 16, 2015·BMJ Quality & Safety·Jared B HawkinsFelix Greaves
Nov 23, 2016·IEEE Transactions on Visualization and Computer Graphics·Roger BeechamJo Wood
Feb 17, 2017·Journal of Attention Disorders·David Williamson, Charlotte Johnston
Jan 26, 2017·Journal of Behavioral Medicine·Marissa G HallNoel T Brewer
Jun 2, 2017·Psychological Science·Amelia GoransonKurt Gray
Aug 9, 2017·Archives of Sexual Behavior·Ronald E RobertsonRobert Epstein
Oct 4, 2017·Risk Analysis : an Official Publication of the Society for Risk Analysis·Yasmina OkanWändi Bruine de Bruin
Nov 29, 2017·Evolutionary Psychology : an International Journal of Evolutionary Approaches to Psychology and Behavior·Jinguang Zhang
Jun 18, 2017·Clinical Psychology & Psychotherapy·Thomas A Fergus, Marcantonio M Spada
Jun 7, 2018·Journal of the Experimental Analysis of Behavior·Justin C Strickland, William W Stoops
Jun 28, 2018·Personality & Social Psychology Bulletin·Svetlana KomissaroukE Tory Higgins
Jul 30, 2018·Cognitive Science·Sara GottliebTania Lombrozo
Nov 21, 2014·Cyberpsychology, Behavior and Social Networking·Thomas A Fergus, Sara L Dolan
Aug 10, 2018·PloS One·Marta WalentynowiczArthur A Stone
Aug 30, 2018·Risk Analysis : an Official Publication of the Society for Risk Analysis·Björn MederMichael R Waldmann
Oct 29, 2018·Journal of Clinical Psychology·Erin E O'Connor, David A Langer
May 5, 2016·American Journal of Speech-language Pathology·Kaitlin L LansfordLukas Bystricky
Mar 21, 2019·PloS One·Chad M TopazKevin M Murphy
Apr 23, 2019·Journal of Traumatic Stress·Rachel Wamser-Nanney
May 10, 2019·AJOB Neuroscience·Erin C ConradAnjan Chatterjee
Feb 2, 2018·Health Communication·Joseph McGlynn, Matthew S McGlone
Sep 14, 2019·Behavior Research Methods·Thomas P CarpenterAlek Chakroff
Jan 8, 2020·Risk Analysis : an Official Publication of the Society for Risk Analysis·Yasmina OkanAndrew M Parker
Aug 7, 2019·Journal of Addiction Medicine·Alexis S HammondEric C Strain
Mar 11, 2020·Psychiatric Services : a Journal of the American Psychiatric Association·Jack TsaiSamuel Wilkinson
Apr 18, 2020·PloS One·Christopher T RobertsonKeith Joiner
May 28, 2020·Journal of Health Psychology·Scout M Kelly, Hayley R Walton
Aug 17, 2019·Assessment·Gabriela Kattan KhazanovCourtney N Forbes
Oct 24, 2019·Social Psychiatry and Psychiatric Epidemiology·Thais M ZimbresLaramie D Taylor
Sep 2, 2020·Journal of Clinical Psychology·Brooke A AmmermanMichael S McCloskey
Sep 13, 2019·Behavior Research Methods·Jesse ChandlerLeib Litman
Sep 15, 2020·Public Health Nutrition·Claire L Jansson-KnodellAndrea Shin
Oct 3, 2020·International Journal of Environmental Research and Public Health·Kristen Cooksey StowersKim Gans
Oct 8, 2016·The American Journal of Drug and Alcohol Abuse·Sebastian BauhoffDeborah Scharf
Jul 7, 2017·The Journal of the Acoustical Society of America·Stephanie A BorrieTessa Bent
Jul 14, 2017·Frontiers in Psychology·Otto WarisMatti Laine
Sep 12, 2017·Journal of Cognition and Development : Official Journal of the Cognitive Development Society·Steven Othello Roberts, Susan Gelman
Nov 3, 2017·Psychological Science·Sara Gottlieb, Tania Lombrozo
May 29, 2018·Personality & Social Psychology Bulletin·Alexa WeissThomas Mussweiler
Jul 7, 2018·Frontiers in Psychology·David J HauserRichard Gonzalez
Feb 23, 2019·Journal of Psychopharmacology·Justin C StricklandWilliam W Stoops
Jan 28, 2019·Prevention Science : the Official Journal of the Society for Prevention Research·Andrew Lac, Jeremy W Luk
Apr 13, 2019·International Journal of Environmental Research and Public Health·Lixin JiangXiaowen Hu
Jan 1, 2018·The Behavioral and Brain Sciences·Gabriele Paolacci, Jesse Chandler
Jul 21, 2019·Archives of Sexual Behavior·Nicole H WeissKatherine L Dixon-Gordon
Aug 15, 2019·Evolutionary Psychology : an International Journal of Evolutionary Approaches to Psychology and Behavior·Neil R Caton, Sean M Horan
Jun 18, 2020·Frontiers in Psychology·Effie J PereiraJelena Ristic
Feb 27, 2018·Obesity Science & Practice·M Oldham, E Robinson
Feb 17, 2018·Experimental Economics·Antonio A ArecharLucas Molleman
Jun 27, 2018·Behavior Research Methods·Saoirse Connor Desai, Stian Reimers
Aug 30, 2019·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Barry DewittJanel Hanmer
Jul 7, 2020·Archives of Sexual Behavior·Nathan D LeonhardtChanae Valdez
May 18, 2020·Journal of Community Health·Kara MathewsonRobert A Bednarczyk
Aug 23, 2020·International Journal of Environmental Research and Public Health·Eboneé N ButlerNoel T Brewer
Jul 12, 2017·Attention, Perception & Psychophysics·Kevin J P WoodsJosh H McDermott
Apr 18, 2017·Journal of Personality Assessment·Rosemond T LoronaThomas A Fergus
Oct 6, 2017·Frontiers in Psychology·Gunnar Brodersen, Gabriele Oettingen
Feb 15, 2020·Journal of the Experimental Analysis of Behavior·Justin C StricklandB Levi Bolin
Nov 26, 2019·Journal of Elder Abuse & Neglect·Maggie L Syme, Tracy J Cohn
Aug 15, 2017·Family Relations·Jodi DworkinJessie H Rudi
Mar 11, 2017·Science and Engineering Ethics·Justin T Pickett, Sean Patrick Roche
Oct 13, 2017·Scientific Reports·Andrew G ReeceEllen J Langer
Feb 26, 2019·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Miroslav Sirota, Marie Juanchich
Dec 4, 2019·Journal of Genetic Counseling·Camille F FisherElizabeth M Petty
Jun 22, 2018·Perspectives on Psychological Science : a Journal of the Association for Psychological Science·Michael D BuhrmesterSamuel D Gosling
Jun 8, 2018·Plastic and Reconstructive Surgery. Global Open·Alexandra BucknorHeather Furnas
Mar 23, 2018·Proceedings of the National Academy of Sciences of the United States of America·Kevin J P Woods, Josh H McDermott
Apr 18, 2018·Health Psychology Open·Elizabeth K SengLiana Fraenkel
Aug 7, 2020·Journal of Trauma & Dissociation : the Official Journal of the International Society for the Study of Dissociation (ISSD)·Rachel Wamser-Nanney, Julia C Sager
May 3, 2019·International Journal of Offender Therapy and Comparative Criminology·Miranda M NadeauRobert Tyminski
Nov 18, 2018·International Journal of Environmental Research and Public Health·Lindsey M LavaysseDavid F Arena
Jan 1, 2015·Journal of Cognitive Psychotherapy·Thomas A FergusHolly K Orcutt
Jul 22, 2020·CNS Spectrums·Campbell InceLeonardo F Fontenelle
Oct 11, 2020·Cognitive Science·Zachary C IrvingChandra Sripada
Oct 20, 2020·Psychiatry, Psychology, and Law : an Interdisciplinary Journal of the Australian and New Zealand Association of Psychiatry, Psychology and Law·Angela M JonesSteven Penrod
Nov 18, 2020·Health Communication·Haijing Ma, Claude H Miller
Jun 14, 2019·Addictive Behaviors Reports·John A CunninghamDavid C Hodgins
Jan 27, 2021·The Journal of Social Psychology·Shih Yung ChouBo Han
Oct 30, 2020·BMC Urology·Laragh L W Harvey-KellyJuliet A Usher-Smith
Nov 17, 2020·Telemedicine Journal and E-health : the Official Journal of the American Telemedicine Association·Yi-Chin Kato-Lin, Shawn T Thelen
Nov 17, 2020·AJOB Neuroscience·Claire T DinhAnjan Chatterjee
Dec 10, 2020·International Journal of Environmental Research and Public Health·Paulina Krzywicka, Katarzyna Byrka

Related Concepts

Repeated Rounds of Survey
Selection of Subjects
Twitter Messaging
Behavioral Research
Attention
Duane Retraction Syndrome
Turks
Sampling - Surgical Action
Double Check
Participant

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Neural Activity: Imaging

Imaging of neural activity in vivo has developed rapidly recently with the advancement of fluorescence microscopy, including new applications using miniaturized microscopes (miniscopes). This feed follows the progress in this growing field.

The Tendon Seed Network

Tendons are rich in the extracellular matrix and are abundant throughout the body providing essential roles including structure and mobility. The transcriptome of tendons is being compiled to understand the micro-anatomical functioning of tendons. Discover the latest research pertaining to the Tendon Seed Network here.

Myocardial Stunning

Myocardial stunning is a mechanical dysfunction that persists after reperfusion of previously ischemic tissue in the absence of irreversible damage including myocardial necrosis. Here is the latest research.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Incretins

Incretins are metabolic hormones that stimulate a decrease in glucose levels in the blood and they have been implicated in glycemic regulation in the remission phase of type 1 diabetes. Here is the latest research.

Chromatin Regulation and Circadian Clocks

The circadian clock plays an important role in regulating transcriptional dynamics through changes in chromatin folding and remodelling. Discover the latest research on Chromatin Regulation and Circadian Clocks here.

Long COVID-19

“Long Covid-19” describes illness in patients who are reporting long-lasting effects of the SARS-CoV-19 infection, often long after they have recovered from acute Covid-19. Ongoing health issues often reported include low exercise tolerance and breathing difficulties, chronic tiredness, and mental health problems such as post-traumatic stress disorder and depression. This feed follows the latest research into Long Covid.

Spatio-Temporal Regulation of DNA Repair

DNA repair is a complex process regulated by several different classes of enzymes, including ligases, endonucleases, and polymerases. This feed focuses on the spatial and temporal regulation that accompanies DNA damage signaling and repair enzymes and processes.