Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Behavior Research Methods
Eyal PeerAlessandro Acquisti

Abstract

Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.

References

Oct 15, 1979·The Biochemical Journal·L M Russell, H Rosenberg
Jun 1, 1984·Journal of Personality Assessment·J T CacioppoC F Kao
Sep 27, 1974·Science·A Tversky, D Kahneman
Oct 12, 2012·Behavior Research Methods·Frederik AustJochen Musch
Jul 10, 2013·Behavior Research Methods·Jesse ChandlerGabriele Paolacci
Jan 1, 2011·Perspectives on Psychological Science : a Journal of the Association for Psychological Science·Michael BuhrmesterSamuel D Gosling

❮ Previous
Next ❯

Citations

Sep 30, 2014·Journal of Behavior Therapy and Experimental Psychiatry·Thomas A Fergus, Wade C Rowatt
Mar 11, 2016·Annals of Behavioral Medicine : a Publication of the Society of Behavioral Medicine·Rebecca A FerrerPaschal Sheeran
Mar 10, 2016·Journal of Health Communication·Elizabeth M GlowackiRobert A Bell
Feb 24, 2015·Aging & Mental Health·Maggie L Syme, Tracy J Cohn
Jan 17, 2016·Annual Review of Clinical Psychology·Jesse Chandler, Danielle Shapiro
Apr 29, 2014·Journal of Pediatric Urology·Jessica C LloydJonathan C Routh
Apr 14, 2016·Behavior Research Methods·Leib LitmanTzvi Abberbock
Jun 10, 2015·Journal of Medical Internet Research·David J McIverJohn S Brownstein
Apr 27, 2016·Behavioral Sleep Medicine·Thomas A Fergus, Michael K Scullin
May 18, 2016·JMIR Mental Health·Scott R BraithwaiteCarl Lee Hanson
Jun 24, 2016·Annals of Behavioral Medicine : a Publication of the Society of Behavioral Medicine·Marissa G HallNoel T Brewer
Oct 16, 2015·BMJ Quality & Safety·Jared B HawkinsFelix Greaves
Nov 23, 2016·IEEE Transactions on Visualization and Computer Graphics·Roger BeechamJo Wood
Feb 17, 2017·Journal of Attention Disorders·David Williamson, Charlotte Johnston
Jan 26, 2017·Journal of Behavioral Medicine·Marissa G HallNoel T Brewer
Jun 2, 2017·Psychological Science·Amelia GoransonKurt Gray
Aug 9, 2017·Archives of Sexual Behavior·Ronald E RobertsonRobert Epstein
Oct 4, 2017·Risk Analysis : an Official Publication of the Society for Risk Analysis·Yasmina OkanWändi Bruine de Bruin
Nov 29, 2017·Evolutionary Psychology : an International Journal of Evolutionary Approaches to Psychology and Behavior·Jinguang Zhang
Jun 18, 2017·Clinical Psychology & Psychotherapy·Thomas A Fergus, Marcantonio M Spada
Jun 7, 2018·Journal of the Experimental Analysis of Behavior·Justin C Strickland, William W Stoops
Jun 28, 2018·Personality & Social Psychology Bulletin·Svetlana KomissaroukE Tory Higgins
Jul 30, 2018·Cognitive Science·Sara GottliebTania Lombrozo
Nov 21, 2014·Cyberpsychology, Behavior and Social Networking·Thomas A Fergus, Sara L Dolan
Aug 10, 2018·PloS One·Marta WalentynowiczArthur A Stone
Aug 30, 2018·Risk Analysis : an Official Publication of the Society for Risk Analysis·Björn MederMichael R Waldmann

❮ Previous
Next ❯

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.