Conducting Clinical Research Using Crowdsourced Convenience Samples

Annual Review of Clinical Psychology
Jesse Chandler, Danielle Shapiro

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

References

Nov 29, 2002·Experimental Psychology·Ulf-Dietrich Reips
Mar 18, 2011·PloS One·Siddharth Suri, Duncan J Watts
Mar 26, 2011·Behavior Research Methods·Tara S BehrendEric N Wiebe
Jul 1, 2011·Behavior Research Methods·Winter Mason, Siddharth Suri
Dec 21, 2011·Proceedings of the National Academy of Sciences of the United States of America·Winter Mason, Duncan J Watts
Mar 1, 2012·Journal of Personality Assessment·Thomas A FergusHyun-Soo Kim
Jun 8, 2012·Proceedings. Biological Sciences·Nichola J Raihani, Redouan Bshary
Mar 22, 2013·PloS One·Matthew J C CrumpTodd M Gureckis
May 15, 2013·Behavior Research Methods·Travis Simcox, Julie A Fiez
Jul 10, 2013·Behavior Research Methods·Jesse ChandlerGabriele Paolacci
Jul 12, 2013·Journal of General Internal Medicine·Benjamin L RanardRaina M Merchant
Sep 5, 2013·Journal of Clinical Psychology·Sarah Elizabeth Victor, E David Klonsky
Oct 9, 2013·Journal of Affective Disorders·E Samuel WinerEmily J Ginger
Nov 13, 2013·Sexual Abuse : a Journal of Research and Treatment·Sandy K WurteleTasha Moreno
Nov 30, 2013·Journal of Interpersonal Violence·Daniel N Jones, Sally G Olderbak
Dec 18, 2013·Addictive Behaviors·Stephanie J TobinClayton Neighbors
Dec 18, 2013·Psychiatric Services : a Journal of the American Psychiatric Association·Matthew S LebowitzWoo-kyoung Ahn
Dec 21, 2013·Behavior Research Methods·Eyal PeerAlessandro Acquisti
Dec 25, 2013·Assessment·Stephanie L Rojas, Thomas A Widiger
Apr 1, 2014·Behavior Research Methods·Joshua R de Leeuw
Apr 3, 2014·Nicotine & Tobacco Research : Official Journal of the Society for Research on Nicotine and Tobacco·Jesse R CougleShivali Sarawgi
Apr 23, 2014·Journal of Affective Disorders·Anthony PapaJulie Kahler
Apr 23, 2014·Nature Communications·David G RandJoshua D Greene
May 24, 2014·Autism : the International Journal of Research and Practice·Gwen E Mitchell, Kenneth D Locke
Jun 5, 2014·Drug and Alcohol Review·Marcella H Boynton, Laura Smart Richman
Jun 21, 2014·Archives of Sexual Behavior·Sara JahnkeJuergen Hoyer
Jun 25, 2014·Psychiatry Research·Margaret S Andover
Aug 5, 2014·Journal of Personality and Social Psychology·C Nathan DeWallJohn B Nezlek
Sep 13, 2014·Journal of Behavioral Addictions·Paul Rose, Daniel J Segrist
Sep 25, 2014·Annual Review of Psychology·Samuel D Gosling, Winter Mason
Sep 27, 2014·Autism Research : Official Journal of the International Society for Autism Research·Adam E GreenBenjamin E Yerys
Sep 30, 2014·Journal of Behavior Therapy and Experimental Psychiatry·Thomas A Fergus, Wade C Rowatt
Oct 2, 2014·Behavior Research Methods·Robert X D Hawkins
Oct 23, 2014·Journal of Autism and Developmental Disorders·Colin J PalmerJakob Hohwy
Nov 13, 2014·Journal of the Experimental Analysis of Behavior·Patrick S JohnsonMatthew W Johnson
Dec 3, 2014·Comprehensive Psychiatry·Amanda M RainesNorman B Schmidt
Dec 3, 2014·Body Image·Tracy L Tylka, Nichole L Wood-Barcalow
Jan 13, 2015·Journal of Communication Disorders·Tara McAllister ByunDaniel Szeredi

❮ Previous
Next ❯

Citations

Jun 11, 2016·Psycho-oncology·Joanna J Arch, Alaina L Carr
Jun 17, 2016·Archives of Sexual Behavior·Emily Ann HarrisFiona Kate Barlow
Sep 13, 2016·Journal of Personality Disorders·Stephanie L Rojas, Thomas A Widiger
Sep 28, 2016·Archives of Sexual Behavior·Yvonne Anisimowicz, Lucia F O'Sullivan
Oct 13, 2016·Cyberpsychology, Behavior and Social Networking·Kaitlyn Burnell, Tara L Kuther
Feb 24, 2017·Evolutionary Psychology : an International Journal of Evolutionary Approaches to Psychology and Behavior·Peter B Gray, Elizabeth Brogdon
Apr 6, 2016·Psychological Science·Henry L Roediger, K Andrew DeSoto
May 23, 2017·Journal of Attention Disorders·Laura E KnouseArthur D Anastopoulos
Jun 18, 2017·Cyberpsychology, Behavior and Social Networking·Charlotte L BeardVasileios Stavropoulos
Jun 24, 2017·American Journal of Public Health·Veronica YankDavid Rehkopf
Jan 6, 2018·Journal of General Internal Medicine·Karoline Mortensen, Taylor L Hughes
Aug 9, 2017·Archives of Sexual Behavior·Ronald E RobertsonRobert Epstein
Oct 31, 2017·Stress and Health : Journal of the International Society for the Investigation of Stress·Sandeep Mishra, Tyler J S Meadows
Oct 4, 2017·Risk Analysis : an Official Publication of the Society for Risk Analysis·Yasmina OkanWändi Bruine de Bruin
Nov 7, 2017·Suicide & Life-threatening Behavior·Stephanie E HudiburghKiara R Timpano
Oct 27, 2017·Journal of Personality·Michael L CroweJoshua D Miller
May 10, 2018·The Quarterly Journal of Experimental Psychology : QJEP·Yulia OganianChristoph W Korn
Jun 18, 2017·Clinical Psychology & Psychotherapy·Thomas A Fergus, Marcantonio M Spada
Jan 13, 2018·Journal of Clinical Psychology·Graham H NelsonDavid Watson
Jun 7, 2018·Journal of the Experimental Analysis of Behavior·Justin C Strickland, William W Stoops
Oct 19, 2017·Psychological Medicine·N J Raihani, V Bell
Jul 11, 2018·Depression and Anxiety·Lisa Hancock, Richard A Bryant
Jul 20, 2018·Journal of Traumatic Stress·Katherine van Stolk-CookeMatthew Price
Sep 25, 2017·Journal of Racial and Ethnic Health Disparities·Leib LitmanJonathan Robinson
Sep 5, 2018·Journal of Personality Disorders·Scott Barry KaufmanW Keith Campbell
Sep 12, 2018·Personality and Mental Health·Hyoun S KimDaniel S McGrath
Nov 7, 2018·Journal of Personality Assessment·Kate Clauss, Joseph R Bardeen
Jan 25, 2018·Journal of Clinical Child and Adolescent Psychology : the Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53·Jessica L Schleider, John R Weisz
Jan 15, 2018·PM & R : the Journal of Injury, Function, and Rehabilitation·Danielle Shapiro, Jesse Chandler
Jan 24, 2019·The Journal of Nervous and Mental Disease·Yosef Sokol, Mark R Serper
Feb 12, 2019·Journal of Personality·Michael L CroweJoshua D Miller
Dec 12, 2018·Obstetrics and Gynecology·Seri AndersonKristen Hassmiller Lich

❮ Previous
Next ❯

Software Mentioned

Google Scholar
Flash
Mechanical Turk
TurkGate
CrowdFlower
MTurk
Turk
Qualtrics
Mechanical
Java script

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.

Related Papers

Annual Review of Clinical Psychology
Stephanie N Mullins-SweattHilary L DeShong
Annual Review of Biochemistry
Yosef Gruenbaum
Annual Review of Clinical Psychology
Stephen T Russell, Jessica N Fish
© 2022 Meta ULC. All rights reserved