Test set bias affects reproducibility of gene signatures

Bioinformatics
Prasad PatilJeffrey T Leek

Abstract

Prior to applying genomic predictors to clinical samples, the genomic data must be properly normalized to ensure that the test set data are comparable to the data upon which the predictor was trained. The most effective normalization methods depend on data from multiple patients. From a biomedical perspective, this implies that predictions for a single patient may change depending on which other patient samples they are normalized with. This test set bias will occur when any cross-sample normalization is used before clinical prediction. We demonstrate that results from existing gene signatures which rely on normalizing test data may be irreproducible when the patient population changes composition or size using a set of curated, publicly available breast cancer microarray experiments. As an alternative, we examine the use of gene signatures that rely on ranks from the data and show why signatures using rank-based features can avoid test set bias while maintaining highly accurate classification, even across platforms. The code, data and instructions necessary to reproduce our entire analysis is available at https://github.com/prpatil/testsetbias.

References

Feb 2, 2002·Nature·Laura J van 't VeerStephen H Friend
Feb 28, 2002·Lancet·Emanuel F PetricoinLance A Liotta
May 16, 2002·Proceedings of the National Academy of Sciences of the United States of America·Robert TibshiraniGilbert Chu
Feb 17, 2005·Journal of the National Cancer Institute·Keith A BaggerlyKevin R Coombes
Dec 22, 2005·Nature Reviews. Genetics·David B AllisonMahyar Sabripour
Nov 1, 2006·BMC Genomics·Annuska M GlasLaura J Van't Veer
Nov 15, 2007·Journal of the National Cancer Institute·Lara LusaMarco A Pierotti
Feb 11, 2009·Journal of Clinical Oncology : Official Journal of the American Society of Clinical Oncology·Joel S ParkerPhilip S Bernard
Jan 26, 2010·Biostatistics·Matthew N McCallRafael A Irizarry
Jul 3, 2010·Science·Paola SebastianiThomas T Perls
Mar 10, 2011·Nature Medicine·Ian J Majewski, René Bernards
Jan 21, 2012·Journal of the National Cancer Institute·Benjamin Haibe-KainsChristos Sotiriou
Nov 30, 2012·Nucleic Acids Research·Tanya BarrettAlexandra Soboleva

❮ Previous
Next ❯

Citations

Jun 1, 2016·Drug Discovery Today·Wilson Wen Bin Goh, Limsoon Wong
Nov 4, 2016·Journal of Bioinformatics and Computational Biology·Wilson Wen Bin Goh, Limsoon Wong
May 10, 2017·Liver International : Official Journal of the International Association for the Study of the Liver·Lu AoXianlong Wang
Jun 13, 2018·Journal of Cellular Biochemistry·Yuanli HuangYuanrong Yang
Apr 24, 2018·Liver International : Official Journal of the International Association for the Study of the Liver·Lu AoZheng Guo
Mar 3, 2020·Briefings in Bioinformatics·Hailong ZhengWenyuan Zhao
Jul 1, 2020·Experimental Biology and Medicine·HongLei WangHongTao Wang
Aug 17, 2018·Experimental and Therapeutic Medicine·Wei LiuHuaqin He
Jul 12, 2017·Frontiers in Oncology·Michaela BayerlováAnnalen Bleckmann
Oct 6, 2020·Journal of the American Statistical Association·Naim U RashidJoseph G Ibrahim
Mar 14, 2018·Proceedings of the National Academy of Sciences of the United States of America·Prasad Patil, Giovanni Parmigiani
May 19, 2020·BioMed Research International·Jiayong ZhongHuiying Liang
Aug 4, 2020·International Journal of Cancer. Journal International Du Cancer·Helena LiljedahlJohan Staaf
Jul 14, 2018·BMC Bioinformatics·Yuqing ZhangW Evan Johnson
Mar 28, 2017·Clinical Cancer Research : an Official Journal of the American Association for Cancer Research·Sean P PitrodaPhilip P Connell
Nov 23, 2019·Clinical Cancer Research : an Official Journal of the American Association for Cancer Research·Naim U RashidJen Jen Yeh
Dec 29, 2018·Cancer Epidemiology, Biomarkers & Prevention : a Publication of the American Association for Cancer Research, Cosponsored by the American Society of Preventive Oncology·Kevin H KenslerYujing J Heng
Aug 8, 2018·Clinical Cancer Research : an Official Journal of the American Association for Cancer Research·Gregory M ChenLevi Waldron
May 5, 2021·Proceedings of the National Academy of Sciences of the United States of America·Rayees RahmanAvner Schlessinger

❮ Previous
Next ❯

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.