Markov decision processes: a tool for sequential decision making under uncertainty

Medical Decision Making : an International Journal of the Society for Medical Decision Making
Oguzhan AlagozMark S Roberts

Abstract

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

References

Jan 1, 1983·Medical Decision Making : an International Journal of the Society for Medical Decision Making·J R Beck, S G Pauker
Oct 9, 1996·JAMA : the Journal of the American Medical Association·L B RussellM C Weinstein
Apr 1, 1997·Medical Decision Making : an International Journal of the Society for Medical Decision Making·A S DetskyD A Redelmeier
Aug 9, 1981·Operations Research·C Lefevre
Feb 17, 2000·Artificial Intelligence in Medicine·M Hauskrecht, H Fraser
Jul 9, 2004·Liver Transplantation : Official Publication of the American Association for the Study of Liver Diseases and the International Liver Transplantation Society·Mark S RobertsLisa Weissfeld
Nov 12, 2005·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Oguzhan AlagozMark S Roberts
Jun 16, 2007·Critical Care : the Official Journal of the Critical Care Forum·Görkem SakaGenIMS Investigators
May 12, 2009·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Brian T DentonSteven A Smith
Jan 1, 2008·The International Journal of Robotics Research·Ron AlterovitzKen Goldberg

Citations

Sep 22, 2011·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Jingyu ZhangBrant A Inman
Nov 13, 2012·PloS One·Elizabeth S BurnsideO Alagoz
Aug 6, 2014·Artificial Intelligence in Medicine·Pablo Escandell-MonteroJosé D Martín-Guerrero
Feb 3, 2016·The Journal of Pediatrics·Ruosha LiPediatric Acute Liver Failure Study Group
Jan 5, 2013·Artificial Intelligence in Medicine·Casey C Bennett, Kris Hauser
Mar 26, 2011·Academic Radiology·Carolyn M RutterPari V Pandharipande
Apr 12, 2012·Risk Analysis : an Official Publication of the Society for Risk Analysis·Louis Anthony Cox
Apr 30, 2013·American Journal of Transplantation : Official Journal of the American Society of Transplantation and the American Society of Transplant Surgeons·Eric K H ChowD L Segev
Jan 3, 2013·American Journal of Transplantation : Official Journal of the American Society of Transplantation and the American Society of Transplant Surgeons·K E SchnierN Turgeon
Apr 29, 2014·IEEE Journal of Biomedical and Health Informatics·Mehrnaz Abdollahian, Tapas Das
May 20, 2015·The Pharmacogenomics Journal·O AlagozK Kasirajan
Sep 18, 2013·Value in Health : the Journal of the International Society for Pharmacoeconomics and Outcomes Research·Matteo RuggeriAmerico Cicchetti
Sep 25, 2012·Value in Health : the Journal of the International Society for Pharmacoeconomics and Outcomes Research·Uwe SiebertISPOR-SMDM Modeling Good Research Practices Task Force
Jul 5, 2016·Neuroscience and Biobehavioral Reviews·Karl J FristonGiovanni Pezzulo
Sep 30, 2016·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Qiushi ChenJagpreet Chhatwal
Dec 28, 2016·Liver Transplantation : Official Publication of the American Association for the Study of Liver Diseases and the International Liver Transplantation Society·Kenneth A McLeanEwen M Harrison
Sep 20, 2012·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Uwe SiebertKaren M Kuntz
Mar 14, 2018·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Marta O SoaresMark J Sculpher
Feb 7, 2018·Techniques in Orthopaedics : TIO·Linda Le-WendlingPatrick Tighe
Sep 10, 2017·Health Care Management Science·Lauren E Cipriano, Thomas A Weber
Sep 5, 2018·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Lauren E CiprianoThomas A Weber
Jan 3, 2019·Journal of Pathology and Translational Medicine·Hye Yoon ChangTae-Yeong Kwak
Jan 18, 2020·Journal of Medical Screening·Silvia CorettiAmerico Cicchetti
Sep 9, 2018·Health Care Management Science·Ting-Yu HoZelda B Zabinsky
Sep 23, 2020·Otolaryngology--head and Neck Surgery : Official Journal of American Academy of Otolaryngology-Head and Neck Surgery·David FornerDavid Naimark
Sep 26, 2017·Frontiers in Computational Neuroscience·Cristóbal Moënne-LoccozDiego Cosmelli
Jan 4, 2020·Aesthetic Plastic Surgery·Xuebing LiangKeming Wang
Oct 17, 2016·MDM Policy & Practice·Greggory J SchellRodney A Hayward
Jul 25, 2018·Applied Health Economics and Health Policy·Matteo RuggeriAmerico Cicchetti
Sep 13, 2018·Cancer Medicine·Annemieke WitteveenMaarten J IJzerman
Jul 13, 2017·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Gregg S GonsalvesA David Paltiel
Apr 27, 2018·The European Journal of Health Economics : HEPAC : Health Economics in Prevention and Care·Francesca Romana RolliA Cicchetti
Apr 21, 2017·Medical Decision Making : an International Journal of the Society for Medical Decision Making·Muge CapanJ Sanford Schwartz
Jul 26, 2017·Hepatology : Official Journal of the American Association for the Study of Liver Diseases·Loreta A KondiliPITER Collaborating Group
Jan 6, 2021·Neural Computation·Noor SajidKarl J Friston

Related Concepts

Decision Making, Shared
Markov Chains
Transplantation of Liver
Uncertainty
Decision Making
Literature
Transplantation
Evaluation
Decision
Simulation

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Lipidomics & Rhinovirus Infection

Lipidomics can be used to examine the lipid species involved with pathogenic conditions, such as viral associated inflammation. Discovered the latest research on Lipidomics & Rhinovirus Infection.

Glut1 Deficiency

Glut1 deficiency, an autosomal dominant, genetic metabolic disorder associated with a deficiency of GLUT1, the protein that transports glucose across the blood brain barrier, is characterized by mental and motor developmental delays and infantile seizures. Follow the latest research on Glut1 deficiency with this feed.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Alzheimer's Disease: MS4A

Variants within the membrane-spanning 4-domains subfamily A (MS4A) gene cluster have recently been implicated in Alzheimer's disease in genome-wide association studies. Here is the latest research on Alzheimer's disease and MS4A.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Laryngeal Neoplasms

Laryngeal Neoplasms occur in the Larynx and are typically associated with smoking and alcohol consumption. Discover the latest research on Laryngeal Neoplasms here.

Cell Atlas Along the Gut-Brain Axis

Profiling cells along the gut-brain axis at the single cell level will provide unique information for each cell type, a three-dimensional map of how cell types work together to form tissues, and insights into how changes in the map underlie health and disease of the GI system and its crosstalk with the brain. Disocver the latest research on single cell analysis of the gut-brain axis here.

Related Papers

Medical Decision Making : an International Journal of the Society for Medical Decision Making
J R Beck, S G Pauker
Medical Decision Making : an International Journal of the Society for Medical Decision Making
F A Sonnenberg, J R Beck
Australia and New Zealand Health Policy
John E Goulionis, Athanassios Vozikis
© 2021 Meta ULC. All rights reserved