Belief state representation in the dopamine system

Nature Communications
Benedicte M BabayanSamuel J Gershman

Abstract

Learning to predict future outcomes is critical for driving appropriate behaviors. Reinforcement learning (RL) models have successfully accounted for such learning, relying on reward prediction errors (RPEs) signaled by midbrain dopamine neurons. It has been proposed that when sensory data provide only ambiguous information about which state an animal is in, it can predict reward based on a set of probabilities assigned to hypothetical states (called the belief state). Here we examine how dopamine RPEs and subsequent learning are regulated under state uncertainty. Mice are first trained in a task with two potential states defined by different reward amounts. During testing, intermediate-sized rewards are given in rare trials. Dopamine activity is a non-monotonic function of reward size, consistent with RL models operating on belief states. Furthermore, the magnitude of dopamine responses quantitatively predicts changes in behavior. These results establish the critical role of state inference in RL.

References

Mar 14, 1997·Science·W SchultzP R Montague
Oct 21, 2003·Nature Neuroscience·Naoshige Uchida, Zachary F Mainen
Aug 27, 2004·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Alicia IzquierdoElisabeth A Murray
Feb 22, 2005·Neurobiology of Learning and Memory·Jenna Kim, Michael E Ragozzino
Mar 12, 2005·Science·Philippe N ToblerWolfram Schultz
Jul 6, 2005·Neuron·Hannah M Bayer, Paul W Glimcher
Jun 13, 2006·Neural Computation·Nathaniel D DawDavid S Touretzky
Jun 24, 2006·Trends in Cognitive Sciences·Aaron C CourvilleDavid S Touretzky
Jul 26, 2006·Genesis : the Journal of Genetics and Development·Cristina M BäckmanAndreas C Tomac
Jan 20, 2007·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Michelino PuopoloBruce P Bean
Jun 29, 2007·Hippocampus·John P AggletonJohn M Pearce
Nov 1, 2007·Neural Computation·Mark C Fuhs, David S Touretzky
Jul 29, 2008·Nature Neuroscience·Christopher D FiorilloWolfram Schultz
Nov 4, 2008·Nature Neuroscience·Mitchell F RoitmanRegina M Carelli
Nov 27, 2008·Cognitive, Affective & Behavioral Neuroscience·Peter Dayan, Nathaniel D Daw
Mar 25, 2009·NeuroImage·Klaas Enno StephanKarl J Friston
Jan 13, 2010·Psychological Review·Samuel J GershmanYael Niv
Jun 12, 2010·Journal of Neurophysiology·Ethan S Bromberg-MartinOkihide Hikosaka
Oct 5, 2012·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Jasper AkerboomLoren L Looger
Jul 23, 2013·Nature·Tsai-Wen ChenDouglas S Kim
Sep 11, 2013·NeuroImage·L RigouxJ Daunizeau
Jan 17, 2014·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Andrew S HartPaul E M Phillips
Jan 28, 2014·Neuron·Robert C WilsonYael Niv
Jun 21, 2014·Cell·Lisa A GunaydinKarl Deisseroth
Oct 7, 2014·Current Biology : CB·William R StaufferWolfram Schultz
Sep 1, 2015·Nature·Neir EshelNaoshige Uchida
Feb 9, 2016·Nature Neuroscience·Neir EshelNaoshige Uchida
Apr 26, 2016·Nature Neuroscience·Nathan F ParkerIlana B Witten
Jun 1, 2016·Neuropsychopharmacology : Official Publication of the American College of Neuropsychopharmacology·Anne L CollinsKate M Wassum
Jun 10, 2016·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Thomas A StalnakerGeoffrey Schoenbaum
Mar 7, 2017·Nature Neuroscience·Clara Kwon StarkweatherSamuel J Gershman
Mar 23, 2017·ELife·Sara MatiasZachary F Mainen
Apr 26, 2017·Annual Review of Neuroscience·Mitsuko Watabe-UchidaNaoshige Uchida
Jun 9, 2017·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Adrià Vilà-BallóAntoni Rodríguez-Fornells

❮ Previous
Next ❯

Citations

Dec 27, 2018·The European Journal of Neuroscience·Rüdiger J SeitzHans-Ferdinand Angel
Oct 2, 2019·Nature Reviews. Neuroscience·Samuel J Gershman, Naoshige Uchida
Nov 30, 2019·Frontiers in Psychology·Eric C AndersonPaul K J Han
Sep 27, 2018·The European Journal of Neuroscience·Carole MorelMing-Hu Han
Oct 10, 2018·Proceedings of the National Academy of Sciences of the United States of America·Matthew M NourOliver D Howes
Feb 23, 2019·Cold Spring Harbor Symposia on Quantitative Biology·Mitsuko Watabe-Uchida, Naoshige Uchida
Feb 16, 2019·Nature Communications·Rani MoranRaymond J Dolan
Jan 12, 2021·ELife·Sashank PisupatiAnne K Churchland
Mar 10, 2021·Nature Neuroscience·Kathryn M RothenhoeferWilliam R Stauffer
Dec 26, 2018·Current Biology : CB·Kaining ZhangIlya E Monosov
Oct 23, 2018·Neuropsychologia·Samuel J Gershman, Bastian Greshake Tzovaras
Nov 29, 2020·Cell·HyungGoo R KimNaoshige Uchida
Apr 23, 2021·Current Biology : CB·Rani MoranRaymond J Dolan
Nov 14, 2020·Current Opinion in Neurobiology·Clara Kwon Starkweather, Naoshige Uchida
Nov 17, 2020·Current Opinion in Neurobiology·Talia N LernerJillian L Seiler
May 27, 2021·Journal of Neurophysiology·Dimitrios J PalidisPaul L Gribble
Jun 15, 2021·Trends in Cognitive Sciences·Andrew WestbrookRoshan Cools
Jul 8, 2021·Cerebral Cortex·Samuel D McDougleAnne G E Collins
Jul 28, 2021·Neuropsychopharmacology : Official Publication of the American College of Neuropsychopharmacology·Roshan Cools, Amy F T Arnsten
Aug 29, 2021·Neuropsychopharmacology : Official Publication of the American College of Neuropsychopharmacology·Anne G E Collins, Amitai Shenhav

❮ Previous
Next ❯

Software Mentioned

LabVIEW
MATLAB

Related Concepts

Related Feeds

Amygdala and Midbrain Dopamine

The midbrain dopamine system is widely studied for its involvement in emotional and motivational behavior. Some of these neurons receive information from the amygdala and project throughout the cortex. When the circuit and transmission of dopamine is disrupted symptoms may present. Here is the latest research on the amygdala and midbrain dopamine.

Basal Ganglia

Basal Ganglia are a group of subcortical nuclei in the brain associated with control of voluntary motor movements, procedural and habit learning, emotion, and cognition. Here is the latest research.