Learning the payoffs and costs of actions.

PLoS Computational Biology
Moritz Möller, Rafal Bogacz

Abstract

A set of sub-cortical nuclei called basal ganglia is critical for learning the values of actions. The basal ganglia include two pathways, which have been associated with approach and avoid behavior respectively and are differentially modulated by dopamine projections from the midbrain. Inspired by the influential opponent actor learning model, we demonstrate that, under certain circumstances, these pathways may represent learned estimates of the positive and negative consequences (payoffs and costs) of individual actions. In the model, the level of dopamine activity encodes the motivational state and controls to what extent payoffs and costs enter the overall evaluation of actions. We show that a set of previously proposed plasticity rules is suitable to extract payoffs and costs from a prediction error signal if they occur at different moments in time. For those plasticity rules, successful learning requires differential effects of positive and negative outcome prediction errors on the two pathways and a weak decay of synaptic weights over trials. We also confirm through simulations that the model reproduces drug-induced changes of willingness to work, as observed in classical experiments with the D2-antagonist haloperidol.

References

Mar 14, 1997·Science·W SchultzP R Montague
Oct 1, 1998·Neurobiology of Learning and Memory·A M Graybiel
Jun 11, 1999·Neuroscience·P RedgraveK Gurney
Mar 29, 2003·Neural Networks : the Official Journal of the International Neural Network Society·K Doya
Nov 6, 2004·Science·Michael J FrankRandall C O'reilly
Apr 10, 2007·Annals of the New York Academy of Sciences·Yael Niv
Apr 11, 2008·Journal of Neurophysiology·Kay ThurleyHans-Rudolf Lüscher
May 29, 2008·Journal of Computational Neuroscience·Dong SongTheodore W Berger
Jun 17, 2008·Journal of Computational Neuroscience·Marco PiriniLorenzo Chiari
Aug 9, 2008·Science·Weixing ShenD James Surmeier
Mar 17, 2009·Science·Kareem A ZaghloulMichael J Kahana
Oct 22, 2010·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Jakob K DreyerJørn D Hounsgaard
Dec 2, 2010·Proceedings of the National Academy of Sciences of the United States of America·Ulrike BastenChristian J Fiebach
Feb 22, 2012·Frontiers in Neuroscience·Mark D HumphriesKevin Gurney
Dec 10, 2013·The European Journal of Neuroscience·Henning SchrollFred H Hamker
May 15, 2015·Cortex; a Journal Devoted to the Study of the Nervous System and Behavior·Trevor T-J ChongMasud Husain
Jul 15, 2015·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Robb B RutledgeRaymond J Dolan
Nov 26, 2015·Nature Neuroscience·Arif A HamidJoshua D Berke
Dec 8, 2015·Nature Neuroscience·Emilie C J SyedMark E Walton
Feb 3, 2016·Proceedings of the National Academy of Sciences of the United States of America·Jackson J ConeMitchell F Roitman
Feb 9, 2016·Nature Neuroscience·Neir EshelNaoshige Uchida
Mar 24, 2016·Proceedings of the National Academy of Sciences of the United States of America·Paul D DodsonPeter J Magill
Jul 28, 2016·Biological Psychiatry·Tiago V Maia, Michael J Frank
Sep 3, 2016·PLoS Computational Biology·John G Mikhael, Rafal Bogacz
Oct 14, 2016·PLoS Computational Biology·Ayaka Kato, Kenji Morita
Aug 26, 2017·Nature Communications·Simon D FisherJohn N J Reynolds
May 16, 2018·Nature Neuroscience·Joshua D Berke

❮ Previous
Next ❯

Citations

May 27, 2020·PLoS Computational Biology·Maaike M H van Swieten, Rafal Bogacz
Jul 8, 2020·ELife·Rafal Bogacz
Dec 21, 2020·Brain and Cognition·Joana CarvalheiroAna Seara-Cardoso
May 9, 2021·Nature Communications·James E M BennettThomas Nowotny
Jun 3, 2021·Brain Sciences·Russell J BoagBirte U Forstmann
Jul 17, 2021·PLoS Computational Biology·Moritz MoellerRafal Bogacz
Nov 11, 2021·Neural Computation·Germain LefebvreRafal Bogacz

❮ Previous
Next ❯

Software Mentioned

OpAL
LOESS

Related Concepts

Related Feeds

Basal Ganglia

Basal Ganglia are a group of subcortical nuclei in the brain associated with control of voluntary motor movements, procedural and habit learning, emotion, and cognition. Here is the latest research.