Recognizing recurrent neural networks (rRNN): Bayesian inference for recurrent neural networks

Biological cybernetics
Sebastian Bitzer, Stefan J Kiebel

Abstract

Recurrent neural networks (RNNs) are widely used in computational neuroscience and machine learning applications. In an RNN, each neuron computes its output as a nonlinear function of its integrated input. While the importance of RNNs, especially as models of brain processing, is undisputed, it is also widely acknowledged that the computations in standard RNN models may be an over-simplification of what real neuronal networks compute. Here, we suggest that the RNN approach may be made computationally more powerful by its fusion with Bayesian inference techniques for nonlinear dynamical systems. In this scheme, we use an RNN as a generative model of dynamic input caused by the environment, e.g. of speech or kinematics. Given this generative RNN model, we derive Bayesian update equations that can decode its output. Critically, these updates define a 'recognizing RNN' (rRNN), in which neurons compute and exchange prediction and prediction error messages. The rRNN has several desirable features that a conventional RNN does not have, e.g. fast decoding of dynamic stimuli and robustness to initial conditions and noise. Furthermore, it implements a predictive coding scheme for dynamic inputs. We suggest that the Bayesian inversion of ...Continue Reading

References

Jan 1, 1990·Biological cybernetics·G Schöner
May 18, 1999·Biological cybernetics·D Mottet, R J Bootsma
Apr 3, 2001·Annual Review of Neuroscience·E K Miller, J D Cohen
May 29, 2002·NeuroImage·K J FristonJ Ashburner
Apr 3, 2003·Neuron·Panayiota PoiraziBartlett W Mel
Sep 2, 2003·NeuroImage·K J FristonW Penny
Mar 10, 2004·Neural Computation·Rajesh P N Rao
Jan 12, 2005·Journal of Motor Behavior·Viktor K Jirsa, J A Scott Kelso
Jan 14, 2005·Proceedings of the National Academy of Sciences of the United States of America·Virginie van WassenhoveDavid Poeppel
Jul 29, 2006·Science·G E Hinton, R R Salakhutdinov
Aug 15, 2006·Annual Review of Psychology·Randolph Blake, Maggie Shiffrar
Sep 6, 2006·EMBO Reports·Kyriaki SidiropoulouPanayiota Poirazi
Nov 25, 2006·Science·Christopher SummerfieldJoy Hirsch
Jan 9, 2007·Neural Computation·Roberto C SoteroJuan C Jimenez
May 23, 2007·Neural Networks : the Official Journal of the International Neural Network Society·D VerstraetenD Stroobandt
May 23, 2007·Neural Networks : the Official Journal of the International Neural Network Society·Herbert JaegerUdo Siewert
May 25, 2007·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Sophie DenèveAlexandre Pouget
Nov 30, 2007·Neural Computation·Sophie Deneve
Feb 6, 2008·IEEE Transactions on Neural Networks·A G ParlosA Atiya
Jan 1, 1994·IEEE Transactions on Neural Networks·J T ConnorL E Atlas
Jan 1, 1994·IEEE Transactions on Neural Networks·J Ting-Ho Lo
Feb 14, 2008·Nature Reviews. Neuroscience·Nelson Spruston
Jan 1, 1990·IEEE Transactions on Neural Networks·K S Narendra, K Parthasarathy
Apr 5, 2008·Neural Computation·Rama NatarajanRichard S Zemel
Apr 25, 2008·NeuroImage·K J FristonJ Daunizeau
Nov 15, 2008·PLoS Computational Biology·Stefan J KiebelKarl J Friston
Jan 16, 2009·Nature Reviews. Neuroscience·Dean V Buonomano, Wolfgang Maass
Apr 11, 2009·Human Brain Mapping·Stefan J KiebelKarl J Friston
Jun 17, 2009·Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences·Karl Friston, Stefan Kiebel
Jun 17, 2009·Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences·Moshe Bar
Aug 15, 2009·PLoS Computational Biology·Stefan J KiebelKarl J Friston
Oct 29, 2009·Physica D. Nonlinear Phenomena·J DaunizeauS J Kiebel
Nov 7, 2009·Frontiers in Computational Neuroscience·Andreea LazarJochen Triesch
Mar 23, 2010·Biological cybernetics·Serafim RodriguesJohn R Terry
Dec 29, 2010·PLoS Computational Biology·Eleftheria Kyriaki PissadakiPanayiota Poirazi
Feb 25, 2011·PloS One·Dionysios PerdikisViktor Jirsa
Mar 8, 2011·PLoS Computational Biology·Martin Boerlin, Sophie Denève
Apr 30, 2011·Physiological Reviews·Dominique DebanneGisèle Alcaraz
Oct 8, 2011·PLoS Computational Biology·Dionysios PerdikisViktor K Jirsa

❮ Previous
Next ❯

Citations

Aug 13, 2015·PLoS Computational Biology·Sebastian BitzerStefan J Kiebel
Apr 11, 2018·Nature Communications·Ezequiel M ArneodoDmitry Rinberg
Jun 14, 2018·BMC Bioinformatics·Shashank GuptaVasudeva Varma
Jun 8, 2021·Frontiers in Artificial Intelligence·Sascha FrölichStefan J Kiebel

❮ Previous
Next ❯

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.

Related Papers

Neural Networks : the Official Journal of the International Neural Network Society
Su Lee Goh, Danilo P Mandic
PLoS Computational Biology
Karl J Friston
Neural Networks : the Official Journal of the International Neural Network Society
M ZavagliaW D Penny
© 2021 Meta ULC. All rights reserved