Markovian architectural bias of recurrent neural networks

IEEE Transactions on Neural Networks
P TinoLubica Benusková

Abstract

In this paper, we elaborate upon the claim that clustering in the recurrent layer of recurrent neural networks (RNNs) reflects meaningful information processing states even prior to training [1], [2]. By concentrating on activation clusters in RNNs, while not throwing away the continuous state space network dynamics, we extract predictive models that we call neural prediction machines (NPMs). When RNNs with sigmoid activation functions are initialized with small weights (a common technique in the RNN community), the clusters of recurrent activations emerging prior to training are indeed meaningful and correspond to Markov prediction contexts. In this case, the extracted NPMs correspond to a class of Markov models, called variable memory length Markov models (VLMMs). In order to appreciate how much information has really been induced during the training, the RNN performance should always be compared with that of VLMMs and NPMs extracted before training as the "null" base models. Our arguments are supported by experiments on a chaotic symbolic sequence and a context-free language with a deep recursive structure. Index Terms-Complex symbolic sequences, information latching problem, iterative function systems, Markov models, recurr...Continue Reading

Citations

Mar 5, 2005·Bioinformatics·Mikael Bodén, John Hawkins
Mar 6, 2009·International Journal of Neural Systems·Yang TangQing-Ying Miao
Mar 29, 2006·Journal of Bioinformatics and Computational Biology·John Hawkins, Mikael Bodén
Aug 16, 2006·Neural Computation·Peter TinoJort van Mourik
Sep 22, 2007·Neural Computation·André Grüning
Oct 29, 2013·Briefings in Bioinformatics·Jonas S Almeida
Nov 6, 2012·Neural Networks : the Official Journal of the International Neural Network Society·He HuangXiaoping Chen
Mar 8, 2011·Neural Networks : the Official Journal of the International Neural Network Society·Claudio Gallicchio, Alessio Micheli
Dec 2, 2008·Neural Networks : the Official Journal of the International Neural Network Society·Zidong WangXiaohui Liu
May 22, 2007·Neural Networks : the Official Journal of the International Neural Network Society·Ganesh K Venayagamoorthy
Jan 13, 2015·IEEE Transactions on Neural Networks and Learning Systems·Lixian ZhangWei Xing Zheng
Oct 19, 2006·IEEE/ACM Transactions on Computational Biology and Bioinformatics·John Hawkins, Mikael Bodén
Feb 7, 2007·IEEE Transactions on Neural Networks·Orlando De Jesús, Martin T Hagan
Jan 29, 2008·IEEE Transactions on Neural Networks·Zhidong Deng, Yi Zhang
Jun 11, 2011·IEEE Transactions on Systems, Man, and Cybernetics. Part B, Cybernetics : a Publication of the IEEE Systems, Man, and Cybernetics Society·Zheng-Guang WuJian Chu
Aug 17, 2011·IEEE Transactions on Neural Networks·Zheng-Guang WuJian Chu
Mar 25, 2005·IEEE Transactions on Neural Networks·Mikael Bodén, John Hawkins
Mar 31, 2015·IEEE Transactions on Neural Networks and Learning Systems·Lixian ZhangWei Xing Zheng
May 12, 2006·Neural Networks : the Official Journal of the International Neural Network Society·Michal CernanskýLubica Benusková
May 28, 2013·Neural Networks : the Official Journal of the International Neural Network Society·He HuangXiaoping Chen
Jul 23, 2013·Neural Networks : the Official Journal of the International Neural Network Society·Yu Zhang
May 11, 2016·Neural Networks : the Official Journal of the International Neural Network Society·Youming XinXia Huang
Jul 19, 2006·Neural Computation·Henrik Jacobsson

❮ Previous
Next ❯

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.