Training neural networks to encode symbols enables combinatorial generalization

Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences
Ivan I Vankov, Jeffrey S Bowers

Abstract

Combinatorial generalization-the ability to understand and produce novel combinations of already familiar elements-is considered to be a core capacity of the human mind and a major challenge to neural network models. A significant body of research suggests that conventional neural networks cannot solve this problem unless they are endowed with mechanisms specifically engineered for the purpose of representing symbols. In this paper, we introduce a novel way of representing symbolic structures in connectionist terms-the vectors approach to representing symbols (VARS), which allows training standard neural architectures to encode symbolic knowledge explicitly at their output layers. In two simulations, we show that neural networks not only can learn to produce VARS representations, but in doing so they achieve combinatorial generalization in their symbolic and non-symbolic output. This adds to other recent work that has shown improved combinatorial generalization under some training conditions, and raises the question of whether specific mechanisms or training routines are needed to support symbolic processing. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

References

Jul 1, 1992·Psychological Review·John E Hummel, I Biederman
Mar 1, 1988·Cognition·J A Fodor, Z W Pylyshyn
Aug 1, 1984·Journal of the American College of Cardiology·F I Marcus
Oct 23, 1997·Neural Computation·S Hochreiter, J Schmidhuber
Jan 20, 1999·Cognitive Psychology·G F Marcus
May 16, 2003·Psychological Review·John E Hummel, Keith J Holyoak
Mar 18, 2006·The Behavioral and Brain Sciences·Frank van der Velde, Marc de Kamps
Apr 28, 2006·Psychological Review·Matthew M Botvinick, David C Plaut
Jan 18, 2007·Psychological Review·Howard Bowman, Brad Wyble
Jan 24, 2008·Psychological Review·Leonidas A A DoumasCatherine M Sandhofer
Jan 1, 1995·IEEE Transactions on Neural Networks·T A Plate
Oct 21, 2009·Psychological Review·Jeffrey S BowersColin J Davis
Jul 6, 2010·Trends in Cognitive Sciences·James L McClellandLinda B Smith
Jul 28, 2010·Psychological Review·Colin J Davis
Sep 26, 2013·Proceedings of the National Academy of Sciences of the United States of America·Trenton KrieteRandall C O'Reilly
Mar 19, 2014·Attention, Perception & Psychophysics·Garrett Swan, Brad Wyble
Feb 27, 2015·Nature·Volodymyr MnihDemis Hassabis
May 29, 2015·Nature·Yann LeCunGeoffrey Hinton
Jul 15, 2015·Physical Review. E, Statistical, Nonlinear, and Soft Matter Physics·Leon A GatysMatthias Bethge
Jul 21, 2015·International Journal of Multimedia Information Retrieval·Ken ChatfieldAndrew Zisserman
Nov 9, 2015·Neural Networks : the Official Journal of the International Neural Network Society·Boshan Chen, Jiejie Chen
Jan 29, 2016·Nature·David SilverDemis Hassabis
Sep 20, 2017·Current Opinion in Neurobiology·Leon A GatysMatthias Bethge
Nov 5, 2017·Trends in Cognitive Sciences·Jeffrey S Bowers

Citations

Dec 17, 2019·Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences·Andrea E Martin, Giosuè Baggio

Related Concepts

In Silico
Homo sapiens
Phenomenography
Symbolism (Psychology)
Neural Network Simulation
Problem
Research
Synthetic Peptide Combinatorial Library
Biological Neural Networks
Structure

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Synthetic Genetic Array Analysis

Synthetic genetic arrays allow the systematic examination of genetic interactions. Here is the latest research focusing on synthetic genetic arrays and their analyses.

Congenital Hyperinsulinism

Congenital hyperinsulinism is caused by genetic mutations resulting in excess insulin secretion from beta cells of the pancreas. Here is the latest research.

Neural Activity: Imaging

Imaging of neural activity in vivo has developed rapidly recently with the advancement of fluorescence microscopy, including new applications using miniaturized microscopes (miniscopes). This feed follows the progress in this growing field.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Epigenetic Memory

Epigenetic memory refers to the heritable genetic changes that are not explained by the DNA sequence. Find the latest research on epigenetic memory here.

Cell Atlas of the Human Eye

Constructing a cell atlas of the human eye will require transcriptomic and histologic analysis over the lifespan. This understanding will aid in the study of development and disease. Find the latest research pertaining to the Cell Atlas of the Human Eye here.

Femoral Neoplasms

Femoral Neoplasms are bone tumors that arise in the femur. Discover the latest research on femoral neoplasms here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.