Deep Supervised Learning Using Local Errors

Frontiers in Neuroscience
Hesham MostafaGert Cauwenberghs

Abstract

Error backpropagation is a highly effective mechanism for learning high-quality hierarchical features in deep networks. Updating the features or weights in one layer, however, requires waiting for the propagation of error signals from higher layers. Learning using delayed and non-local errors makes it hard to reconcile backpropagation with the learning mechanisms observed in biological neural networks as it requires the neurons to maintain a memory of the input long enough until the higher-layer errors arrive. In this paper, we propose an alternative learning mechanism where errors are generated locally in each layer using fixed, random auxiliary classifiers. Lower layers could thus be trained independently of higher layers and training could either proceed layer by layer, or simultaneously in all layers using local error information. We address biological plausibility concerns such as weight symmetry requirements and show that the proposed learning mechanism based on fixed, broad, and random tuning of each neuron to the classification categories outperforms the biologically-motivated feedback alignment learning technique on the CIFAR10 dataset, approaching the performance of standard backpropagation. Our approach highlights a ...Continue Reading

References

Oct 23, 1997·Neural Computation·S Hochreiter, J Schmidhuber
Jan 18, 2002·Nature·Natasha Sigala, Nikos K Logothetis
Feb 20, 2003·Neural Computation·Xiaohui Xie, H Sebastian Seung
Jun 13, 2006·Neural Computation·Geoffrey E HintonYee-Whye Teh
Jul 29, 2006·Science·G E Hinton, R R Salakhutdinov
May 29, 2007·IEEE Transactions on Neural Networks·S HimavathiA Muthuramalingam
Jan 1, 1996·IEEE Transactions on Neural Networks·G Cauwenberghs
May 29, 2015·Nature·Yann LeCunGeoffrey Hinton
Nov 9, 2016·Nature Communications·Timothy P LillicrapColin J Akerman
Jan 18, 2017·Neural Computation·Arash SamadiDouglas B Tweed
May 20, 2017·Frontiers in Computational Neuroscience·Benjamin Scellier, Yoshua Bengio
Jul 7, 2017·Frontiers in Neuroscience·Emre O NeftciGeorgios Detorakis
Aug 8, 2017·IEEE Transactions on Neural Networks and Learning Systems·Hesham Mostafa

❮ Previous
Next ❯

Citations

Nov 30, 2019·Nature·Kaushik RoyPriyadarshini Panda
Jun 2, 2020·Frontiers in Neuroscience·Jacques KaiserEmre Neftci
Jun 11, 2019·Frontiers in Neuroscience·Brian CraftonArijit Raychowdhury
Feb 11, 2020·Frontiers in Bioengineering and Biotechnology·Yasunari Matsuzaka, Yoshihiro Uesawa
Mar 9, 2021·Frontiers in Neuroscience·Thomas F TiottoNiels A Taatgen
May 15, 2021·Nature Neuroscience·Alexandre PayeurRichard Naud
Jun 5, 2021·Current Opinion in Neurobiology·Spyridon Chavlis, Panayiota Poirazi
Nov 16, 2021·Frontiers in Neuroscience·Gopalakrishnan Srinivasan, Kaushik Roy
Jul 28, 2020··Siddharth JoshiClemens JS Schaefer

❮ Previous
Next ❯

Methods Mentioned

BETA
chip

Software Mentioned

PyTorch

Related Concepts

Trending Feeds

COVID-19

Coronaviruses encompass a large family of viruses that cause the common cold as well as more serious diseases, such as the ongoing outbreak of coronavirus disease 2019 (COVID-19; formally known as 2019-nCoV). Coronaviruses can spread from animals to humans; symptoms include fever, cough, shortness of breath, and breathing difficulties; in more severe cases, infection can lead to death. This feed covers recent research on COVID-19.

Blastomycosis

Blastomycosis fungal infections spread through inhaling Blastomyces dermatitidis spores. Discover the latest research on blastomycosis fungal infections here.

Nuclear Pore Complex in ALS/FTD

Alterations in nucleocytoplasmic transport, controlled by the nuclear pore complex, may be involved in the pathomechanism underlying multiple neurodegenerative diseases including Amyotrophic Lateral Sclerosis and Frontotemporal Dementia. Here is the latest research on the nuclear pore complex in ALS and FTD.

Applications of Molecular Barcoding

The concept of molecular barcoding is that each original DNA or RNA molecule is attached to a unique sequence barcode. Sequence reads having different barcodes represent different original molecules, while sequence reads having the same barcode are results of PCR duplication from one original molecule. Discover the latest research on molecular barcoding here.

Chronic Fatigue Syndrome

Chronic fatigue syndrome is a disease characterized by unexplained disabling fatigue; the pathology of which is incompletely understood. Discover the latest research on chronic fatigue syndrome here.

Evolution of Pluripotency

Pluripotency refers to the ability of a cell to develop into three primary germ cell layers of the embryo. This feed focuses on the mechanisms that underlie the evolution of pluripotency. Here is the latest research.

Position Effect Variegation

Position Effect Variagation occurs when a gene is inactivated due to its positioning near heterochromatic regions within a chromosome. Discover the latest research on Position Effect Variagation here.

STING Receptor Agonists

Stimulator of IFN genes (STING) are a group of transmembrane proteins that are involved in the induction of type I interferon that is important in the innate immune response. The stimulation of STING has been an active area of research in the treatment of cancer and infectious diseases. Here is the latest research on STING receptor agonists.

Microbicide

Microbicides are products that can be applied to vaginal or rectal mucosal surfaces with the goal of preventing, or at least significantly reducing, the transmission of sexually transmitted infections. Here is the latest research on microbicides.

Related Papers

Neural Networks : the Official Journal of the International Neural Network Society
Ashley Prater
Neural Networks : the Official Journal of the International Neural Network Society
Pierre Baldi, Peter Sadowski
© 2022 Meta ULC. All rights reserved