Generic decoding of seen and imagined objects using hierarchical visual features

Nature Communications
Tomoyasu Horikawa, Yukiyasu Kamitani

Abstract

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

References

Jun 16, 1994·Nature·S A EngelM N Shadlen
Apr 29, 1998·Nature·R Epstein, N Kanwisher
Oct 20, 1999·Nature Neuroscience·M Riesenhuber, T Poggio
Jun 11, 2002·Nature Neuroscience·Shimon UllmanErez Sali
Apr 27, 2005·Nature Neuroscience·Yukiyasu Kamitani, Frank Tong
Jan 17, 2007·IEEE Transactions on Pattern Analysis and Machine Intelligence·Thomas SerreTomaso Poggio
Mar 7, 2008·Nature·Kendrick N KayJack L Gallant
May 31, 2008·Science·Tom M MitchellMarcel Adam Just
Nov 5, 2008·Frontiers in Neuroscience·Alexander Borst
Feb 6, 2009·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Mark StokesJohn Duncan
Feb 20, 2009·Nature·Stephenie A Harrison, Frank Tong
Sep 26, 2009·Neuron·Thomas NaselarisJack L Gallant
Jan 1, 2009·Frontiers in Neuroscience·Nikolaus Kriegeskorte
Mar 17, 2010·Current Biology : CB·Martin J ChadwickEleanor A Maguire
Jun 15, 2011·Cerebral Cortex·Radoslaw M CichyJohn-Dylan Haynes
Nov 24, 2012·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Heidi M BonniciEleanor A Maguire
Apr 6, 2013·Science·T HorikawaY Kamitani
Jun 21, 2013·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Yue XingDenis Schluppeck
Jul 23, 2013·Current Biology : CB·Anke Marit AlbersFloris P de Lange
Jul 24, 2013·Trends in Cognitive Sciences·Nikolaus Kriegeskorte, Rogier A Kievit
Nov 26, 2013·Journal of Vision·Daniel D LeedsMichael J Tarr
Apr 20, 2014·PLoS Computational Biology·Hamed NiliNikolaus Kriegeskorte
May 9, 2014·Proceedings of the National Academy of Sciences of the United States of America·Daniel L K YaminsJames J DiCarlo
Jun 27, 2014·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Grace E RiceTimothy J Andrews
Nov 7, 2014·PLoS Computational Biology·Seyed-Mahdi Khaligh-Razavi, Nikolaus Kriegeskorte
Dec 19, 2014·PLoS Computational Biology·Charles F CadieuJames J DiCarlo
Jul 15, 2015·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·Umut Güçlü, Marcel A J van Gerven
Jul 16, 2015·Proceedings of the National Academy of Sciences of the United States of America·Joel Pearson, Stephen M Kosslyn
Sep 29, 2015·Trends in Cognitive Sciences·Joel PearsonStephen M Kosslyn

❮ Previous
Next ❯

Citations

Nov 24, 2017·Scientific Reports·Keisuke SuzukiAnil K Seth
Feb 13, 2019·Scientific Data·Tomoyasu HorikawaYukiyasu Kamitani
Dec 10, 2019·Human Brain Mapping·Xiaoxiao WangBensheng Qiu
Jan 15, 2019·PLoS Computational Biology·Guohua ShenYukiyasu Kamitani
Aug 7, 2019·Nature Reviews. Neuroscience·Joel Pearson
Jan 30, 2020·Scientific Reports·Yoshiaki TsushimaKazuteru Komine
Apr 25, 2020·Nature Communications·Michael B BoneBradley R Buchsbaum
Oct 24, 2017·Cerebral Cortex·Haiguang WenZhongming Liu
May 29, 2018·ELife·Nadine DijkstraMarcel Aj van Gerven
Apr 30, 2019·Frontiers in Computational Neuroscience·Guohua ShenYukiyasu Kamitani
Dec 6, 2018·Nature Communications·Thomas P O'Connell, Marvin M Chun
Mar 24, 2020·Nature Protocols·Lada KohoutováChoong-Wan Woo
Aug 9, 2019·Annual Review of Vision Science·Thomas Serre
Sep 5, 2019·Frontiers in Human Neuroscience·Zhen Li, Hiroaki Shigemasu
Apr 15, 2020·Annals of Biomedical Engineering·Wei HuangHuafu Chen
Aug 18, 2020·Computational Intelligence and Neuroscience·Renzhou GuiHan Nie
Sep 10, 2020·Journal of Neural Engineering·Chong WangHuafu Chen
Oct 28, 2020·Sensors·Zarina RakhimberdinaAnd Tsuyoshi Murata
Oct 6, 2020·Trends in Cognitive Sciences·Gary LupyanAndy Clark
Feb 18, 2021·Proceedings of the National Academy of Sciences of the United States of America·Johannes MehrerTim C Kietzmann
Mar 5, 2021·Cortex; a Journal Devoted to the Study of the Nervous System and Behavior·Dongha Lee, Jorge Almeida
May 4, 2020·Current Biology : CB·Jesse L BreedloveThomas Naselaris
Sep 4, 2021·Neural Networks : the Official Journal of the International Neural Network Society·Wei HuangHuafu Chen
Oct 16, 2021·Nature Methods·Jérôme LecoqChristof Koch

❮ Previous
Next ❯

Software Mentioned

WordNet
GIST
SLR
VLFeat
HMAX
MATLAB

Related Concepts

Related Feeds

Brain Injury & Trauma

brain injury after impact to the head is due to both immediate mechanical effects and delayed responses of neural tissues.

Related Papers

Frontiers in Computational Neuroscience
James M TromansS M Stringer
Frontiers in Systems Neuroscience
Alexander G HuthJack L Gallant
The Journal of Neuroscience : the Official Journal of the Society for Neuroscience
Umut Güçlü, Marcel A J van Gerven
© 2022 Meta ULC. All rights reserved