Real-time decoding of question-and-answer speech dialogue using human cortical activity

Nature Communications
David A MosesEdward F Chang

Abstract

Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance's identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.

References

Mar 1, 1997·Cortex; a Journal Devoted to the Study of the Nervous System and Behavior·D BoatmanB Gordon
Jun 10, 2000·Cerebral Cortex·J R BinderE T Possing
Sep 28, 2005·Progress in Brain Research·Steven LaureysSerge Goldman
Oct 30, 2008·Frontiers in Human Neuroscience·Aurélie Bidet-CauletOlivier Bertrand
Nov 5, 2008·Frontiers in Neuroscience·Ryan T CanoltyRobert T Knight
Apr 8, 2011·Journal of Neural Engineering·Eric C LeuthardtGerwin Schalk
Feb 4, 2012·PLoS Biology·Brian N PasleyEdward F Chang
Jul 21, 2012·Journal of Neural Engineering·Connie Cheung, Edward F Chang
Feb 22, 2013·Nature·Kristofer E BouchardEdward F Chang
Feb 1, 2014·Science·Nima MesgaraniEdward F Chang
May 20, 2014·Journal of Neural Engineering·Emily M MuglerMarc W Slutzky
Oct 10, 2014·Science Translational Medicine·Eric W SellersChristopher K Hauser
Mar 12, 2015·Frontiers in Human Neuroscience·Fabien LotteGerwin Schalk
Jul 1, 2015·Frontiers in Neuroscience·Christian HerffTanja Schultz
Jul 19, 2015·Orphanet Journal of Rare Diseases·Marie-Christine RousseauPascal Auquier
Dec 14, 2016·The New England Journal of Medicine·Mariska J VansteenselNick F Ramsey
Dec 21, 2016·Nature Communications·Matthew K LeonardEdward F Chang
Apr 12, 2017·NeuroImage·Vikram R RaoEdward F Chang
Jan 31, 2018·Journal of Neural Engineering·David A MosesEdward F Chang
Feb 14, 2018·The Journal of Neuroscience : the Official Journal of the Society for Neuroscience·David F ConantEdward F Chang
Jun 30, 2018·Cell·Benjamin K DichterEdward F Chang

❮ Previous
Next ❯

Citations

Dec 28, 2019·JAMA : the Journal of the American Medical Association·Edward F Chang, Gopala K Anumanchipalli
Nov 26, 2019·Journal of Neural Engineering·Stephanie Naufel, Eran Klein
Feb 6, 2020·Journal of Neural Engineering·Sergey D StaviskyJaimie M Henderson
Jun 2, 2020·Disability and Rehabilitation. Assistive Technology·Melanie Fried-OkenBruce Wojciechowski
Aug 12, 2020·BioTechniques·Jenny Straiton
Apr 23, 2020·Frontiers in Neuroscience·Debadatta DashJun Wang
Apr 2, 2020·Nature Neuroscience·Joseph G MakinEdward F Chang
Jul 28, 2020·AJOB Neuroscience·Walter J KoroshetzChristine Grady
Sep 9, 2020·Nature Biotechnology·Daniel B SilversmithKarunesh Ganguly
Mar 17, 2020·Frontiers in Neuroscience·Christian HerffPieter Kubben
Apr 2, 2020·Nature Neuroscience·Gregory B Cogan
Nov 26, 2020·Journal of Neural Engineering·Guy H WilsonKrishna V Shenoy
Dec 30, 2020·Briefings in Bioinformatics·Jesse A Livezey, Joshua I Glaser
Oct 27, 2020·The Linacre Quarterly·Christopher M Reilly
Feb 23, 2021·Journal of Neuroscience Methods·Zheng Zhang, Timothy G Constandinou
Apr 9, 2021·Neuron·Zhe Sage Chen, Bijan Pesaran
Jan 12, 2021·Cerebral Cortex·Justin D Yao, Dan H Sanes
May 15, 2021·The Journal of Neuropsychiatry and Clinical Neurosciences·Nicholas L BormannAaron D Boes
Jun 13, 2021·Trends in Cognitive Sciences·Xiaorong GaoShangkai Gao
Jun 17, 2021·International Journal of Neural Systems·Emanuela DelfinoLuciano Fadiga
Jul 15, 2021·The New England Journal of Medicine·David A MosesEdward F Chang
Aug 12, 2021·BMC Biology·Benjamin WittevrongelMatthew J Brookes
Aug 20, 2021·Nature Reviews. Neuroscience·Jennifer A ChandlerJudy Illes
Sep 4, 2021·Neural Networks : the Official Journal of the International Neural Network Society·Wei HuangHuafu Chen
May 30, 2021·Advanced Healthcare Materials·Mengjia ZhuYingying Zhang
Oct 1, 2021·International Journal of Neural Systems·Ujwal ChaudharyNiels Birbaumer
Nov 11, 2021·Proceedings of the National Academy of Sciences of the United States of America·Alexander A AabediShawn L Hervey-Jumper
Oct 28, 2021·Journal of Neural Engineering·Chun-Hsiang ChuangTzyy-Ping Jung
Dec 11, 2021·Frontiers in Physiology·Rhaíra Helena Caetano E Souza, Eduardo Lázaro Martins Naves

❮ Previous
Next ❯

Methods Mentioned

BETA
PCA

Software Mentioned

Linux
p2fa
rtNSR
vSMC
hyperopt Python package
Neural Speech Recognition ( rtNSR )
real
Synthesizer
time
Python

Related Concepts

Related Feeds

Brain-Computer Interface

A brain-computer interface, also known as a brain-machine interface, is a bi-directional communication pathway between an external device and a wired brain. Here is the latest research on this topic.

Related Papers

Journal of Neural Engineering
Abdulwahab AlasfourVikash Gilja
Conference Proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society
Emily M MuglerMarc W Slutzky
BioRxiv : the Preprint Server for Biology
Gopala K AnumanchipalliEdward F Chang
© 2021 Meta ULC. All rights reserved