DOI: 10.1101/481267Nov 29, 2018Paper

Intelligible speech synthesis from neural decoding of spoken sentences

BioRxiv : the Preprint Server for Biology
Gopala K AnumanchipalliEdward F Chang


The ability to read out, or decode, mental content from brain activity has significant practical and scientific implications. For example, technology that translates cortical activity into speech would be transformative for people unable to communicate as a result of neurological impairment. Decoding speech from neural activity is challenging because speaking requires extremely precise and dynamic control of multiple vocal tract articulators on the order of milliseconds. Here, we designed a neural decoder that explicitly leverages the continuous kinematic and sound representations encoded in cortical activity to generate fluent and intelligible speech. A recurrent neural network first decoded vocal tract physiological signals from direct cortical recordings, and then transformed them to acoustic speech output. Robust decoding performance was achieved with as little as 25 minutes of training data. Naive listeners were able to accurately identify these decoded sentences. Additionally, speech decoding was not only effective for audibly produced speech, but also when participants silently mimed speech. These results advance the development of speech neuroprosthetic technology to restore spoken communication in patients with disabli...Continue Reading

Related Concepts

Cerebral Cortex
Nervous System Disorder
Structure of Vocalis Muscle
Psyche Structure
Research Activities

Related Feeds

BioRxiv & MedRxiv Preprints

BioRxiv and MedRxiv are the preprint servers for biology and health sciences respectively, operated by Cold Spring Harbor Laboratory. Here are the latest preprint articles (which are not peer-reviewed) from BioRxiv and MedRxiv.

© 2021 Meta ULC. All rights reserved