DOI: 10.1101/478644Nov 27, 2018Paper

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

BioRxiv : the Preprint Server for Biology
Miguel AngrickTanja Schultz

Abstract

Objective: Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach: Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results: In a study with six participants, we achieved correlations up to r=0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel f...Continue Reading

Related Concepts

Electrocochleography
Persons
Neuroma
Spatial Distribution
Speech
Body of Uterus
Brain Activity
Neurodegenerative Disorders
Participant
Neural Stem Cells

Related Feeds

BioRxiv & MedRxiv Preprints

BioRxiv and MedRxiv are the preprint servers for biology and health sciences respectively, operated by Cold Spring Harbor Laboratory. Here are the latest preprint articles (which are not peer-reviewed) from BioRxiv and MedRxiv.

Related Papers

Conference Proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society
Christian HerffTanja Schultz
BioRxiv : the Preprint Server for Biology
Gopala K AnumanchipalliEdward F Chang
Conference Proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society
Kristofer E Bouchard, Edward F Chang
© 2021 Meta ULC. All rights reserved