Decoding Speech and Language Representations from the Brain

Gopala Anumanchipalli will be talking about his work to measure and reconstruct speech signals from human neural activity. Gopala is uniquely situated to do this work, having graduated from Prof. Alan Black’s speech synthesis lab at CMU and he is now part of Prof. Eddie Chang’s excellent ECoG lab at UCSF. At UCSF there are patients who are undergoing surgery for epilepsy, and some of the patients allow recordings to be made as they listen and respond to sounds. One important question is how is speech represented in the brain. Gopala’s work uses deep neural networks (DNNs) to reconstruct speech sounds from human ECoG data. Speech from spikes.

Abstract
Spoken communication is basic to who we are. Neurological conditions that result in loss of speech can be devastating for affected patients. This talk will summarize recent efforts in decoding neural activity directly from the surface of the speech cortex during fluent speech production, monitored using intracranial Electrocorticography (ECoG). Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. I will first describe the articulatory encoding characteristics in the speech motor cortex and compare them against other representations like the phonemes. I will then describe deep learning approaches to convert neural activity into these articulatory physiological signals that can then be transformed into audible speech acoustics or decoded to text. We show that such biomimetic strategies make optimal use of available data; generalize well across subjects, and also perform silent speech decoding. These results set a new benchmark in the development of Brain-Computer Interfaces for assistive communication in paralyzed individuals with intact cortical function.

Bio:
Gopala Anumanchipalli, PhD is an associate researcher at the Department of Neurological Surgery at University of California, San Francisco. His research is in understanding neural mechanisms of human speech production towards developing next generation Brain-Computer Interfaces. Gopala was a postdoctoral fellow at UCSF working with Edward F Chang, MD and has previously received PhD in Language and Information Technologies from Carnegie Mellon University working with Prof. Alan Black on speech synthesis.