Auditory Separation of a Conversation from Background via Attentional Gating

Where: CCRMA Seminar Room
Why: Speech is a seriously interesting signal and we want to make it clearer.
We present a model for separating a set of voices out of a sound mixture containing an unknown number of sources. Our Attentional Gating Network (AGN) uses a variable attentional context to specify which speakers in the mixture are of interest. The attentional context is specified by an embedding vector which modifies the processing of a neural network through an additive bias. Individual speaker embeddings are learned to separate a single speaker while superpositions of the individual speaker embeddings are used to separate sets of speakers. We first evaluate AGN on a traditional single speaker separation task and show an improvement of 9% with respect to comparable models. Then, we introduce a new task to separate an arbitrary subset of voices from a mixture of an unknown-sized set of voices, inspired by the human ability to separate a conversation of interest from background chatter at a cafeteria. We show that AGN is the only model capable of solving this task, performing only 7% worse than on the single speaker separation task.
Speaker Bio:
Shariq Mobin is the founder of AudioFocus, a startup developing new hearing aid technology using deep learning. He recently obtained his PhD in auditory neuroscience at the Redwood Center for Theoretical Neuroscience under Professor Bruno Olshausen. His research focused on connecting human attention to deep learning models of sound.