Bryan Pardo on Voice Assist and Cerberus

In this talk, Bryan Pardo will describe two recent projects developed at Northwestern University’s Interactive Audio Lab: VoiceAssist and Cerberus. VoiceAssist is a system that helps inexperienced users produce high quality recordings by providing real-time visual feedback on audio quality. Cerberus is a single deep learning architecture that can both separate an audio recording of a musical mixture into constituent single-instrument recordings and transcribe these instruments into a human-readable format at the same time.

VoiceAssist: Voice recording is a challenging task with many pitfalls due to sub-par recording environments, mistakes in recording setup, microphone quality, etc. Newcomers to voice recording often have difficulty recording their voice, leading to recordings with low sound quality. Many amateur recordings of poor quality have two key problems: too much reverberation (echo), and too much background noise (e.g. fans, electronics, street noise). VoiceAssist is a system that helps inexperienced users produce high quality recordings by providing real-time visual feedback on audio quality.

Cerberus is a single deep learning architecture that can both separate an audio recording of a musical mixture into constituent single-instrument recordings and transcribe these instruments into a human-readable format at the same time, learning a shared musical representation for both tasks. This novel architecture, which we call builds on the Chimera network for source separation by adding a third “head” for transcription. By training each head with different losses, we are able to jointly learn how to separate and transcribe up to 5 instruments in our experiments with a single network. We show that the two tasks are highly complementary with one another and when learned jointly, lead to Cerberus networks that are better at both separation and transcription and generalize better to unseen mixtures.

BIO: Bryan Pardo is co-director of the Northwestern University HCI+Design institute and head of the Interactive Audio Lab. Prof. Pardo has appointments in in the Department of Computer Science and Department of Radio, Television and Film. He received a M. Mus. in Jazz Studies in 2001 and a Ph.D. in Computer Science in 2005, both from the University of Michigan. He has authored over 100 peer-reviewed publications. He has developed speech analysis software for the Speech and Hearing department of the Ohio State University, statistical software for SPSS and worked as a machine learning researcher for General Dynamics. While finishing his doctorate, he taught in the Music Department of Madonna University. When he is not teaching or researching, he performs on saxophone and clarinet with the bands Son Monarcas and The East Loop.