DDSP: Differentiable Digital Signal Processing

Abstract: Classical DSP techniques have recently been overshadowed by deep learning for applications such as image recognition and audio generation. End-2-end learning has been key to this shift, enabling the optimization of high-dimensional nonlinear functions. However DSP techniques provide interpretability, modularity, and efficiency lacking from black-box deep networks. In this talk, I'll review the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning.

Bio: Jesse Engel is lead research scientist on Magenta, a research team within Google Brain exploring the role of machine learning in creative applications. He did his Bachelors, and Ph.D., at UC Berkeley, studying the martian atmosphere and quantum dot nanoelectronics respectively, and a joint postdoc at Berkeley and Stanford on neuromorphic computing. Afterward, he worked with Andrew Ng to help found the Baidu Silicon Valley AI Lab and was a key contributor to DeepSpeech 2, a speech recognition system named one of the ‘Top 10 Breakthroughs of 2016’ by MIT Technology Review. He joined Google Brain in 2016, where he his research on Magenta includes creating new generative models for audio (DDSP, NSynth), symbolic music (MusicVAE, GrooVAE), adapting to user preferences (Latent Constraints, MIDI-Me), and work to close the gap between research and musical applications (NSynth Super, Magenta Studio). Outside of work, he is also a professional-level jazz guitarist, and likes to include in his bio that he once played an opening set for the Dalai Lama at the Greek Theatre.