Emerging Technologies for Musical Audio Synthesis and Effects

Julius Smith, with Romain Michon
CCRMA, Stanford University

Linux Audio Conference 2015 - Mainz, Germany - April 9, 2015

Introduction

  • Controllers
  • Embedded Systems
  • Mobile Platforms
  • Webaudio
  • 3D Printing
  • Faust
  • Trendy Research Topics

Controllers

Microcontroller Boards

What are they?

Why are they so successful?

The Arduino IDE

Arduino Shields

The family grew very big!

Are there alternatives to Arduinos?

Texas Instrument LaunchPad

The Wiring S Board

Where are we going?

Smaller and cheaper...

DigiStump's DigiSpark

More connected...

Spark's Core and Electron

More wearable...

Arduino Lilypad (Jacek Spiewla’s BeatGlove) and Adafruit Flora

More powerful...

Arduino Yun and Intel Edison

What's next?

Controller examples

Hybrid approach to control physical models

Latest Prototype of the BladeAxe

Haptic feedback

Ed Berdahl's FireFader / Apple's "Taptic" Technology

Embedded Systems

Embedded Linux Boards

From Gumstix to Raspberry Pi

Raspberry Pi

BeagleBone

UDOO

Pros

  • Extremely easy to use
  • Runs Linux
  • Power of the community
  • Price
  • Simplicity
  • Versatility
  • Plurality

Cons

  • Plurality
  • It is what it is...
  • Limited power
  • Booting time
  • Compatibility with external hardware
  • Sometime victim of their own success

Embedded programming environments

DSP shields and FPGA boards

Audio Codec Shield

FPGA Boards

Gadget Factory's Papilio

DSP shields

Axoloti / Simple DSP / Arduino DSP Shield

Stanford / Texas Instrument DSP Shield

Related Projects

ModDuo / Owl / LeafLabs Maple / StompShield / Open Stomp Coyote

Mobile Platforms

Example: the Moforte Guitar and Geo Shred

The MoForte Guitar

  • Full electric guitar + effects (implemented in Faust): 6 strings, general excitations, distortion, compression, phaser, 5 bands parametric equalizer, feedback, wah pedal and autowah, flanger, reverb
  • Responds to accelerometers, gyros, touches (pluck), swipes (strumming), etc.
  • Historically related to the Stanford Sondius Project, ca. 1995

A Word About Software Licenses

  • moForte guitar is a commercial offering using FOSS
  • Liberal licensing such as in FAUST makes this possible
  • GPL motivates complete rewrites from scratch
  • MIT license is one of the best (“just run with it”), as are BSD, STK
  • moForte presently consists of two guys (Pat and Nick) plus JOS (consulting), and commercial is their only option right now
  • We can contribute freely when we can, but otherwise, commercial products based on FOSS should be ok
  • For general purpose tools, consider the MIT license extended to request that bugfixes be submitted upstream
  • GPL makes sense for finished apps, but unrestricted libraries ideal
  • "Free for free use" + reasonable commercial license fee also works

Supporting and Organizing FOSS Development

  • We can create better systems for prioritizing, addressing, and funding FOSS development
  • Kickstarter model is good, but need something lighter weight for smaller software components
  • Google Summer of Code (SOC) is great, but also large-grained and hard to get into

FOSS Small-Task Example


On the Faust list, a company wanted the optimized filter-bank cases in filter.lib to be extended to higher order

  • Perfect example of a bid/hire item
  • Company receives a lower rate if contributing the improvement for all (e.g., 50%)
  • Developer sees the same compensation, making it possible to do the work
  • A general FOSS fund can support the discounted rate

iOS vs. Android: the sad truth

Device Touch-to-Sound Round-Trip
iPhone 5 36ms 13ms
iPhone 6 30ms 11ms
iPad 2 45ms 15ms
iPad Air 2 38ms 12ms
Samsung Galaxy S5 72ms 78ms
Google Nexus 5 90ms 92ms
Google Nexus 7 130ms 130ms

Measured Latency on Different Mobile Devices at 44.1KHz (Buffer Length 512 samples)

Webaudio

A quick example...

I Dropped my Phone the Screen Cracked

Two more complete example...

WAAX

DAW Plugins for Web Browsers (Aalto University)

What is good? What is working?

  • Cross-platform
  • Great for collaboration
  • Open source in essence
  • Wide variety of pre-made objects
  • Simple to use
  • Precise scheduling

What will be working?

(may be)

  • MIDI support
  • Audio workers
  • Better performances
  • More than stereo output
  • Compatibility of all the web browsers with Jack?

A more global vision of the use of the webaudio API in the open source world:

The Faust PlayGround

An interesting thing to look at:

Ionic

SonicCanvas: a good example by Trijeet Mukhopadhyay

3D Printing

3D Printing and Open Source Software

  • Lots of "Open Source" DIY 3D Printers

VoxVel by John Granzow and Fernando Lopez-Lezcano

3D Printing and Open Source Software

  • Lots of "Open Source" DIY 3D Printers
  • Huge community of "Makers" across the world
  • Open Source CAD softwares are still far away from commercial counterparts
  • Good medium for "electronic lutherie" and prototyping: CCRMA Workshop
  • Huge potential for research in music and Acoustics

3D Printing of Vocal Tract Models Derived from MRI of Expert Pitch Bends on the Harmonica

Scott Summit's 3D Printed Acoustic Guitar

Faust

Faust Architectures (1)

faust2alqt faust2alsa faust2jackserver
faust2alsaconsole faust2andoid faust2jaqt
faust2api faust2asm faust2javaswing
faust2asmjs faust2au faust2ladspa
faust2caqt faust2ck faust2lv2
faust2csound faust2dssi faust2mathdoc
faust2graph faust2ios faust2max6
faust2jack faust2jackconsole faust2netjackconsole
faust2netjackqt faust2octave faust2owl

Faust Architectures (2)

faust2paqt faust2plot faust2puredata
faust2ros faust2rosgtk faust2rpialsoconsole
faust2rpinetjackconsole faust2sc faust2svg
faust2vst faust2webaudio

Getting Started with Faust

Research

Nonlinear Wave Digital Filters

Kurt Werner and Julius Smith at CCRMA
based on recent work at Aalto University and elsewhere

Scattering Delay Networks (SDN)

Enzo De Sena, H. Hacihabiboglu, Z. Cvetković, & J. Smith
  • Wall-to-wall reflections modeled by digital waveguides
  • Extra waveguides match direct signal and early reflections

Conclusions

Thanks for your attention! :)