Digital Signal Processing in the Browser

Created by Myles Borins / @the_alpha_nerd

##What is the Web Audio Api? ![HTML 5](img/h5_logo.png)
The [Web Audio Api](https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html) is a high-level JavaScript API for processing and synthesizing audio in web applications. It is currently being spearheaded by Chris Rogers at Google.
The Web Audio API comes with a number of natively compiled audio nodes capable of doing quite a bit of advanced synthesis. You can check out Hongchan's [WAAX](https://github.com/hoch/waax) library for an example of extensive work being done with native nodes.
But What if you want something more?
I present to you ##The JavaScriptNode
The JavaScriptNode allows individuals to create their own web audio nodes in pure JavaScript. This allows individuals to extend the Web Audio Api with custom nodes.
Web Audio Libraries such as [Flocking](flockingjs.org) by Colin Clark and [Gibber](http://www.charlie-roberts.com/gibber/) by Charlie Roberts make extensive use of the JavaScriptNode.
======== #WARNING ======== Currently native Web Audio nodes and JavaScriptNodes don't play so nicely together, most implementations of Web Audio tend to pick one or the other. So what do you want, stability or the bleeding edge
##What does faust look like? Below is an example of Noise Written in Faust ``` random = +(12345)~*(1103515245); noise = random/2147483647.0; process = noise * vslider("Volume[style:knob]", 0, 0, 1, 0.1); ```
##What does a WebAudioNode look like? Below is an example of White Noise taken from Flocking ``` flock.ugen.whiteNoise = function (inputs, output, options) { var that = flock.ugen(inputs, output, options); that.gen = function (numSamps) { var out = that.output, i; for (i = 0; i < numSamps; i++) { out[i] = Math.random(); } that.mulAdd(numSamps); }; that.onInputChanged = function () { flock.onMulAddInputChanged(that); }; that.onInputChanged(); return that; }; ```
##But Doesn't Faust Already compile to web audio?
##But does it work? [Current Faust2Webaudio Noise](examples/current-noise.html)
##Why did that break?
There is only one answer...
##Before we start a flame war ![Flame War](img/flame_war.gif)
#I freaking love JavaScript
##But there are some things it can't do...
#Like Integer Arithmetic
##asm.js to the rescue! [![asm.js](http://static.techspot.com/images2/news/thumbs/2013-05-23-teasercc7.jpg)](http://asmjs.org/)
asm.js is a strict subset of JavaScript that can be used as a low-level, efficient target language for compilers. The asm.js language provides an abstraction similar to the C/C++ virtual machine: a large binary heap with efficient loads and stores, integer and floating-point arithmetic, first-order function definitions, and function pointers.
##What does an asm.js ##WebAudioNode look like? An example from the asmjs Flocking Branch ``` flock.ugen.asmSin.module = function (stdlib, foreign, heap) { "use asm"; var sin = stdlib.Math.sin; var pi = 3.14159; var out = new stdlib.Float32Array(heap); function gen (numSamps, freq, phaseOffset, mul, add, sampleRate, phase) { numSamps = numSamps|0; freq = +freq; phaseOffset = +phaseOffset; mul = +mul; add = +add; sampleRate = +sampleRate; phase = +phase; var i = 0; for (; (i | 0) < (numSamps | 0); i = i + 1 | 0) { out[i >> 2] = +(sin(phase + phaseOffset) * mul + add); phase = +(phase + (freq / sampleRate * pi * 2.0)); } return +phase; } return { gen: gen }; }; ```
##So this is all great, but I don't want to hand roll asm.js like a sucker...
##Introducing [![Emscripten](img/emscripten.jpg)](http://emscripten.org/)
Emscripten is an LLVM to JavaScript compiler. It takes LLVM bitcode (which can be generated from C/C++ using Clang, or any other language that can be converted into LLVM bitcode) and compiles that into JavaScript, which can be run on the web (or anywhere else JavaScript can run).
##Faust -> C++ So using the faust compiler (specifically the faust2-asmjs branch) we can compile from [faust](https://gist.github.com/TheAlphaNerd/5710239#file-noise-dsp) to [C++](https://gist.github.com/TheAlphaNerd/5710239#file-noise-emscripten-cpp) with the following command ``` faust -a minimal.cpp -i -uim -cn Noise \ dsp/noise.dsp -o cpp/faust-noise.cpp ```
In order to get access to the verious parts of a C++ class via emscripten we need to write a simple wrapper on top of our Noise class. ``` // Adapted From https://gist.github.com/camupod/5640386 // compile using "C" linkage to avoid name obfuscation extern "C" { float** fInChannel; float** fOutChannel; int numInputs; int numOutputs; //constructor void *NOISE_constructor(int samplingFreq, int bufferSize) { // Make a new noise object Noise* n = new Noise(); // Init it with samplingFreq supplied... should we give a sample size here too? n->init(samplingFreq); // Lets get this once so we don't need to keep calculating every call to compute numInputs = n->getNumInputs(); numOutputs = n->getNumOutputs(); // This Needs to be dealt with... curently only able to return a single buffer... rather // Way too mono... // This is due to the Channels not being properly initialized... need to look at how // Other architecture fiels do this a bit better for (int i = 0; i < numInputs; i++) { fInChannel[i] = new float[bufferSize]; } for (int i = 0; i < numOutputs; i++) { fOutChannel[i] = new float[bufferSize]; } return n; } FAUSTFLOAT *NOISE_compute(Noise *n, int count) { n->compute(count, fInChannel, fOutChannel); // Returning due to problem as mentioend above... return fOutChannel[0]; } int NOISE_getNumInputs(Noise *n){ return n->getNumInputs(); } int NOISE_getNumOutputs(Noise *n){ return n->getNumOutputs(); } void NOISE_destructor(Noise *n) { for (int i = 0; i < numInputs; i++) { delete fInChannel[i]; } for (int i = 0; i < numOutputs; i++) { delete fOutChannel[i]; } delete n; } } // Number of channels gotten from getNumInputs / getNumOutputs // fOutChannel[i] = (float*)ioData->mBuffers[i].mData; ```
We can then compile the resulting C++ file to asm.js using emscripten with the following command ``` emcc cpp/faust-noise.cpp -o js/faust-noise-temp.js \ -s EXPORTED_FUNCTIONS="['_NOISE_constructor','_NOISE_destructor', \ '_NOISE_compute', '_NOISE_getNumInputs', '_NOISE_getNumOutputs']" ``` Which creates this [JavaScript file](https://gist.github.com/TheAlphaNerd/5710239#file-noise-emscripten-js)
Finally we apply the following JavaScript wrapper in order to break out the functions we wrapped earlier in C++ ``` // Adapted From https://gist.github.com/camupod/5640386 var faust = faust || {}; (function() { faust.context = new webkitAudioContext(); var NOISE_constructor = Module.cwrap('NOISE_constructor', 'number', 'number'); var NOISE_destructor = Module.cwrap('NOISE_destructor', null, ['number']); var NOISE_compute = Module.cwrap('NOISE_compute', ['number'], ['number', 'number']); var NOISE_getNumInputs = Module.cwrap('NOISE_getNumInputs', 'number', []); var NOISE_getNumOutputs = Module.cwrap('NOISE_getNumOutputs', 'number', []); faust.noise = function () { that = {}; that.ptr = NOISE_constructor(faust.context.sampleRate); that.getNumInputs = function () { return NOISE_getNumInputs(); } that.getNumOutputs = function () { return NOISE_getNumOutputs(); } that.compute = function (count) { return NOISE_compute(that.ptr, count); } that.destroy = function () { NOISE_destructor(that.ptr); } return that; } }()) noise = faust.noise(); ```
#But does it work?
#Sure, #[take a listen](demo/index.html)
##But it is unnecessarily complicated
###Managing memory in C++ while maintaining persistence when compiled to JavaScript can be tricky.
###Specifically because faust represents inputs and outputs as a float **
That means the input and output variables are in fact a pointer to an array of channels
Each channel in the array is a pointer to a buffer
Each buffer is an array consisting of the samples represented as a floating point number between -1 and 1.
##This seems unnecessarily complicated
##Why should we be spending time managing these pointers?
##How are we supposed to deal with the pointers in JavaScript land without creating lots of garbage?
##But didn't I say Faust is Functional?
##Indeed it is
###So why are am I going from Functional -> Object Oriented -> Functional
![Clever Girl](img/clever-girl.png)
##Emscripten comes with its own virtual machine as well as a number of compilation optimizations
##Simply put, a bunch of performance for free!
##That made it a tempting starting point
##As a primary research goal is to create very efficient web audio based unit generators
#Moving Forward ##Faust -> asm.js directly
###Prior to being compiled to C++, faust is compiled to Faust Intermediate Representation (FIR) It is in this form that Faust is compiled to specific language architectures such as C++ and java.
##It is from FIR that we can compile directly to asm.js
##While Faust -> JavaScript directly might be more elegant
###There is no guarantee it will be faster
###As such we need to continue to develop both and compare via benchmarks to see which method ends up being more efficient
##In Conclusion Faust in the browser would be awesome
##But we are not quite there yet.
###Even if we solve the compilation issues, we will still run into the ceiling of the browser due to all processes running in a single thread
##But if WebAudio is going to have a future, it is important to be able to extend it.
#Find the source on [Github](https://github.com/TheAlphaNerd/faust2webaudio)

Thank You

by Myles Borins