Software engineering in real-time audio applications: Master's thesis

I started this research in my Master's thesis. The main contribution of my Master's thesis is the application of generative programming or meta-programming in general to sound processing and fixed-point arithmetic. My primary goal in my thesis was to implement a real-time audio processing application on a handheld computer, the Compaq IPAQ H3600 series (PocketPC OS and Intel StrongARM processor):

 I first compared fixed-point and floating-point arithmetic on the handheld device by implementing a benchmark that performs a multiplication x = a.y where x and y are vectors and a is a scalar. The multiplication was implemented as a for loop and repeated several thousand times. The results showed that fixed-point arithmetic on the handheld device was roughly 10 times faster than floating point arithmetic, which is logical since the handheld does not have a floating-point unit and relies on a (slow) library of floating-point routines.Below you can see a screenshot of the benchmark application.

I wanted to improve the fixed-point arithmetic code before using it in my audio processing application, by using generative programming to automatically generate code for different fixed-point datatypes. This allows the software engineer to use multiple fixed-point types (instantiations of my C++ template class Fix32) which have different wordlengths for the fractional part of the number, and use them in algorithms without worrying about how many bits to shift left or right. The generative programming code automatically generates the correct source code at compile time to do operations on fixed-point datatypes with different wordlengths for the fractional part, allowing the software engineer to concentrate on the main issues of the implementation. 

In the beginning of the design and implementation, I implemented my own Audio I/O code for the PocketPC, using the Win32 API. There were some problems with deadlocks (I suspect a feature in the Win32 API was not properly documented) and after a while I ported the popular toolkit PortAudio to the PocketPC platform, somewhere in March 2001. Eventually other people started porting PortAudio to this platform as well, so now you can download PortAudio and the necessary extensions on the PortAudio website in case you want to do your own audio programming for handhelds. Or you can also use FMOD, an advanced audio I/O library.

The applications I implemented for my Master's thesis used MFC on PocketPC for user interfaces, but I recently have been using GapiDraw and my own widget library on top of GapiDraw instead. GapiDraw is a very fast graphics library typically used to implement computer games for handheld devices. It runs on many platforms and is very easy to use and was developed by Johan Sanneblad of the Viktoria Institute.  The reason I am not using MFC anymore is that it is not flexible enough to build custom user interfaces which are essential when performing Human-Computer Interaction experiments. Also, MFC is very slow compared to the direct screen access you can get through GapiDraw.

I am currently working on a software sampler application for StrongARM PDAs such as the Asus A620BT Pocket PC. The application will be used in usability research for musical applications on handheld devices. Below you can see 2 screenshots of this application.

Same deck screenshot (8 sample decks in total)

Sequencer screenshot (16 steps)

The application consists of 8 sample decks which you can flip through using the cursor pad on the handheld device. You can load wave files (.wav) into the decks by tapping the file name. Next to the deck number are three buttons, (M)ute, (S)olo and (P)lay, which can be used to (you guessed it) mute, solo or play the deck. Below the filename are a number of rotary knobs which can be used to change settings such as panning and bit reduction (to simulate old drum machines for example). You can also pitch sounds up or down and change the length of the sound (truncation, not time-stretching).

 I find this application interesting because it is impossible to transfer an existing user interface from a desktop application to this handheld device without drastic modifications. The user interface needs to be completely re-designed because there is only a small screen and stylus available, and the screen is limited to 320X240 pixels. If you would transfer an interface using rotary knobs from a desktop application to the handheld device, the knobs would be impossible to use with the stylus. What I find even more interesting is whether these knobs are the right GUI control or not for this type of application. My theory is that musicians using handheld devices are primarily interested in sound exploration while they are on the road. If they want to compose, they will use a laptop. Handheld computers are very interesting for ambient sound exploration, in a way that is similar to the Surfman, a rare device similar to a walkman, but continuously playing a certain type of noise in which the listener would detect patterns and other sounds after an extended period of listening.

You can download the PowerPoint presentation for my thesis defense here. Please feel free to send me an email if you want more information or if you find this research interesting and if you have comments or suggestions.