Creating a Cross-platform Application for the Sonification of Complex Data

Michelle Daniels
Music 220c - Spring 2001

The Motivation

Background

For several months I have been working on a sonification project with Jonathan Berger and Oded Ben-Tal, in which we are using subtractive synthesis and changing formant filters to create sonic representations of complex data. Check out our website for details.

STK (Synthesis Toolkit)

• STK is more efficient than CLM in general
• better than CLM for real-time sound

Application/GUI

• cross-platform potential with Qt libraries
• real-time interaction with parameters
• streaming input of data for sonification in real-time
• OOP

The Steps

Translating the CLM vowel instrument to an STK instrument

• translating the instrument itself and learning STK
• writing an STK pulse-train unit generator
• writing an STK multi-breakpoint envelope
• translating the CLM formant filter into STK

Creating the application and GUI

• learning how to use the Qt libraries
• designing an effective GUI - what tools does a user need?
• implementing the GUI framework
• implement data loading/saving
• planning an efficient way of saving state/preferences between sessions (file format)
• using the GUI to call the vowel instrument and create/save sound files (multi-threading)
• integrating with Bryan Cook's parameter engine and possibly his existing GUI for real-time control of data mapping to sound parameters

The Results So Far

Vowel Instrument in STK:

• pulse-train and noise input options
• can be initialized from phoneme or array of formants
• formants can be specified with frequency, bw, and gain envelopes
• uses a translation of CLM's formant filter for consistency between environments

Current GUI:

Implemented GUI Features:

• plays 5 different reference phonemes
• ability to change input from noise to pulse train and edit input amplitude and pulse train frequency
   
  
   
• multi-threaded so GUI remains functional while sound plays
• ability to load and display a data set with three columns and a variable number of rows
   
  
   
• nice "about" box
   
  

The Future

Vowel Instrument in STK:

• allow file input of various kinds
• allow minimal spatialization (amplitude panning to start with)

The GUI:

• complete goals listed above, beginning with the following:
• limited sonification ability
• more flexibility with data format
• saving sound files
• integration with Bryan Cook's parameter engine

Further Down the Road:

• a fully interactive sonification application running in real time
• a portable sonification application for PDAs