SoniMime
Jesse Fox, Jen Carlile
Music 250a, CCRMA
December 5, 2004


Introduction

SoniMime is a simple system to sonify human movement. The idea for SoniMime arose from discussions with Professor Jonathan Berger during another CCRMA course, Music 120: Auditory Remapping of Bioinformatics. From these discussions we realized we could develop a system for real-time sonification of movement data using ideas in interaction design combined with sensor and mapping techniques as taught in Music 250a by Professor Bill Verplank.


Design

SoniMime uses two ADXL-ADC accelerometer boards to collect data. We use an I2C communication protocol to send digital data from the boards to Port C of the AVRmini . The Atmel AVR processor is running a C program which unpacks the data and sends OSC messages via a serial connection to a linux machine running PD. All of the mapping and sound synthesis is done in PD.







Presentation

Because many of our mappings highlight the movement of a performer's hands, we built a 'stage' that isolates and focuses attention on the hands. The stage is a piece of painted black wood with two 8 inch holes and black cloth sleeves glued to the back of the board coming through the holes.






We also made elastic rings with cushioned pouches to hold the accelerometer boards that slip onto a finger so that the performer doesn't have to hold onto the boards.





Generalized Controller

When we started thinking about this project, we had the idea that our controller should be able to be used for many different purposes—a live composition tool or a teaching tool for dance just to name a few. We spent much of our time designing PD patches, each of which uses a separate mapping of the accelerometer data for different sonic ends.



PD Patches

[guppy_interface2.pd]
Modeled after Matt Wright's Pd patch for analog accelerometer data interpretation, a patch called "guppy" was written that encompasses OSC data retrieval, various filtering schemes and useable data output. All OSC data is received as a chunk of three 10-bit numbers, sent at a baud rate of 115200 bits per second that are "unpacked" within Pd. This data is split into the respective X, Y and Z coordinates measured by each accelerometer board and sent to a bank of filters.

It turns out that the raw accelerometer data is almost unusable as is. Each directional component of raw acceleration data is first sent through a DC blocker that reduces the DC offset of our signal before being split into two paths. The first path is a set of cascaded high-pass filters that output "impact" data. The second path is similarly a set of cascaded low-pass filters that output "tilt" data. The DC offset is removed with a settable value that lets us calibrate our system to each user.

The 10 total outputs of "guppy" correspond to three high-pass outputs (one for each direction), three low-pass outputs, three "delta" outputs that are essentially low-pass outputs once-derived as "jerk", and finally one binary output that detects relative stillness (or lack of a drastic change in low-pass output).

For our patches, we further modified our original "guppy" to interpret two sets of 3x10 bit numbers corresponding to the output of our two digital accelerometer boards. The end result is an object with three inputs (leftmost accepts a bang to calibrate, the other two accept stillness detection threshold values in milliseconds) and 20 (!) outputs. This abstraction was used as the main interface between Pd and the AVRmini in all of our work.


line_demo.pd/ plane_demo.pd
These patches were the first ones written to utilize accelerometer data in making sound. Prof. Verplank expressed his interest in the ability of a system that sonifies movement to teach dancers, for example, how to move precisely. The line-test.pd patch was designed to lend audible feedback to the user as to whether he or she was moving in a straight line. The patch maps any change in acceleration ("jerk", or delta(tilt) data in this case) to switch synthesized sound output from a harmonious major chord to filtered noise. When moving steadily in a line (or more realistically, moving without a sudden jerk in any direction), the user hears the nice chord, but after abrupt deviation from this smooth path, the user hears noise.

Plane-test is a similar patch, though instead of abrupt change in any direction, the user is limited to the horizontal plane (relative to calibration) before hearing noise.


mover.pd
In our quest to map acceleration data to something fun and responsive, we came up with "mover.pd". This patch quite simply exponentially maps any change in tilt to the frequency of a pair of sine-wave oscillators. Each hand controls one channel of sound output (L and R for example). When still, no sound is heard. Slow movements or tilts correspond to low ferquencies, while quick movements and sudden changes in direction correspond to high frequencies. The user has continuous control of frequency here, and pitch smoothly ramps from one value to another.


navigator.pd
Prof. Jonathan Berger encouraged us to experiment with synthesis techniques that allow one to navigate a virtual "timbre space". In this patch, we derived the tri-stimulus timbre model for use with an intuitive, tilt-based mapping. The basic idea of the patch was to map each hand to something different. One hand would control pitch and overall amplitude while the other hand controls the timbre of the resulting sound. The tri-stimulus model dictates a triangular "space" whose three corners are intuitively mapped to a tilt of the hand (left, forward, right). It is easiest to visualize this effect by imagining a triangular game of pinball balanced on the back of one's hand, held in front of the user. One corner of the game points left, another points right, and the third points straight ahead. By tilting the hand and therefore rolling the pinball to each corner of the game, we can achieve a position within our timbre space. Our mapping put the fundamental of the tri-stimulus model towards the left, the middle harmonics towards the center, and the high harmonics towards the right.

Functionally, this patch consists of a "guppy_interface2" object, and we are exclusively looking at guppy's tilt output for each hand. One hand's left/right tilt is mapped to the fundamental frequency of a bank of oscillators, while that same hand's forward/back tilt is mapped to the overall amplitude of the output. The other hand controls the relative amplitudes of three separate banks of oscillators that together form our tri-stimulus timbre model. The three banks are: fundamental frequency only, integer multiples 2-5 of the fundamental frequency (middle harmonics), and integer multiples 6-16 of the fundamental (high harmonics). Again, each control is continuous, and one can smoothly shift between banks to achieve a linear mix based upon position within our virtual model. Finally, each bank of oscillators is statically attenuated (based upon empricial measurements) in order to even out the relative amplitude of each oscillator.


scrubberST.pd
Another of Prof. Berger's ideas manifested itself in a patch we call "scrubber". The idea behind this patch was to enable the user to scrub through a sound file by tilting their hands. We gave each hand separate playback control (and separate channel output) of the same, buffered sound file. Once playback begins, the tilt of each hand controls the playback rate of the sound file. Tilt is mapped directly to playback rate, so a tilt to the right results in an increase in playback speed while a tilt to the left slows things down. Enough of a tilt to the left will even run the file backwards. Furthermore, each channel's output volume is controlled by fore-aft tilt of each corresponding hand. Finally, still hands result not only a normal playback rate, but if both hands are kept still, the two separate channels return to simultaneous, in-sync playback (at the same sample within our buffered sound file).


lightsaber.pd
This patch, by virtue of its title, is rather self-explanatory. By placing one accelerometer at the end of a broomstick, we can mimic the sound of a Jedi's lightsaber. Here, change in tilt is mapped to the amplitude of a wavetable oscillator, whose source is a sound byte from an "actual" lightsaber slicing through the air. Furthermore, a background hum is being looped to further increase the ferocity and realism of our broomstick.



Future Work

Wireless

One limitation of our current system is the performer must be wired. In our original vision, a dancer could move freely about a stage, unhindered by wires. At the present time we are limited to a radius of 10 feet or so centered around the computer due to signal degredation over long cables. Ideally, the dancer would have the two accelerometer boards attached to an AVR board worn in a beltpack with a transmitter sending data to another AVR board connected to the linux machine.


Sonified Golf Club

In the coming months we would like to use the building blocks of SoniMime to sonify a golf swing. We plan to attach an accelerometer and possibly some other combination of sensors to a golf club to optimize a person's golf swing through auditory feedback. This is a difficult task due to the fact that each golfer has a different optimal swing. However, if a person can achieve consistency in movement (which is aided by auditory feedback), it is easy to discover how different aspects of the swing motion affects the path of the golf ball.



Appendices

- C Code

With help from Wendy Ju, we developed a C program which we loaded onto the AVR board which interprets data from the accelerometer boards and sends OSC messages to the linux machine. Because we were using the digital outputs of the accelerometers, we were able to send data from each board over the same wires. We differentiated between accelerometer 1 and 2 by physically setting them to different addresses. By default, the address of the accelerometer board is ADS7828_I2C_ADDR, which is defined as 0x90. By simply unconnecting the two jumpers on the accelerometer board, you change its address (for example, by unconnecting both jumpers, the address changes to ADS7828_I2C_ADDR + 6). The digital data is actually comprised of 8 channels packed into 1. We only use channels 4, 5, 6 which correspond to acceleration in the x, y, and z directions, respectively.


Pd patches : a bundle of all of our pd patches
*note, patches are available separately under their description above


-NIME 2005 Abstract


Contact Information
Jesse Fox
Jen Carlile