Analysis of Expressive Timing and Body Movement in Violin Performance
For Stanford University Anatomy of Movement 2006
Matt Wright, Sheena Chandran, and John Stoecker
Sonification Walkthrough
Direct Sonification
We started with directly creating sound files from our data. Here are a few:
The force in the X direction on the forceplate of the normal piece
The X position on the forceplate of the no emotion piece
Listen for the ghostly moans in the background of the X-force sound file.
Timbre Sonification
One of our hypotheses was that movement shows up in musical timing. We found that the moment around the vertical axis corresponded with note timing (which is unsurprising, considering that he mostly bows once per note), and the mediolateral movement corresponded to measure timing.
Sonification of these results must show the data change in relation to the music, so we overlaid timbre modulation on audio recordings of the performances.
We loaded the direct sonification files into Pure Data, a musical programming environment, and mapped values to timbre.
Timbre is the perceived quality of sound, and although no quantifiable measure of timbre exists (as Professor Berger pointed out in his lecture on March 8), most models aknowledge that harmonic content determines a large part of timbre.
Harmonic content is all the tones of a note that aren't the actual frequency of a note - for example, if you play an 'A' (at 440 Hz) on the piano, you are creating not only volume at 440Hz, but also 880Hz, 1320 Hz, and many other frequencies. These all add together to create a complex tone, and your ear detects these harmonics as timbre.
To modulate timbre, we used a band-pass filter. This filter allows certain frequencies within a range to pass through, and rejects the other frequences. When we change the range, the harmonic content of a tone is altered, and so is the timbre.
Band Pass Filter - www.sfu.ca
So using Pure Data and a band-pass filter, we altered the timbre of a tambura drone. We chose the tambura, an Indian stringed instrument, because of its rich harmonic content - a band pass filter modulation over a tambura sound gives a nice timbre sweep. Here is the first bit of mediolateral movement mapped to timbre:
A clip of the normal piece sonification, X-position on the forceplate
We can hear the violinist moving away and then back towards the origin (swaying right and then left).
Sonification of the normal piece, X-position with audio overlaid
This graph shows measure timing overlaid onto the X-position. The sonification not only shows the correlation, but we can reconstruct the violinist's movement in real-time in relation to the piece. The sonification might even show the correlation more effectively than the graph!
Here is the first bit of the no emotion moment-Z sonification overlaid onto the no emotion piece audio recording:
A clip of the no emotion moment Z sonification
We hear the modulations in the sonification indicate that the violinist twists every note. This is consistent with the above graph, where the dotted lines represent beginnings of every note.
When we combine the two sonifications into one piece, we get this:
Mournful piece sonifications with audio overlaid
We hear a higher tone modulating faster (this is the moment-Z), and a lower tone modulating very slow (the position-X). Note that this sonification isn't as clean as the other two; this shows up in his movement. In the no emotion piece, we had a very clean moment-Z rotation (i.e. the distance between peaks and valleys stays roughly the same). This was probably because the violinist wasn't moving around the forceplate, but staying in one spot when he played with no emotion. But when movement around the forceplate is introduced, the violinist's moment-Z isn't as clean.
Frequency Sonification
Frequency is the pitch of a note. To sonify our rho correlations, we mapped each data point in the correlation set to a sine tone. The frequency of each sine tone modulates higher and lower as the data moves up and down.
Consider our most significant rho value correlation of -.44 between X-position on the forceplate and the tempo in the no emotion piece. What does a rho value of -.44 mean? The sonification will give an idea:
X-position mapped to frequency
Listen for the frequency to move (generally) from high to low. This means the violinist is moving generally towards the origin. (You remember that we found X-position to modulate with respect to the measures - in this piece, this was not the case because he stood in one spot mostly and didn't sway.)
Tempo mapped to frequency
Listen for the frequency to move from low to high. This means the violinist is speeding up throughout the piece.
Both together
Listen for contrary motion, when one tone goes up while the other goes down. This contrary motion gives the rho correlation its negative value. (You might need pretty good speakers to hear them both all the way through.)
Alright then! Hope you enjoyed the tunes! For your listening pleasure, here are all of the unnamed violinists recordings:
Normal
Angry
Mournful
No Emotion
Playful
Searching
Sparse Motion
Trepidation