Mapping tap gestures to sound and video clips

While in the Master of Arts in Music, Science and Technology program at CCRMA, Stanford University, I helped Luigi Castelli and Eto Otitigbe in designing a system for mapping tap dance gestures to audio and video clips. The system would be used by tap dancer and MIT professor Thomas Defrantz, in a play called Monk's Mood. In the play, Professor Defrantz tries to tell the story of jazz musician Theloneous Monk through tapdance.

While designing the system, we decided to use PlayStation2 Dance-Dance-Revolution pads which are often used with dance games on this popular game console. To connect the dance pads to a computer, we used a PlayStation to USB adapter (PsJoy). We decided to use MAX/MSP to do the audio processing backstage and Resolume to show and mix video clips during the play. To connect the audio and video systems to the dance pads, I implemented DirectInput (part of the DirectX game programming library from Microsoft) to MIDI mapping software in C++. The setup for the play is shown below.

Because of the nature of tap dancing, Eto decided to build a wooden enclosure for the dance pads, and inserted foam to make them less sensitive. Close communication with our tap dancer, Thomas Defrantz, was necessary to understand how to adapt the technology to his needs. Below you can see a picture of the finished dance pads on stage. Note the white tape in the corner to make the pads visible during the performance.

During the performance, my DDR to MIDI software would map tap dance gestures to MIDI notes which would be sent to Resolume for video triggering and to MAX/MSP for audio triggering. Between scenes in the play, banks of video material would have to be switched by me manually, while Luigi would load up a new bank of sound clips in MAX/MSP.

The performance was a great succes. If you also would like to use dance pads in your performance, feel free to send me an email and I will help you build a similar setup. STEIM in The Netherlands also has software available (called JunXion) for mapping joystick (and dance pad) data to MIDI, and you can also use insprock for MAX/MSP to get data from USB game devices.