Programming Etude #3 (Final Deliverable): "Wekinate Your World"


In this programming etude, I used Wekinator to create three interactive AI utilities/toys.

Phone MIDI Pitch Controller

Download Files

For this system, I explored using an iPhone as input and using MIDI output. For the iPhone input, I downloaded an app called MotionSender. MotionSender allows you to send the phone's accelerometer, gyroscope, and attitude data to Wekinator through OSC. After playing around with the different data types, I felt like the attention data was the most useful. I used Wekinator to map the phone's rotation to an OSC output with a range of 0-127. I then used Processing to turn the incoming OSC data into a MIDI output. To do this on a Mac, you must open the Audio MIDI Setup application, go to Window -> Show MIDI Studio -> IAC Driver, and click on "Device is online." This allows Processing to send notes to a virtual MIDI device. Lastly, I used Logic Pro to create sounds, and I played around with various instruments. You can see in the demo what it sounds like with a guitar VST.

Video Camera Pitch Bend Controller

Download Files

For this system, I continued my exploration with using MIDI output. I used a program that allowed me to use my webcam camera to track a colored object in the x and y directions. I then used Wekinator to map the y direction of the movement to a number between 0 and 127. For the output, I used Processing code to send a MIDI pitch bend message based on the OSC input. In the video, you can see me moving my left hand up and down with a green piece of paper wrapped around my finger. This controls the pitch bend while I play on a MIDI controller keyboard with my right hand (not visible in the video).

AI Theremin

Download Files

For this system, I continued to use the webcam x, y object tracking as input. For the output, I used a SinOSC in ChucK to produce sound. In the ChucK code, the x direction is used to adjust the frequency of the sound, and the y direction is used to adjust the gain. This allowed me to control the sound in a somewhat similar way to the way that a theremin is played.