Webcam + Wekinator

3 Interactive Machine Learning Applications for Wekinator and Webcam

1. Celebrity in Disguise

Description:

Testing Webcam and Wekinator for interactive machine learning. This is a simple application that detects whether I am a normal person, wearing sunglasses, or a celebrity in disguise.

Processing Pipeline:

1. Webcam capture with Processing 4, downsampled to 18 pixels of RGB values, totalling 54 input dimensions
2. 54 input dimensions => Wekinator via OSC
3. Wekinator uses 3 neural networks with linear layers to perform linear regression mapping to 3 output dimensions
4. 3 output dimensions => ChucK via OSC
5. ChucK performs argmax over 3 dimensions---I'm a normal person, wearing sunglasses, or a celebrity in disguise

2. A Blue Gestural Synthesizer

Description:

Using the color blue to control a synthesizer in ChucK. Wekinator allows for a really diverse mapping and linear interpolation of inputs. Here, I explore color, in both the visual and sonic sense of the word. Tracked with my webcam in Python, blue becomes a wand for gresture control. Gestures map to sliders controlling polyphony, frequency modulation paramters, inversions, and the blending of voices. This allows Locke's Two Treatises of Government to be "read" in a new way, as a gestural instrument.

Processing Pipeline:

1. Webcam capture with Python and the cv2 library. RBG values are extracted and we create an image mask for the color blue.
2. I track the mean location of all blue blue pixels (x,y) and number of pixels that are blue (size)
3. 3 input dimensions => Wekinator via OSC
4. Wekinator uses 5 neural networks with linear layers to perform regression to 5 output dimensions
5. 5 output dimensions => Chuck via OSC
6. ChucK performs FM synthesis with 7 oscillators. The parameters that are controlled and rough mappings are:
   • (x) Pitch
   • (x,y) Modulation Frequency
   • (x,y) Modulation Gain
   • (x,y) Inversion of voice positions
   • (size) Number of voices and chordal blending

3. Image Contour Music Scrubbing

Description:

Using Image Processing techniques in Python and interactive machine learning with Wekinator for musical control in ChucK. Image contour detection is similar to edge detection but also connects edges into closed paths. In this Webcam + Wekinator application, I'm finding image contours for object detection, and getting the centroid of the contour as well as the arc length of all the contours. As the centroid leans to the right, the music speeds up, and vice versa. With the presence of more contours, aka more edges in the picture, the music becomes more distorted. This comes together for a unique and interactive mapping between image contours and music. This works best on a clean background but can be easily retrained with Wekinator's interactive machine learning approach on the fly.

Music is Candyland by Tobu

Processing Pipeline:

1. Webcam capture with Python and the cv2 library. Input is grayscaled, and contours are extracted using threshold analysis cv2.threshold() and cv2.findContours()
2. Image contours are cleaned, denoised and centroid and arc length are computed
3. 2 input dimensions => Wekinator via OSC
4. Wekinator uses 2 neural networks with linear layers to perform linear regression mapping to 2 output dimensions
5. 2 output dimensions => ChucK via OSC
6. ChucK uses SndBuf2 for music file playback, and continually updates playback rate with output 1 (rate between .5x and 1.5x) (reverbed when slowed)
7. Output 2 is used for distortion mapping, using ChucK's Gen17 UGen for Chebyshev polynomials. Distortion is modified from Colin Sullivan 2009.

My Reflections on Webcam + Wekinator

For this Wekinator Etude, it was a lot of fun only using Webcam as the input device. Python and the cv2 library make this really easy to do. It's admirable in a way how I've put all my image processing knowledge traditionally for medical imaging techniques to use to make funny sounds. Wekinator is pretty easy to use and nice to be able to leverage it's interactive machine learning approach for training on the fly. I was able to get a lot of success and control using extremly low dimensional input data.

The biggest hurdle that I had to overcome in this assignment was noise. Especially because I'm using webcam input, the location that I'm in is super important. The blue synthesizer works best indoors but even then there can be a lot of blue leakage from lights or digital noise. The image contour one works best on a clean background and color doesn't matter. If I move to a new location, I need to train the entire system again. Thankfully Wekinator makes this easy. Sometimes, that noise is part of the charm. But despite all this, webcam is a really fun input device to use, especially in all the ways that the webcam wasn't designed to be used. The precision is really good and overall super responsive.

Download all the source code: here

Learn more about this programming etude with Wekinator: here