In this programming etude, I used Wekinator to create fun musical instruments that are controlled through various types of inputs. I train Wekinator on inputs such as object position, camera input, and face detection and use ChucK to generate the output sounds. This is based off the CS 470 Programming Etude 3 Assignment.
For this pong project, I created a fun tool that will output sounds according the position of an object in a game. I trained this on 2 inputs, the x and y position, and the the 5 outputs control frequency, harmonics, and envelop parameters. For the sound, I designed it to sound more playful.
For this project, I wanted to re-create the Harry Potter ticking song. I thought it would be fun to train a classifier on images of the different Harry Potter characters, and the user can show different images to play the section of the song corresponding to that character. I train on camera data as input and output 1 number corresponding to which character is being shown.
I used FaceOSC to take in the mouth width and height, left and right eye, and jaw data as input. The output is 4 values for the frequency and gain of FM synthesis. I trained the model on 5 expressions (absent, asleep, bored, confused, note-taking/attentive, asking questions) relating to how a student might feel in class to create a music generator based on facial expressions. I thought it might be fun to have music play depending on the user's facial expression and mood while in class. I tried to associate appropriate sounds according to the mood.
Overall, I found this to be a highly entertaining process. It was really fun to be able to experiment with all the various input types and how I wanted the output to sound. I wanted to explore different modes of input so I tried to make something with each of them. The fact that they could be whimsical and useless was freeing because I could just create something and not have to think about its purpose.
It was also surprising how simple and straightforward it was to train the Wekinator system. Each one led to fairly accurate results even with a few training examples. The input the system seemed to have the most problem with was the camera input. With the camera, I had to re-train the inputs multiple times in order to get the system to be more accurate. I also struggled with designing the output sounds. I had wanted to create the 8-bit style music, but I wasn’t able to do so, and I felt constrained by my ability to design and create the sounds that I wanted. In the end, I just tested out various of the ChucK examples to find sounds that I could tune instead.
This etude also made me really think more about the purpose of these machine learning systems and music. I was talking with a friend about this project, and I showed them some of these tools. However, they were confused as to what the point of them was and why I had spent time making these. While I had no good answer to this, I do think there is merit in thinking about and creating these systems. For example, with my pong tool, there is no apparent use for it and is not necessarily obvious that the music was trained. However, it’s interesting to think about how machine learning could be integrated in small ways that we might not notice, and it could already be part of our everyday life.
I used example output ChucK code (face2wek.ck, fmsynth-5outputs.ck) from the assignment to help create these. I also used the Wekinator input examples for simple mouse position, camera input, and face OSC. For the Pong game, I used code from Gabriel Lovato and made some slight modifications. For the DVD bounce, I used code from Elias and made some slight modifications. I also used example code from the ChucK website for help with sound design.