This is a more developed version of my milestone wekinator (described below). Using HandOSC, each part of my hand is mapped to a different control of the audio of a crumpled piece of paper that I recorded. Using various gestures (fist, index finger, pinky, backwards hand, etc.) the reverb, rate, pan, and gain change, and the performer plays a new sound everytime the space key is pressed. The "instrumentalist" is able to be quite expressive with this since they control the number of sounds made with the space key, and the timbre of the sound with their OSC detected hand. This wekinator is an expressive instrument using Wekinator Regression (continuous).
This is a fun instrument inspired by the kid's song/game (?) "Boom, snap, clap" (see https://www.youtube.com/watch?v=lBSteR_0vdQ). I used to do this all the time as a kid, not even with another person -- just in a fun "sassy" way. With this wekinator, the idea is that instead of imitating a boom sound or snap or clap, you just do a gesture, and an actual drum kick / hihat / clap sound happens! I used classification to map the hand gestures (hand OSC) to sounds.
This insturment is another "computer music-y" one that makes fun sounds! I used the body-recognition poseOSC to map different body positions to components of a granualr synthesis of a recording of me shaking my water bottle. You can change parameters like grain size, grain randomness, rate, position within the audio, and reverb. While exact mapping isn't clear withh the way Wekinator was trained, it's very fun to put headphones in and physically move and feel how the sound changes with your movements.
I was initially very excited to start working on this project because I had thought that the possible things you could do with Wekinator were limitless. But when I started, I realized that because our assignment was so limitless, it was really hard to choose what I wanted to do and what sounds I wanted to be mapped from which gestures. I experimented a lot with the regular video-input, but as shown in class, I had a lot of trouble with it and so it wasn't until I used the OSC hand recognition or face recognition or body recognition did I get any results that I was actually pleased with. I was really excited that with the hand recognition I was able to improve my crumpled paper milestone, because I was not happy with how it turned out for the checkpoint last week. Now, it actually feels like an instrument that is somewhat expressive and controllable. Overall, I had a lot of trouble choosing what I wanted the instruments to feel like (which I felt like was the important part of this assignment) and I am glad that I put in a lot of effort in experimenting once I came up with an initial idea. For example, with the granular synthesis instrument, once I chose the splashing water audio, it was easy to play around with what I wanted the physical mapping to feel like. It was fun to play with it until I found a mapping that felt right that I could train and reproduce. Another example is the boom snap clap one. This wekinator one was super fun just because it already had a gesture that I was used to. However, the training process was still an interesting experience because I'm not used to just moving my hand to create a sound. Overall, I honestly wish I had spent more time and energy focusing on one wekinator rather than three small ones, but I feel prepared to work on my final project now that I have learned how wekinators work and what the OSC inputs could be.
This project used VisionOSC, Wekinator, Lisa chuck starter code, and the basic wek-output.ck starter code.
I am not as pleased with this wekinator as I had hoped. I struggled a lot to figure out what mapping I wanted from camera input to sound, so going with a basic tri-osc wave for the pen was a first attempt. Then, I decided to play around with "weirder" computer music stuff (that I love) and went with an audio recording of me crinkling a piece of (pink) paper that was sitting on my desk -- so what better to map from visual to sound than that same type of pink paper? I wish the mapping was much more clear however. The vision for this instrument, as seen in the third video was the kind of mapping and control I wanted (ironically not using any of the AI component...). Big takeaway: maybe camera input was not the way to go for this kind of expressive instrument I created. Maybe something else would have been better (like a GameTrak or cursor position). I ended up having the sound only play when the space key was pressed (because that felt like more human interaction than no control with how the video is interpreted...), and that way you could speed up / slow down num bufs as you move. I think i want to tweak this first wekinator a bit more before moving on to my second two.