By: Dominic DeMarco
Have you ever been whistling by yourself, imagining complex harmonies in your head yet never able to hear them? Well, worry no more, as the Whistling Companion is here to sonify your deepest harmonic dreams (coherency and accuracy is not guaranteed). Now, you can whistle to an adaptive backing track that listens to the pitches you're whistling and watches your every (hand) movements to control pitches, loudness, and timbre.
Hand post recognition via Wekinator and HandPoseOSC
Pitch recognition via Chroma
Adaptively choosing intervals and chords to play based on most dominant pitches
Keyboard controls (z,x) to increase/decrease new chord frequency
Low-pass filter to reduce the high-frequency noise from the input hand recognition signal
These 10 weeks have flown by at a breakneck pace. I started this class unimpressed with AI, and three months can really change one's perspective. Yet, throughout this class, I also learned the value in simplicity. Sometimes the coolest concepts are based on deceptively simple implementations, and I think the Chroma tool is an excellent example of this. It feels like magic, having the program detect what you are whistling to generate a chord that fits your pitch, yet using the FFT and Chroma framework it takes only a few lines of code. The magic comes with how you use these tools to create newer, more expressive, instruments. I really like what I've ended up with, and it was fun designing an algorithm to map different pitch concentrations to sonorities. Simply put, the code works and is fun to play around with. Still, there are a couple of additional nice-to-have features, including different chord sonority generation modes. I could envision using a different algorithm that chooses chords according to a graph corresponding to western tonal harmony, for example. By adding more genre-specific rules into the chord generation, I could have my whistling companion suitable for a wider variety of musical styles.
When it came to deploying this into the real world, I do think I annoyed my roommate quite a bit, as he is not a whistler. While he did try singing (which also works as an input), he was not appreciative of me whistling to myself and my computer for hours on end, which is very understandable. The larger issue I ran into was a complete shutdown and freezing of ChucK after running this program that happens non-deterministically but prevents me from recording ChucK audio to a WAV file. As such, I went to plan B, recording my computer sounds to Audacity and then using my phone to capture my whistling. I didn't quite stitch back the video and audio perfectly, but I think it's accurate to within half a second and looks decently convincing. Doing any more fine-tuning in Windows Video Editor is not, unfortunately, very time effective.
Chroma Synth: chroma-synth.ck Hand Relay: hand-relay.ck Wekinator Project: wekinator.zip