Alex Han
Click here to watch without dumb YouTube compression
I am happy to present my final submission for Homework #2 "Sound Peeking"! I titled my project cloud9--the piece features stormy weather, and a whirling battery of (yup, 9!) clouds working in concert to create a continuous unfolding of musical phrases. It slips in and out of chaos, but my intention was not to evoke the angst, tumultuousness, or dark mood that thunderstorms are usually associated with. I tried to imagine thunderclouds as a creative force, one that is ultimately out of our control.
Controls and Technical Details
The visualizer is highly interactable, and the video I present is a live "play-through" with very little hard-coded or through-composed (the thunder and rain sounds I made from scratch in ChucK!). I hope to extend functionality further in the future, as there was a good deal more I originally wanted to add. The player can toggle on/off camera control with mouse-look by pressing 'C'. When not in camera-control mode, players can control any number of the clouds that appear. There are 9 cloud objects in this scene, each representing a different instrument or sound. Each cloud is mapped to a number 0-8: by pressing the NumPad digits players can spawn and despawn specific tracks, and with the AlphaNum digits players can toggle on and off movement control of each cloud separately. AlphaNum 9 allows a "reset" by enabling control of all 9 clouds. Moving the clouds affects the volume of the tracks, with clouds reaching maximum volume just as their lightning-tendrils flicker around the powerlines (spectrum).
Cloud reactive features
The clouds themselves respond differently to each track. The visual flashes are tied to the loudness of the track within a certain frequency band. I ran audio analysis in ChucK to find the centroid of the FFT, and then used the centroid as a rough guide to picking a frequency band to pay attention to. I ended up hard-coding the bandwidth/endpoints due to some issues, but theoretically that should also be calculated automatically. I then took the RMS of the banded frequency spectrum's magnitudes, and sent this to Unity, where it controls the flashing/distortion of the clouds and the brightness of the lightning flashes.
Lightning reactive features
The lightning bolts are represented by the waveform of the total audio signal, with some noise and randomness integrated to make it feel more lively yet still unified. The lightning bolts are always spawned within their parent cloud, and periodically abruptly adjust their angle of descent and displacement relative to their parent cloud. I originally wanted them to seek out the current centroid of the spectrum below, but I didn't have enough time.
Frequency Spectrum
The frequency spectrum is represented by glowing lines below the clouds, showing history extending into the distance. The lines are supposed to evoke telephone power lines--another idea I had was to make them black, and they would light up with electricity when struck by lightning. More future directions to explore! The number of history lines, their spacing, and whether they extend in one direction or also mirrored, are all controllable parameters
Build details
I did my build in Windows. I didn't use any 3rd party assets or libraries, other than a free skybox from the Unity asset store. I learned a LOT from YouTube, looking at the tutorials posted in Discord dozens of times over. I watched a lot of videos from a couple creators in particular: Peer Play, Gabriel Aguiar, and Ned Makes Games
Reflections
This was a lot of work. I won't pretend that it wasn't difficult and insanely frustrating at times, and I spent a lot of time working super late into the night the last two weeks. I am grateful for the help from my amazing peers in my cohort, the active discussion in Discord, and advice from the masters (Julia, Ge, Chris). I learned so many concepts and skills in a short amount of time, more than I thought I was capable of. While there are many hopes and dreams I had for my project that didn't pan out, I do feel a sense of accomplishment with what I did do given my lack of experience with computer graphics, or just coding in general. I think the biggest benefit I got out of this, above the tangible technical skills, was learning how to learn--rigorously and rapidly. Learning coding concepts on the fly is something that is still pretty new to me, so having to "drink from the fire hose" via this class is both challenging and exciting. I'm excited to keep going!