My AudioVisualizer, Lipsythizer, is a rendering of a human/robot's mouth. It responds to sound and volume as the waveform makes up the lips, the lips grow in size depending on amplitude, and the spectrogram history comes out of the mouth as it intakes audio or live sound. The narrative/composition that I have constructed is meant to showcase the feeling behind this visualizer-- one of isolation and pain-- from the perspective of a robot. The sentiments expressed are very human, however, and as none of us identify directly as being a robot, it is easier to let ourselves out of reality for a moment, listen directly to it's message, and feel the transcendence and rawness of what it is expressing as something that we all feel sometimes. The vocal effects and performance of this composition were also used to emphasize this same overarching message.
To use the Lipsythizer, you simply press play. Then, you can press "1" or "v" to change the background to dark blue, "2" or "f" to change it to peach. "v" also adds a low, growling vocoder effect. "c" adds a soft chorus effect and a medium-harmonic pitch shift. "f" adds a drastic upward pitch shift and chorus effect, "d" adds a chorus effect and a pitch downward effect, and "g" adds a delay effect. Also, on top of these effects, the y-position of your mouse in Chunity will add another pitch shifter that will move the pitch up and down as you move your mouse up and down the screen. To change from audio input to microphone input, you would have to go to vocoder.ck file and change buf input that is going to dac to adc and not spork the play function. Also, if it bugs and builds with spectrograms only on one side, you must clit Spectrogram1 -> scale -> x -> and change it from negative to positive or vice versa and same for Spectrogram 2.
To create the final version of this AudioVisualizer, I used a few platforms. I used Chuck to create the background music for the narrative and the sounds that you hear, and then exported the music that I made in chuck and loaded it into unity as the background music tracks. I also made all of the vocal patches that occur on key command, and I am running all of my chuck through the vocoder.ck file in streaming assets. I also used the Chuck Follower.ck code (modified) to make the lips and the spectrograms move when sound occurs. I used Unity to create the visuals and to code the pitch-mapping of the pitch-shifter following the mouse key. I used sound samples that I recorded from Logic x Pro and bounced them as wavs in my streaming assets for the vocals of the song and for the mouse click sounds. Big shout out to Julia, Celeste, Aaron, Taylor, Alex, and Terry for helping at times and working on ideas together :) My biggest struggle in the end was the delay (I ended up shortening the size of the spectrogram that was being taken in, and might still benefit from shortening the length of the history array if I want to use it more in the future), and the reflections of the spectrograms! The placements of the spectrograms (I have four) are still buggy sometimes, but can always be placed in the correct position on start. Also, the balance of the effects was hard to strike to get the audio understandable but definitely distorted as heck (I always want MORE effects/to drown out my voice, so maybe I'll play with that going forward too, but I also really like the balance I struck). Overall, I think this Lipsythizer is a reallyyy cool tool that I can definitely see myself using in my songs/performances and I am so happy that I made it. I am proud of the work I've done and look forward to continue working with this project and tweaking it as I hopefully continue to use it going forward.
Update on learning Chunity & Ideas for Project:
Learning and implementing my new skills in Chunity is going well! The tutorials took me a while but were very helpful getting me started on these platforms. I have not used C# before, but it is quite similar to Java/JavaScript so not too tough to get the hang of. Also, I did not realize that Unity exports/launches games to play directly on your computer, or elsewhere, which is super cool and fun to experience and play with. I like the use of sound, especially with a concept like Rollaball where I feel there are certain sounds that ‘should’ be there and make sense to my brain when looking at this ball rolling around. Also, I love spectrographs and look forward to combining all of these platforms together to make a unique audiovisualization. Some ideas I have right now are: A record player, spinning, with the arm going up and down leaving a line that represents the spectrograph; Waves in the sea with a person floating on top as the waves bob them up and down either harshly or slowly; Someone’s mouth twisting into different seeming emotions as their lips twinge to represent the spectrograph.
Creating this project was a lot more difficult for me than the tutorials- but also very fun. I think just losing track of all of the moving parts was difficult, especially in the end when it came to implementing chunity aspects and having all of my objects respond to sound. I am glad that I envisioned my idea in such a way that utilizes basic shaped and a pretty straight-forward set-up, and I feel that my goals for what I am trying to do here are very achievable and that I am on a very good path where I'm at now. I really like what it looks like. I think it's chic and usable & I can imagine using it in a concert or something, talking to the audience before the performer is on stage/visible to them. I am, however, still having some issues that I would like to fix for my final deliverable. There is some lagging from the audio to the visual response. Also, I am getting errors (although all of the program executes) each time it runs that just say "UnityEngine.Debug:LogError (object) Chuck:StderrCallback (string) (at Assets/Chunity/Scripts/Chuck.cs:1751) Chuck:ManualAudioCallback (uint,single[],single[],uint) (at Assets/Chunity/Scripts/Chuck.cs:107) ChuckMainInstance:OnAudioFilterRead (single[],int) (at Assets/Chunity/Scripts/ChuckMainInstance.cs:1122)" which I still don't really know the meaning of. Also, I am interested to hear the feedback of my peers and see if it comes across to everyone as a mouth. Perhaps I should make the inner spectrograms/waveforms thicker to exentuate the speaking-nature of the mouth. I hope they get what I'm going for! I am looking forward to seeing how it goes.