CONTROL CAMERA with TOP, DOWN, LEFT, RIGHT keys. START NARRATIVE DEMO with SPACE KEY.
FINAL UNITY PROJECTNight Flight is an audio visualizer using the height of procedurally generated buildings to represent the spectrogram of an audio input. The intensity of building lights represents the average magnitude of an audio block. A series of waveforms appears in bewteen two buildings with the same height. The DEMO mode demonstrates how the algorithm behaves to an audio signal. Over the course of 90 seconds, the sequence plays various sounds that dynamically move along the road to give an impression of spatial audio. One of the known issues is an occasional click that can be heard even when the frame rate remanins high. I have noticed that it happens less frequently in the executable compared to the Unity editor. Despite having spent a lot of time applying long fades to each audio source, I have not found the cause of this issue. The first half of the video above showcases the DEMO mode, and the latter half demonstrates the software's behavior to a real-time audio input.
For this week's milestone 2, I spent a large amount of time prototyping a basic spectrogram using a matrix of cubes. In order for me to optimize for performance, I parameterized the resolution and depth/size of the matrix such that I can access them from Unity Editor. Once I decided to represent frequency bins and their magnitude as buildings, I converted 512 bins into a float array of the specified size with a non-linear scale. The buildings are procedurally generated with a random size and number of "windows". A user can specify the rate at which the new buildings are spawned, which along with the "scale" parameter determines the speed of their movement. Every time a new row of buildings is generated, "EM_City" class references the float array called "history" in "EM_Spectrum" class, which is updated every frame based on the FFT algorithm using Chuck.
1: My first idea for the AudioVisualizer assignment is to design a software that understands a custom programming language which allows a user to program the behavior of the visualizer based on audio parameters derived from analysis. The syntax of the language will be based on C and Java, and allow instantiation of visual class objects as well as creation of variables inside them. I will prepare base functions such as start() and update() for code update as well as utility functions such as repeat(). There will also be audio event functions such as onAmplitudeChange(). Each visual class will inherit from a parent class with functions such as setLocation(), setScale(), setMaterial() that can be called anywhere in start(), update(), or event functions.
2: My second idea for the AudioVisualizer assignment is to design a racing game in which you steer the vehicle using the pitch of an audio input. The speed is controlled by the instantaneous loudness of the audio input. In a setup where there can be more than one channel of input, participants can compete against each other using their voice. In order for the game to have a musical component, participants would have to hold an assigned musical note in each section of the circuit to keep their vehicle straight.
The Roll-a-Ball Unity tutorial was extremely helpful for figuring out basic tools, features, and functions that are available for the software. As a heavy Unreal Engine user, I took notes on similarities and differences between the two platforms. For example, collision in Unity has a simple trigger mode that only allows overlapping of objects whereas that of Unreal Engine has various settings for a user to specify which types of objects to overlap and block. The Chunity tutorial gave me a great understanding of how Chuck is designed to run inside Unity. Although I have not implemented anything practical outside the scope of tutorials, the ability to reference Chuck global variables from another Unity script seems very useful for crafting visual events based on audio behavior/parameters. Playing with sndpeek was an inspiring experience that gave me several ideas for the AudioVisualizer assignment. It was helpful to see what types of audio analysis are possible using Chuck.