jack xiao
Note: For some reason, it works reasonably smoothly when not recording, but during screen recordings/audio recordings it occasionally lags out and clips.
the concept of this audiovisualizer is a giant, ominous cosmic clock floating in space. when running the audiovisualizer, it displays the actual real-world time of the day while also visualizing sound. the yellow second hand is the spectrum history (showing the history for a few seconds before older spectrums fade out) and the green minute hand is the waveform. i was intending to also use the hour hand to represent some aspect of the sound, but did not have enough time. some of the ticking is spatialized (source is the center of the clock) but i also included non-spatialized ticking in order to include it in the audio-visualization. the camera can move around using the WASDQE keys and the up/down arrow keys (more details provided below). i also threw in a starry/space skybox to complete the aesthetic.
on a more narrative level (after pressing 'n' to view the animation(s)), this audiovisualizer represents the inescapability of time. the cosmic clock is a large circular object in space, and in this way it resembles a black hole. i intended for the clock (while it does not look quite like the pure darkness of a black hole) to have a similar effect, slowly drawing the camera deeper and deeper in, closer to the center of the clock as time continues ticking forward without end. no matter how hard we try to escape, in the end time will always prevail and pull us into its grasp.
rotations (WASDQE)
movement (always parallel to direction you are currently facing)
narrative
as with many of my other ccrma projects, the primary theme is time (which is ironic considering that i always do things last minute and i pulled nearly 2 straight all-nighters trying to figure it all out). i think this time element works well especially since sound/music is itself grounded in time, and so having a clock that displays the real-world time while also doing the audio visualization adds another layer to the experience.
i think i am somewhat happy with the final product. it definitely achieves my aesthetic goal, and while i did not implement every feature that i wanted to have, i feel that it nonetheless still achieves what i intended for it to achieve. i did have a lot of fun playing around with it and moving the camera around after i completed the coding parts and was messing around with the build.
a lot of things went wrong, and i thought as a whole that this project was extremely challenging. primarily because i find unity to be very unwieldy and confusing, and also because unity can be slow and crashes often (not sure if it's normal, or if it's something wrong with my laptop, or if i am doing something wrong that's causing the crashes). the only thing that went well was the musical part of the narrative, since i have had experience with ChucK from MUSIC 220B and i enjoy writing music in general. however, every single part of working with unity was pretty painful in my opinion. i still have a lot to learn, and i hope that the more i use unity (and suffer) the easier it will get, but so far i haven't felt much improvement. from spending 8+ hours on camera movement and rotation to trying multiple tutorials and strategies for using line renderers instead of arrays of cubes, i was not even close to completing as much as i wish i could have. i tried implementing line renderers instead of using cubes on at least 5 different occasions, each with a different tutorial strategy, and each failed. i also struggled with latency issues (which did manifest themselves in the final product as well). i was unable to figure out a solution that preserves detail along with having good latency, so i had to settle for reducing detail and generally reducing scope in order to keep the updates relatively smooth and consistent. even then, after building and screen recording, there are still some inconsistencies and jitters here and there. i would love to know if there are other ways around this. i assume it is because i have so many cubes to draw that it is giving unity a hard time keeping up. overall, i'd say the most challenging parts of this project for me were 1. dealing with latency, 2. having good camera movement, and 3. line renderers.
i used the circle and emmission tutorials, and other than that primarily relied on unity documentation to complete the project. many things (such as line renderers and smooth camera rotation) i was unable to find tutorials or webpages that contained the information i needed, and had to resort to experimentation on my own (most of which did not work). to add a finishing touch to the project, i downloaded and used a space/stars skybox from the asset library (for the full cosmic clock aesthetic). for the music in the narrative, i used logic pro x to create some smaller clips combined with some older short sound clips (and reused some code that i had used for 220B assigmnets), but put everything together and played it using a ChucK script.
some things i wanted to implement but didn't have the time/couldn't figure out: line renderers for spectrum history, using the clock circle itself to emit spectrum history in circles (that fly outward), smooth camera movement, low latency spectrum history, hour hand that follows the envelope/intensity of the sound and reflects it via emissiveness, widening the second hand cubes based on how far from the center to make it look more smooth. Most of these ideas i discussed and had initial strategies for, but simply either ran out of time for most of them (or was unable to compelete despite extensive effort).
video link (in case embedded video above doesn't work)
while in general it was pretty fun and rewarding to get a funcional audio visualizer that (so far) is matching my aesthetic goal, unity was (and is) a bit of a pain in the a** to work with. it took me a while to get used to the UI of the unity editor and to develop a reasonable workflow. handling the scene view/game view is a bit awkward, and occasionally i would accidentally press some key and the scene view would unexpectedly shift to one that I didn't want, and it would take me several minutes t get it back to the one i wanted. while it didn't take me too long to get the pure clock elements working (watching the circle tutorial and using DateTime to get the current time-of-day information), incorporating the waveform and spectrum was trickier than expected (specifically in regads to spectrum history). this was especially challenging due to the large amount of GameObjects present, which caused a signficant amount of lag and made it difficult to develop. so, in order to make the program run reasonably, i had to substantally lower the amount of spectrum bins (128 instead of 512). i experimented with a line renderer, but found it hard to work with since they couldn't be transformed/translated/rotated as easily as an array of cubes. i might give line renderers another try, with the hopes that it will reduce the number of GameObjects and make the program less laggy.
i have all the core elements of my audio visualizer working, so for the final submission i'd like to add a few more visual elements to make the experience more interesting. more specifically, i want to add some kind of movement to the circle of the clock itself, perhaps additional spectum visualization or something related to volume. i'll also add some more objects in the background to provide more visual context to the clock (such as a distant terrain, stars, etc). maybe i'll also do something with the hour hand instead of keeping it static. i also want to play around more with colors, add to add some camera movement to view the clock from different perspectives (especially since the elements of the visualizer operate in different dimensionos).
i found the all 3 tutorials to be very engaging and fun to play around with! although the first roll-a-ball tutorial was admittedly a bit long and tedious (playing the videos at 2x speed and trying to keep up made it a bit more exciting), the result was quite worth it. i might have lost an hour or two of sleep just repeatedly rolling the ball around and collecting the rotating cubes. having previously taken music 220a and 220b, integrating ChucK into the mix (at least with what the tutorial laid out) wasn't too bad. i will have to do some experimenting with what a good workflow will be, since it seems that the majority (if not all) of ChucK integration into C# seems to be using the RunCode function with the ChucK code passed in as a string argument. that would seem a little clunky to type out directly, so maybe i'll develop the ChucK in MiniAudicle or some some other ChucK friendly code editor before pasting it into the C# scripts that are used in the Unity project.
i see a lot of overlap between the way unity works and the principles of OOP (object-oriented programming), and this makes a lot of sense since designing a 'scene' involves defining, creating, and manipulating objects. so, things like prefabs (like a class in OOP basically) and using the heirarchy (kinda like inheritance? or just generally OOP heirarchy stuff too) seemed logical and very practical. for me, the biggest challenge i think will be getting used to the intricacies of the Unity editor (which has a lot of features and buttons; i personally found it difficult to arrange the right view of the scene while designing, as the view kept shifting unpredictably). also, i'll need to spend a bit of time re-familiarizing myself ChucK in general and to get used to the syntax of C# (which i guess is similar to C and Java but with it's own unusual quirks).
at the moment, i think i would struggle trying to start and finish a project from scratch, but given some starting point with some code structure down for C# and ChucK (which are readily available just from the tutorials alone), i feel confident that i'll be able to handle the challenges that Chunity will throw at me. maybe this confidence will be short-lived, but the tutorials all made a lot of sense and i'd say that i feel somewhat comfortable with ChucK and the Unity workflow on a basic level (which should be enough for now, i hope).
while i haven't had the time to do extensive experimentation with Chunity, it was still enjoyable to mess around with the colors and numbers that were already defined in the tutorials. for the roll-a-ball project, i found that the height of the walls in the tutorial was a little too small, and thus led to the player ball ocassionally falling off the edge of the platform if it collided with the wall with too much strength. perhaps this might be a fun part of the game, but i also decided to remedy it by doubling the height of the walls to keep the player safe. it was interesting to see how a few numbers or colors drastically changed the look and feel of the resulting scene, and working on a more complex project is definitely something i'm looking forward to.
roll-a-ball scene view
playing roll-a-ball
aroll-a-ball win
falling off the roll-a-ball platform
taller walls to prevent falling
audio vizualizer
something related to nature? or like someone shaking a rope (like that rope exercise that crossfit ppl do)? or have the waves come out of pac-man's mouth? could have a sound battle between 2 sources? make it a game where u need to hit certain frequencies or amplitude?