Inspired by recent efforts in interactive environments and game based learning I've decided to build an Interactive Music Environment that allows the user to control ChucK programs by interacting with a virtual environment. Exploration-puzzle games such as The Witness and educational games such as Variant: Limits seek to unify the interesting and entertaining aspects of virtual environments with intellectually stimulating content. I'd like to extend this toward computer music, taking advantage of the chunity plugin for Unity 3D to make a 3D environment that controls ChucK scripts.
In The Witness, the player roams an Island that is divided into many distinct and beautiful environments. In each environment the player encounters puzzles of a different theme. I was impressed by how its immersive environment was seamlessly integrated with puzzles, providing a memorable experience. One section involved the use of sounds:
In this puzzle the player hears two competing bird calls and must decide which is appropirate to focus on to select the correct path through the maze on the panel. The size of the dots on the panel correspond to the pitch of the sequence of notes in the bird's call, so a smaller dot means a higher pitched note. The game includes several variations on these kinds of interesting puzzles.
Through my project I intend to explore creating a 3D environment, navigable either on the computer or with a VR headset, that allows the user to control musical ChucK scripts by interacting with or controlling features in the environment. As I familiarize myself with Unity and reacquaint myself with ChucK I will look for interesting ways in which to link the two.
Week 2
For my initial foray into Chunity I started by following an intoductory Unity tutorial that Jack had extended to include Chunity. The tutorial has the participant create a small game in which a player rolls around a ball and picks up several objects. You can play the result of this tutorial here.
After this I embarked on getting going with Chunity. I was fairly quickly able to add basic communication between my Unity and ChucK scripts, here allowing the number of balls the player has picked up to determine the tone of the SinOsc in ChucK:
This demonstration builds on Jack's tutorial and helped me get acquainted with the basic project architecture I'll need to move forward with designing interactive environments. It uses the Chunity functionality needed to pass information back and forth between Unity and ChucK, which worked like a charm! With this under my belt I'm ready to move on to a more elaborate demo.
Looking ahead, I am considering the possibility of using VR. A key part of getting this going will be to familiarize myself with the control interfaces provided with the VR sets available at ccrma. I need to get a feeling for the various types of input coming in from the VR to start brainstorming how they could be used to control aspects of the environment that are mapped to parameters in my ChucK scripts. The next stage of the project will involve this investigation into the VR lab and creation of a more involved ChucK script... perhaps mapping a granular synth to user control of fire!
Week 3
Granular Torch
As I reflected on various ways in which ChucK could be linked to Unity, I recalled creating a granular synth in ChucK. The grains of sound overlap each other and are very short lived. While considering various models that could be implemented in Unity to mimic this behavior, the idea of a torch came to mind. One way a torch could be created in unity is using a particle emitter, where each bit of flame is a short-lived particle with its own trajectory and so on. The idea I had was that each bit of flame could correspond to a single granule of sound. If the player were to hold the torch, perhaps to a fire or near a better source of oxygen, the speed at which the grains and the flames were spawning could intensify and/or lengthen.
The design I settled on is sketched out above. I imagined a cave with a hole in the roof where smoke could escape. The player would hold a torch, moving it about the cave to produce different effects. In granular synthesis there are several common techniques to modify the grains as they are played. For example, each grain may start at a different starting point in the audio sample, may play at a different speed, go on for a unique duration and so on. (There are many other techniques that can be used here, but these are some of the basics) These parameters can then be mapped into the cave-space. The idea is that the floor of the cave would correspond to the beginning of the audio sample, while the roof of the cave would be the end. A given flame particle would start somewhere between the floor and the ceiling, corresponding to its starting position in the audio file. As it plays through the file, it would travel upward through the audio file at a speed proportional to its playback speed. In this way there would be a visual representation of the starting position, the playback speed, and the duration of each audio grain.
Week 4
With slow progress attempting to get a Granular Torch working I decided to move on to designing a more immersive environment that will allow me to get more proficient working with Unity and the Unity-Chunity combo. I settled on the idea to try to create an environment with streams of water that correspond to signals in a Chuck connection graph. In other words each stream represents a signal coming from a UGen and by interacting with the streams in the virtual environment the user can alter the audio being produced. For example, raising the water may increase the signal gain while changing the water speed may change the signal frequency. For starters I drew the following sketch:
This will require me to experiment with unity terrains, materials, and to get further into understanding how particle systems work as well as continuing to learn how Unity scripting works in general.
Week 5
River Environment v1
Looking to further familiarize myself with Unity and start exploring connecting a natural environment with a ChucK connection graph I set out to create an initial version of the river environment outlined above. Progress was slow and steady as I ran into various roadblocks while trying to implement my vision. I'm very happy with how the initial version turned out, but there's still a long way to go:
The player simply raises and lowers the water level, and correspondingly the ChucK signal, simply by pressing buttons on the keyboard. In the future the goal will be to make the player interact more directly with the environment and to hook these up with more interesting aspects of the ChucK script. However, even at this low level of complexity several issues have come up. It is readily apparent I will need a more organized script system to coordinate inputs updating multiple objects in the environment and keeping properly up to date with Chuck scripts.
Week 6
After creating the first version of the river environment, I decided that I wanted to stick with creating a natural scene that corresponded to a chuck connection graph. However, one issue that came up was mixing signals with different gain levels. If I wanted to have several voices mixed together, where each voice was represented by a stream, I would have to have the ability to independently manipulate the levels of several different streams. I decided this would be a good opportunity to get on my way writing a bit more involved unity script that could take some parameters defining where a river should be placed and then to provide a public scripting interface that could be used to control things like the water level and speed. This would continue to familiarize me with Unity and allow me more room to experiment with different chuck graphs without having to do so much work to update my scene to reflect and changes to my ChucK scripts.
Week 7
I attempted to create a unity script that could automatically generate a river network given some parameters. However, I've run into a few problems. I think the proper way to implement this component is to dynamically generate a mesh to represent the river network and to apply translations to its vertices through method calls. This involves creating a set of all the appropriate vertices and listing out all the triangles to draw using these vertices within the script. This task alone is quite challenging, but basically boils down to finding the intersection points of many different lines.
There is an additional complication beyond this of applying textures to these triangles. This is done using whats called UV-mapping, with the idea of a change of coordinates from x,y to u,v in mind. The way this works is that each vertex is assigned a point in the image space. Then for each triangle that is drawn, values are sampled for each pixel by interpolating between the mapped coordinates of each vertex. For my purposes, I would potentially want the same vertex at a bend in the river to sample from different points of the underlying texture. Here the solution is to store multiple vertices at the same point in 3d space, so that they can be assigned different UV mappings.
Yet another problem arises here and at the junctions between rivers of different heights, as shown below. Currently I am using a reflective water texture that looks very nice, but has problems at the junctions and with changes in direction. The way the reflective shader seems to work is by using a low resolution camera, perpendicular to its surface, to create the reflective texture during each frame. Yet it only runs these calculations once, so if different surfaces are using this material the reflection will only appear correct on one of them. I can make duplicates of the shader so that it calculates multiple reflections per frame, although this comes at a performance cost. However, this solution appears to have an issue where the reflections of near-vertical water surfaces seem to break and the whole material turns to white or gray.
Going forward I may opt for a more simple implementation that will run more efficiently, but be a little rough around the edges. I think my solution above is going in the right direction but I would really need more time to learn to write my own water shader that would play nice with what I am trying to do.
Week 8
Turning my attention back toward the granular torch, I've continued to debug my problems. The main issues are that the particles are simply not showing up and that the code is a bit clunky. In my current system, ChucK controls the spawning of new sound grains according to parameters passed in to Unity. Then each time Unity is going to draw a frame, it must ask ChucK for a list of all the current grains and their positions in the audio file. Using this information I then overwrite the positions of the particles using a SetParticles command. I have concerns about the amount of communication I have going back and forth here. To get information back out of ChucK into Unity I must ask for the data and then use it in a callback function that runs separately from the Update thread. There also appears to be a ChuckFloatSyncer class that may help me clean up the code here a little bit, but the general picture seems to be that it would be more simple if I could decide when to spawn grains within Unity and simply pass that information on to ChucK. The downside of this approach, however, is that the granules are limited to spawning only at times the graphics engine is actually drawing a frame.
For now I am planning on moving toward this latter approach in order to ensure I can get the torch working. Yet I think my original approach should be the correct way to implement this in the future, provided I can design a clean way of passing all the necessary information out of ChucK without bogging things down too much.
Week 9
For my final presentation I've decided to make one scene that includes the components I've been working on, as well as an additional "Rhythm Wheel" that will sit at the base of a waterfall and play sounds as the water slaps the slats on the wheel. I think a nice, small environment where each component can be seen more or less in isolation or brought together with the others will be a nice way to introduce people to the direction I've been going with Chunity.
