Audio-Driven Timing in 12 Sentiments for VR




Figure 14. Movement 7 of 12 Sentiments uses audio-driven timing to control the jumping seedling arpeggio.

The seedlings jump forward at a regular rhythm in the direction of the user's wind gestures; the tempo decreases or increases gradually depending on the wind speed.

The arpeggio of seedling notes is created using exact timing in ChucK, the audio programming language used. ChucK informs Unity that a graphics event should occur (i.e. a seedling should jump forward) after every note played.

The other way to architect this scene would have been to compute the timing in Unity using system clocks (e.g. Unity's InvokeRepeating method), and have Unity inform ChucK every time it needed to play a note. This would have resulted in jittery and unsatisfying rhythms.

(See the main page for the journal article.)