Audio

With 4 narratives, at length of 9 minutes each with 3 different angles (WS, MS, and CU), along with 4 fixed audio tracks for each of the 4 narratives, there is at its base, 16 tracks of 9 minutes in length that need to synchronize and play back at PAL rate (25 fps). Additional to this is a database of nearly 60 voiceovers (short clips of audio nearly 2 seconds in length), for each of the 4 narratives that should be triggered by eye-tracking data, giving a total of 240 additional buffers of audio that can be triggered at any time. How could eye-tracking data accurately coordinate all of this data while still realising that goal within the limits of real-time CPU and GPU processing?

Figure 3: The topmost level of the audio and eye-tracking processing patch.

Figure: This shows the actual output portion of the audio. Fixed audio is on the right side while the soundclips are on the left. The signal faders' volumes are also controlled by the interpolating weights from the interpolate subpatch shown in Figure-9.