Real-time AudioVisual Composition with RayTone
Overview:
Are you interested in experimenting with sound and visuals to create an original multimedia performance? This five-day workshop offers an introduction to real-time audiovisual composition using RayTone - a node-based sequencing environment designed for artists to express their imagination in a more intuitive way.
The workshop is hosted by authors of RayTone, Eito Murakami and John Burnett, both of whom are artists/engineers in the fields of computer music and graphics. We will welcome CCRMA Associate Professor Ge Wang as a guest lecturer, an author of the audio programming language ChucK, which is utilized as the sound engine of RayTone. With ChucK embedded in RayTone, users can script behaviors of audio units that perform digital signal processing at run time. Additionally, RayTone allows users to load universal shader languages (GLSL and HLSL) for graphics programming such that audio and visual elements can be controlled simultaneously.
The workshop is structured into three categories: an introduction to RayTone sequencing, lectures on ChucK and shader programming, and activities to design original RayTone patches. In addition to learning from instructors, attendees will exchange knowledge and collaborate with each other to develop creative ideas. Upon completion of the workshop, attendees will learn to combine digital signal processing with graphics programming to perform real-time audiovisual composition. There will be an opportunity to present an original piece using RayTone on the last day of the workshop. Prior musical and programming experience may be helpful but not required.
If you have any questions, please feel free to contact Eito at eitom@ccrma.stanford.edu
Requirements:
Please bring a laptop (Windows / MacOS) and a pair of headphones to the workshop. CCRMA has a few extra pairs of headphones for those who cannot bring their own.
Schedule
Day 1: Introduction to RayTone & Digital Signal Processing
• Self introduction
• RayTone example projects
• Digital Signal Processing (DSP) with ChucK
- discrete time sampling
- additive / subtractive synthesis
- filter design
Day 2: More DSP + Shader Programming
• More DSP with ChucK
- granular synthesis
- physical modeling
• Shader programming
- HLSL examples
Day 3: More Shaders + RayTone development
• More shader programming
• Anatomy of RayTone Engine
• Feature requests
Day 4: Final project preparation
• Musical composition
• Audiovisual integration
Day 5: Presentations
• More preparation time
• Presentations by participants!
About the instructors:
Eito Murakami
Eito Murakami is a master's student at Center for Computer Research in Music and Acoustics (CCRMA) at Stanford University. He graduated from University of California San Diego with bachelor's degrees in Interdisciplinary Computing and the Arts Music (ICAM) and Political Science/International Relations. Eito is an electronic composer, performer, sound designer, and virtual reality developer. By combining his classical music training with proficiency in audio and graphics software, he creates digital interfaces and instruments that promote intuitive musical performance. Specifically, he specializes in Unreal Engine to develop audiovisual infrastructure that allows multiplayer interactions in virtual 3D environments via network. At CCRMA, his research involves designing audio playback systems in virtual reality to achieve dynamic spatial reverb.
John Burnett
John Burnett (b. 1993) is a multimedia artist based in San Diego, California. Drawing from a background in music composition, sound design, and technology, they create technologically-augmented and reactive multimedia installation works, sound and projection design for dance and theater productions, as well as concert works and film scores. John is also a member of the Sonic Arts research team, based in the Qualcomm Institute at UC San Diego, where they research audio spatialization and audiovisual technology. John is a graduate of Oberlin Conservatory and is currently a PhD Candidate at UC San Diego.
As former members of the Sonic Arts Research and Development group at UC San Diego's Qualcomm Institute, Eito and John presented a virtual reality composition titled "Becoming" at ACM SIGGRAPH 2022 - Immersive Pavilion.
Diversity in Computer Music Scholarship
A limited number of scholarships are available to promote the engagement of students from underrepresented backgrounds in the field of electronic and computer music, including gender, ethnicity, age, disability, area of study, or financial ability. Please click on the link below to learn more and apply!
https://forms.gle/qjJQwHAZtwRteCBb6
