Coretet is a virtual reality instrument that explores the translation of performance gesture and mechanic from traditional bowed string instruments into an inherently non-physical implementation. Built using the Unreal Engine 4 and Pure Data, Coretet offers musicians a flexible and articulate musical instrument to play as well as a networked performance environment capable of supporting and presenting a traditional four-member string quartet. This paper discusses the technical implementation of Coretet and explores the musical and performative possibilities enabled through the translation of physical instrument design into virtual reality and realized through the composition Trois Machins de la Gráce Aimante (Coretet no. 1).
Trois Machins de la Gráce Aimante, Mvt. II concert performance September 27, 2018, Graz, Austria.
With deliberate reference to traditional stringed instrument performance practices, Coretet was designed as a futuristic '21st Century' implementation of the core gestural and interaction modalities that generate musical sound in the violin, viola and cello. By coupling the real-time fluid interactivity offered by the Unreal Engine 4 game engine with procedurally driven STK physical string models in Pure Data, Coretet offers musicians a flexible and articulate musical instrument capable of supporting improvisation as well as notated solo or ensemble performance.
Coretet also leverages the Unreal Engine's native network layer to create a networked virtual performance environment capable of supporting and presenting a traditional four-member string quartet to performers through head-mounted displays and to audiences through an auxiliary screen or projector. In a concert performance such as is utilised for Trois Machins de la Gráce Aimante this game server hosts each Coretet client instance (representing each performer) connecting across a local ethernet network. Performers in Coretet see one another in real-time with imperceptible latency within a virtual concert space, allowing for the use of communicative visual gesture both of the body and of the instrument and bow.
Fundamentally Coretet is a single instrument which can be shaped and scaled by performers into different configurations. Parameters such as neck length, body size, and number of strings can be manipulated in real-time to either recreate traditional stringed instruments such as violin, viola or cello or to create new and physically impossible instruments.
For ease of use during performance parameter presets for violin, viola and cello can be chosen and recalled instantly as can an experimental spherical instrument configuration known as the orb. Parameter values representing string length and body scale are also used to procedurally generate each instrument's sound affecting the frequency range and timbral identity of the current Coretet instrument.
Performers in Trois Machins de la Gráce Aimante are given a bow modeled after a traditional violin bow which activates the bowed string physical model when it comes into contact with a specific bowing bar on the instrument. A blue outline shown around the bowing bar indicates a collision between bow and bar, and a tracking marker indicating the collision is represented as a white sphere on the bow.
IEEE Computing Now Research Overview: The Industry Perspective: Rob Hamilton
In this month’s video, Rob Hamilton, an assistant professor of music and media at Rensselaer Polytechnic Institute, discusses connections between video games and musical instruments. His artistic pursuits have led to new perspectives on what it means to compose, play, and experience music. Rather than adapt VR technologies to emulate musical instruments, he and his collaborators have extended virtual environments with creative, interactive sonifications. Exploring these virtual spaces and objects constitutes a new kind of musical performance.
Trois Machins de la Gráce Aimante for Coretet was performed at La Friche, in Marseille, France, as part of the 2019 CMMR Conference.
Elegy (Ready, Set, Rapture) for Coretet doublebass was performed at Transitions 2019, held at Stanford University's Center for Computer Research in Music and Acoustics (CCRMA).
Trois Machins de la Gráce Aimante for Coretet was featured at a symposium of game-centric musical works hosted by Marko Ciciliani at the IEM in Graz, Austria. The piece was performed and will also serve as a "proposition" for which there will be presented two separate "responses", one artistic - in the shape of a new musical work composed by Pedro Gonzales Fernandez in response to the Coretet - and one analytic, in the form of a lecture.
Coretet was run as a demonstration and video presentation at the 2019 IEEE VR Conference in Osaka, Japan. A paper about Coretet will be presented during our workshop entitled "The Future of Audio for VR". More information @ http://favr2019.github.io.
Coretet was used in a performance with doublebassist Jeremy Baguyos at the University of Nebraska, Omaha and was presented in a lecture as part of their Music and Technology series.
Coretet was presented at the 2018 Ecos Urbanos Festival in Mexico City, Mexico. Trois Machins de la Gráce Aimante for virtual string quartet and the Coretet VR instrument was performed at the Escuela Superior de Música in the SUM (Salón de Usos Múltiples), Avenida Río Churubusco 79, Country Club, Coyoacán, Ciudad de México, at 8pm. [concert link]
Trois Machins de la Gráce Aimante for virtual string quartet and the new Coretet VR instrument was premiered at the IEM in Graz as part of the GAPPP research initiative headed up by Marko Ciciliani.
More information/video: link
Hamilton, R., "Coretet: a 21st Century Virtual Interface for Musical Expression", In Proceedings of the 14th International Symposium on Computer Music Multidisciplinary Research (CMMR), Marseille, France, 2019.
Hamilton, R., "Trois Machins de la Grace Aimante: a Virtual Reality String Quartet", In Proceedings of the International Computer Music Association Conference, New York City, NY, USA, 2019.
Hamilton, R., "Mediated Musical Interactions in Virtual Environments. In: Holland, S., Mudd T., Wilkie-McKenna K., McPherson A., Wanderley M. (Eds.) New Directions in Music and Human-Computer Interaction. Springer Series on Cultural Computing. Springer, Cham., 2019
Hamilton, R., "Collaborative and Competitive Futures for Virtual Reality Music and Sound," In Proceedings of the 2019 IEEE Virtual Reality Conference. IEEE Press, Osaka, Japan, 2019.
Hamilton, R., "Musical Sonification of Avatar Physiologies, Virtual Flight and Gesture," Lecture Notes in Computer Science: Computer Music Multidisciplinary Research (CMMR) Journal, Springer-Verlag, 2014.
Hamilton, R., "Maps and Legends: Designing FPS-based Interfaces for Multi-User Composition, Improvisation and Interactive Performance," Lecture Notes in Computer Science: Computer Music Modeling and Retrieval 2007 Journal, Springer-Verlag, 2008.
Hamilton, R., Caceres, J, Nanou, C., Platz. C, "Multi-modal musical environments for mixed-reality performance", Journal for Multimodal User Interfaces (JMUI), Vol. 4, pp. 147-156, Springer-Verlag, 2011.
Hamilton, R., Platz, C., "Gesture-based Collaborative Virtual Reality Performance in Carillon", In Proceedings of the International Computer Music Association Conference, Utrecht, Netherlands, 2016.
Hamilton, R., "Mediated Musical Interactions in Virtual Environments" In Proceedings of the ACM CHI Conference, San Jose, CA, USA, 2016.
Hamilton, R., "Measuring the Effectiveness of Sonified Crossmodal Attribute Pairings Using Contour Matching, Symmetry and Perceived Similarity" In Proceedings of the International Computer Music Association Conference, Denton, Texas, USA, 2015.
Hamilton, R., "Perceptual Coherence as an Analytic for Procedural Music and Audio Mappings in Virtual Space" In Proceedings of the 2015 IEEE Virtual Reality. IEEE Press, Arles, France, 2015.
Hamilton, R., "Designing Game-Centric Academic Curricula for Procedural Audio and Music" In Proceedings of the AES 56th International Conference: Audio for Games, London, UK, 2015.
Hamilton, R., "The Procedural Sounds and Music of ECHO::Canyon" In Proceedings of the International Computer Music Association Conference, Athens, Greece, 2014.
Hamilton, R., "Musical Sonification of Avatar Physiologies, Virtual Flight and Gesture" In Proceedings of the 10th International Symposium on Computer Music Multidisciplinary Research (CMMR), Marseille, France, 2013.
Hamilton, R., "Sonifying Game-Space Choreographies with UDKOSC" In Proceedings of the New Interfaces for Musical Expression Conference, Daejeon, Korea, 2013.
Hamilton, R., "UDKOSC: An Immersive Musical Environment" In Proceedings of the International Computer Music Association Conference, Huddersfield, UK, 2011.