Interactive sound and haptics present unique needs, not natively supported by the game engines on which virtual worlds are typically built. The Virtual Acoustic Sound and Haptics Engine (VASHE) project from the CCRMA VR Lab aims to design a free, open-source pipeline for real-time, physically-based sound synthesis in V/AR. Among the core values of VASHE is to enable true virtual audio-haptic play (tapping, striking, scraping, throwing, etc).
A core component of modern 3D game engines is the presence of a dedicated physics module, typically for the purpose of handling rigidbodies, joints, collisions, gravity, and other physical forces. The context of processing thousands of objects (containing orders more vertices) every frame has made it necessary for these engines to optimize their performance. On the other hand, the in-built audio facilities provided by these engines are typically based on replaying sound clips, occasionally with some additional effect processing. In principle, though, it should be possible to leverage these pre-existing powerful physics engines in order to drive synthesis models in real time.
The typical game title also contains a wealth of graphical information that can be repurposed for the sake of audio and haptics. For the sake of computing real-time lighting on detailed objects, game engines make use of 2D texture atlases that allow its rendering shaders to determine the surface properties of different points in space. These maps, often in the form of normal maps, displacement maps, or height maps contain a compact topological representation, which we hope to use to synthesize audio-haptic feedback.
If you have the Anaconda distribution of Python installed, you can try out VASHE texture-to-audio synthesis by running anaconda download cfoster0/vashe_demo For a read-only preview, I have made the notebook available at this Anaconda Cloud link.
| Reference Type | Contributors | Titles | Dates | Publisher |
|---|---|---|---|---|
| 32 | Dipl-Inform Dominik, Rausch | Modal Sound Synthesis for Interactive Virtual Environments | 2017|14 December 2017 | RWTH Aachen University |
| 10 | James, Doug L.|Barbič, Jernej|Pai, Dinesh K. | Precomputed acoustic transfer: output-sensitive, accurate sound generation for geometrically complex vibration sources|ACM SIGGRAPH 2006 Papers|Precomputed acoustic transfer | 2006|2006 | ACM Press |
| 17 | Avanzini, F.|Serafin, S.|Rocchesso, D. | Interactive Simulation of rigid body interaction with friction-induced sound generation|IEEE Transactions on Speech and Audio Processing | 2005|9/2005 | |
| 10 | Zheng, Changxi|James, Doug L. | Toward high-quality modal contact sound|ACM SIGGRAPH 2011 papers | 2011|2011 | ACM Press |
| 10 | Lloyd, D. Brandon|Raghuvanshi, Nikunj|Govindaraju, Naga K. | Sound Synthesis for Impact Sounds in Video Games|I3D '11 | 2011|2011 | ACM |
| 10 | Lloyd, D. Brandon|Raghuvanshi, Nikunj|Govindaraju, Naga K. | Sound synthesis for impact sounds in video games|Symposium on Interactive 3D Graphics and Games | 2011|2011 | ACM Press |
| 17 | Li, Dingzeyu|Fei, Yun|Zheng, Changxi | Interactive Acoustic Transfer Approximation for Modal Sound|ACM Transactions on Graphics | 2015|2015-12-29 | |
| 17 | Böttcher, Niels | Current problems and future possibilities of procedural audio in computer games|Journal of Gaming & Virtual Worlds | 2013|2013-9-1 | |
| 17 | Böttcher, Niels | Current problems and future possibilities of procedural audio in computer games|Journal of Gaming & Virtual Worlds | 2013|2013-9-1 | |
| 10 | Sterling, Auston|Lin, Ming C. | Interactive modal sound synthesis using generalized proportional damping|the 20th ACM SIGGRAPH Symposium | 2016|2016 | ACM Press |
| 10 | Imran, Muhammad|Jeon, Jin Yong | Feature Based Impact Sound Synthesis of Rigid Bodies Using Linear Modal Analysis for Virtual Reality Applications|Audio Engineering Society Conference: 61st International Conference: Audio for Games | 2016|2016/02/02 | Audio Engineering Society |
| 10 | Rausch, D.|Hentschel, B.|Kuhlen, T. | Efficient modal sound synthesis on GPUs|2014 IEEE VR Workshop: Sonic Interaction in Virtual Environments (SIVE) | 2014|March 2014 | |
| 17 | Aramaki, M.|Besson, M.|Kronland-Martinet, R.|Ystad, S. | Controlling the Perceived Material in an Impact Sound Synthesizer|IEEE Transactions on Audio, Speech, and Language Processing | 2011|February 2011 | |
| 10 | Ren, Z.|Yeh, H.|Lin, M. C. | Synthesizing contact sounds between textured models|2010 IEEE Virtual Reality Conference (VR) | 2010|March 2010 | |
| 10 | Picard, Cécile|Tsingos, Nicolas|Faure, François | Audio texture synthesis for complex contact interactions|VRIPHYS 2008 - 5th Workshop On Virtual Reality Interaction and Physical Simulation | 2008|2008/11/13 | Eurographics Association |
| 17 | Raghuvanshi, N.|Lin, M. C. | Physically Based Sound Synthesis for Large-Scale Virtual Environments|IEEE Computer Graphics and Applications | 2007|January 2007 | |
| 17 | O'Brien, James F|Shen, Chen|Gatchalian, Christine M | Synthesizing Sounds from Rigid-Body Simulations | ||
| 17 | Wang, Jui-Hsien|Qu, Ante|Langlois, Timothy R.|James, Doug L. | Toward wave-based sound synthesis for computer animation|ACM Transactions on Graphics | 2018|2018-7-30 | |
| 10 | Nakatsuka, T.|Morishima, S. | Friction Sound Synthesis of Deformable Objects Based on Adhesion Theory|SCA '16 | 2016|2016 | Eurographics Association |
| 17 | Mullan, Eoin|Eng, B | Physical Modelling Sound Synthesis by Digital Waveguide Extraction with Application to Computer Games and Virtual Environments | ||
| 10 | Mullan, E. | Driving sound synthesis from a physics engine|2009 International IEEE Consumer Electronics Society's Games Innovations Conference | 2009|August 2009 | |
| 17 | Raghuvanshi, Nikunj|Lauterbach, Christian|Chandak, Anish|Manocha, Dinesh|Lin, Ming C. | Real-time sound synthesis and propagation for games|Communications of the ACM | 2007|2007-7-1 | |
| 10 | Dolphin, A. | Compositional applications of a game engine|2009 International IEEE Consumer Electronics Society's Games Innovations Conference | 2009|August 2009 | |
| 3 | Dingzeyu Li | SIGGRAPH 2016 Course: Physically Based Sound for Computer Animation and Virtual Environments|SIGGRAPH 2016 Course | ||
| 17 | Chadwick, Jeffrey N|Zheng, Changxi|James, Doug L | Precomputed Acceleration Noise for Improved Rigid-Body Sound | ||
| 17 | Chadwick, Jeffrey N|An, Steven S|James, Doug L | Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells | ||
| 17 | Michon, Romain|Martin, Sara R|Smith, Julius O | MESH2FAUST: a Modal Physical Model Generator for the Faust Programming Language – Application to Bell Modeling | ||
| 47 | Stefania Serafin|Romain Michon|Smilen Dimitrov | Blender2faust: from drawn 3d objects to physically based sound models|Blender2faust | 2018|2018-07-04 | |
| 17 | Vasil, Peter|Berlin, Technische Universität | Graphical Design of Physical Models for Real-Time Sound Synthesis | ||
| 17 | Henry, Cyrille | Mass-Spring-System model for real time expressive behaviour synthesis | ||
| 12 | Online Materials Information Resource - MatWeb | |||
| 12 | MakeItFrom.com: Material Properties Database | |||
| 12 | Find Materials & Suppliers - Matmatch | |||
| 12 | Materials Project | |||
| 17 | Provancher, William R | Creating Fully Immersive Virtual and Augmented Reality by Emulating Force Feedback with Reactive Grip™ Touch Feedback | 2014|2014 | |
| 17 | Serafin, Stefania | THE SOUND OF FRICTION: REAL-TIME MODELS, PLAYABILITY AND MUSICAL APPLICATIONS | ||
| 17 | Green, David W|Winandy, Jerrold E|Kretschmann, David E | Mechanical Properties of Wood | ||
| 17 | Campos, Guilherme|Fonseca, Nuno|Ferreira, Anibal|Davies, Matthew | Proceedings of the 21st International Conference on Digital Audio Effects (DAFx-18) | 2018|2018 | |
| 17 | Perng, Chao-Yu Jack|Smith, Julius|Rossing, Thomas | HARPSICHORD SOUND SYNTHESIS USING A PHYSICAL PLECTRUM MODEL INTERFACED WITH THE DIGITAL WAVEGUIDE | 2011|2011 | |
| 17 | Green, David W|Winandy, Jerrold E|Kretschmann, David E | Mechanical Properties of Wood | ||
| 17 | Lee, Chan-Su | Physically Accurate Haptic Rendering of Elastic Object for a Haptic Glove | ||
| 10 | Yano, H.|Igawa, H.|Kameda, T.|Muzutani, K.|Iwata, H. | AudioHaptics: audio and haptic rendering based on a physical model|12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings.|AudioHaptics | 2004|March 2004 | |
| 10 | An Example of Creative Physical Modeling: A Simulation of the Haptic Drum|An Example of Creative Physical Modeling | 2012|2012 | ||
| 17 | Berdahl, Edgar|Florens, Jean-Loup|Cadoz, Claude | USING PHYSICAL MODELS IS NECESSARY TO GUARANTEE STABILE ANALOG HAPTIC FEEDBACK FOR ANY USER AND HAPTIC DEVICE | ||
| 17 | Salisbury, K.|Conti, F.|Barbagli, F. | Survey - Haptic rendering: introductory concepts|IEEE Computer Graphics and Applications|Survey - Haptic rendering | 2004|03/2004 | |
| 17 | Berdahl, Edgar|Iii, Julius O Smith|Niemeyer, Gunter | A PHYSICALLY INTUITIVE HAPTIC DRUMSTICK | ||
| 17 | Antani, Lakulish|Chandak, Anish|Savioja, Lauri|Manocha, Dinesh | Interactive sound propagation using compact acoustic transfer operators|ACM Transactions on Graphics | 2012|2012-01-01 | |
| 10 | Nakatsuka, Takayuki|Morishima, Shigeo | Simulating the friction sounds using a friction-based adhesion theory model|20th International Conference on Digital Audio Effects, DAFx 2017 | 2017|2017 | University of Edinburgh |
| 10 | Barrass, Stephen|Adcock, Matt | Interactive Granular Synthesis of Haptic Contact Sounds|Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic, and Entertainment Audio | 2002|2002/06/01 | Audio Engineering Society |
| 17 | Barrass, Stephen|Adcock, Matt | Interactive Granular Synthesis of Haptic Contact Sounds | ||
| 12 | DEM contact models — Woo 1.0+rev4245-git-064952173 documentation | |||
| 17 | Salisbury, K.|Conti, F.|Barbagli, F. | Survey - Haptic rendering: introductory concepts|IEEE Computer Graphics and Applications|Survey - Haptic rendering | 2004|03/2004 | |
| 17 | Murphy, D.|Kelloniemi, A.|Mullen, J.|Shelley, S. | Acoustic Modeling Using the Digital Waveguide Mesh|IEEE Signal Processing Magazine | 2007|03/2007 | |
| 10 | Raghuvanshi, Nikunj|Lin, Ming C. | Interactive sound synthesis for large scale environments|the 2006 symposium | 2006|2006 | ACM Press |
| 12 | (PDF) A robust rigid body interaction model for friction-induced sound synthesis|ResearchGate | |||
| 17 | Majumdar, A.|Bhushan, B. | Fractal Model of Elastic-Plastic Contact Between Rough Surfaces|Journal of Tribology | 1991|1991/01/01 | |
| 17 | Ibrahim, R. A. | Friction-Induced Vibration, Chatter, Squeal, and Chaos—Part II: Dynamics and Modeling|Applied Mechanics Reviews|Friction-Induced Vibration, Chatter, Squeal, and Chaos—Part II | 1994|1994 | |
| 56 | Introduction to Contact Modeling|Racing Lounge | |||
| 6 | Mainsah, E.|Greenwood, J. A.|Chetwynd, D. G. | Metrology and Properties of Engineering Surfaces | 2013|2013-03-14 | Springer Science & Business Media |
| 10 | Potter, K.|Johnson, D.|Cohen, E. | Height field haptics|12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings. | 2004|2004 | IEEE |