Apr, 2021
Center for Computer Research in Music and Acoustics (CCRMA)
Stanford University

Navigate this deck with keyboard or touch

type "m" for the menu of pages and
arrow or page keys to advance or rewind

two-finger tap for the menu of pages and
the usual left/right swipe to advance or rewind

So... let's hear music

(click below) to open https://ccrma.stanford.edu/~cc/deck.js/nowNet2020/DemoLinksOnePage.html

There's magic on the horizon!

We're creatures who in our physical realms listen and function with inherent delays. We are adapted to live in lag, so to speak. Our senses of place and presence make profound use of time delays as sounds bounce around. Sounds allow us to synchronize with each other. The instruments we make for music derive their pitch from recirculating waves. We know dolphins and bats are exquisitely capable of using echoes for orientation and we are the same. I like to think Internet acoustics is the new realm into which we're expanding. Pre-COVID, that was more on the level of a thought experiment and now we're accelerating towards it.

There's magic on the horizon! (cont'd)

Packet loss concealment research to cover gaps from transmission dropouts is proving that we already have technologies which can do decent "ahead of real-time" prediction. "Pretention" (in phenomenlogy-speak) could be almost science fiction-like. But it's actually not, our brains do it, and it's a part of what philosophers and psychologists would call time consciousness. Add in that simultaneity across larger network distances can be achieved through prediction -- a possibilty which is already under investigation -- and here we go... Musicians enclosed in soundscapes that can be transmitted. Sound objects that are made of network delays.

JackTrip is a multi-machine technology which supports bi-directional flows of uncompressed audio over the internet at the lowest possible latency. Developed in the early 2000's, it was used in intercontinental telematic music concerts and a variety of musical experiments using high-speed research networks as the audio medium. Its ability to carry hundreds of channels simultaneously and its lightweight architecture led to a range of applications from IT for concert halls to small embedded systems.

JackTrip – Unlocking music performance during the lockdowns (a 1-year history)
  • metro regions ultra-low-latency (1000 km / 600 mi)
  • cloud-based hub servers
  • Raspberry Pi clients
  • Sphere of interoperability
    • JackTrip release v1.3 (Jan, 2021)
    • JackTrip Foundation (non-profit clearing house for open-source development)
    • JackTrip-WebRTC

The pandemic has ushered in a new phase of development driven by musicians seeking solutions during lockdown. Major improvements have focused on ease of use and the ability to scale across worldwide cloud infrastructure. With orchestral-sized ensembles urgently in need of ways to rehearse on the network and most participants running their systems over commodity connections, this "new reality" runs counter to what's required for ultra-low-latency rhythmic syncronization. Many developers and musical practitioners have joined in the cause of finding adequate solutions. JackTrip which has generally been run as a native software application is now complemented by dedicated solutions including numerous Raspberry Pi-based systems, standalone physical web devices, and browser-based WebRTC and Pure Data versions.

The recently established JackTrip Foundation is a non-profit clearing house for open-source development, training, and support of partners and affiliates providing their own roll-outs of the technology.

JackTrip Roadmap a Year Ago -- pre-COVID-19
(Nov 2019 to Feb 2020)
  • NowNetArts conference lab ensemble with hub mode
    Berlin / Ghent / NY / Stanford (Nov, 2019)
  • hub mode gets hubPatch added
  • 10-member Raspberry Pi ensemble, Berlin
  • Hub Mode Concert at Silent Green, Berlin (5 Feb, 2020)

JackTrip Roadmap -- COVID-19
(Mar 2020 to Apr 2021)
  • cloud-based hub servers
  • metro regions ultra-low-latency (1000 km / 600 mi)
  • solution for lessons, rehearsal, ensembles
  • large ensembles with 80 singers / instrumentalists
  • JackTrip release v1.3 (Jan, 2021)

session 1 -- intro -- 6-Apr-2020
This course inaugurates the Internet Ensemble Tech Force which is needed urgently worldwide and locally to support music ensembles going online. Calling it urgent is not an exaggeration and we can provide a valuable service. Course participants will quickly come up to speed on low-latency audio collaboration technology and will then pair with ensembles interested in using it.

153b participants will work from home and be able to use CCRMA facilities remotely. The course can be audited or coordinated with another course. Let's help make group playing possible during this public health challenge and period of 'musical distancing.' Ensemble rehearsals, coaching and concert broadcasting are planned in conjunction with several local ensembles.

Sphere of interoperability: Hub Servers
  • "always on" physical servers
    (scripted command line operation)
    • 23+ desktops running 24/7 at CCRMA
    • 6 diagnostic test servers
    • testing a Raspberry Pi (as new hub server)

23+ desktops in the building running JackTrip hub servers 24/7

Sphere of interoperability: Hub Servers
  • "always on" physical servers
    (scripted command line operation)
    • 23+ desktops running 24/7 at CCRMA
    • 7 diagnostic test servers
    • testing a Raspberry Pi (as new hub server)
  • "on demand" cloud-based servers
    (Web-based GUI operation)
    • Musicians Together Apart
    • Virtual Studio Web App

Sphere of interoperability: Clients
  • command line operation
  • GUI applications

    Qjackctl with jacktrip script option enabled
  • GUI + Raspberry Pi

    Sprawl (built-in touchscreen)
    JackStreamer (browser at Pi's wifi access point)
    Virtual Studio Device (web service)
    Embedded systems

Qjackctl & Qjacktrip interfaces (JACK & JackTrip)

session connected to hub server

JackStreamer interface served up from Raspberry Pi WiFi access point

Box32 DJ system based on Raspberry Pi

Sphere of interoperability: Other systems
  • compatible (but untested, yet)

    Quacktrip (Pd and Max/MSP)
    other Raspberry Pi projects
  • not compatible (but could be)


JackTrip-WebRTC uncompressed audio 128FPP, up to 3 clients

2020 Scientific studies
  • Mathematical models of rhythm syncronization and anticipation

    Iran Roman
    CCRMA / Stanford dissertation (Aug, 2020)
  • Synchronization of complex human networks

    Shir Shahal, Ateret Wurzberg, Inbar Sibony, Hamootal Duadi, Elad Shniderman, Daniel Weymouth, Nir Davidson, Moti Fridman
    Nature Communications (Aug, 2020)

Recent engineering improvements
  • Digital signal processing (limiters, etc.)
  • Multiple buffer sizes
  • New jitter buffer strategies
  • Adaptive queue length
  • Stream security
  • Ensemble mixing, panning, reverb
  • Persistent rehearsal setups (re-connect to room on reboot)

Engineering improvements underway
  • Packet loss concealment
  • Sample clock syncronization

Round-trip audio latency and lost packets (home to hub server and back)

(top) packet loss concealment using Burg auto-regressive prediction
(bottom) signal with packet loss

zoomed in -- packet loss concealment using Burg auto-regressive prediction

Clock skew eliminated by using GPS time for sample clock

Online rooms and auditoriums of the future
  • Binaural rooms
  • Illusory rooms
    • Haas Effect for self-monitoring
    • sympathetic strings
    • internet acoustics

free online course for background (mostly videos)

(click below) to open https://www.kadenze.com/courses/online-jamming-and-concert-technology-x/info