Live Auralization of the kontakion Ho hypsōtheis en tō staurō (Lifted up on the Cross) for the Feast of the Exaltation of the Cross in Hagia Sophia

Hagia Sophia’s reverberant acoustics were integral to the experience of services in the Great Church. Based on impulse response measurements, we explore the character of this building’s soundfield and fine-tune the process, known as auralization, in which we digitally imprint Hagia Sophia’s acoustics on live chant. Whereas previous studies reported temporal and spectral features of measurements collected in the nave, in our work we concentrate on the spatial character of the soundfield by gathering directional impulse responses from both the nave and narthex.

The performance and experience of music is heavily influenced by the space in which it is heard—halls, churches, and even rooms all have their distinctive “sound.” As a listener, it is just not possible to separate a sound from the space in which it was created. This is because sound arriving at a listener’s ears comes from thousands of reflected and diffracted paths in addition to the sound directly radiated from the source. In a large, highly reflective building such as Hagia Sophia, the overwhelming majority of acoustic energy heard by a listener arrives after substantial interaction with the architectural features and objects in the space. This sound encodes the size, materials, and geometry of the space: in Hagia Sophia the listener is enveloped by waves of diffused arrivals emerging from the colonnades, sparkling reflections raining down from the dome and semi-domes, and pulses of acoustic energy from the apse and Imperial Gate. The experience of hearing chant in Hagia Sophia is almost that the building itself is singing.

Hagia Sophia is so reverberant and prolongs sound so significantly, that it shapes the way the chanters perform. Its more than 10 seconds of reverberation sets the slow tempi of the music and guides the musical phrasing as the notes linger and overlap in time. The long decay time and bright reflections from the domes create a dense resonance structure that invites the production of glittering high-frequency harmonics. Add to that the involvement of sound from the domes and colonnades, and the result is a unique sonic experience.

Near the center of the Byzantine Empire, Hagia Sophia was once the largest building in the world. Much music was naturally written to be performed in Hagia Sophia due to its religious significance in the Orthodox Church. During a large portion of the last six centuries, Hagia Sophia served as a mosque, and today it is a UNESCO world heritage site and a museum. The Byzantine chant that is the focus of our work, while meant to be performed in Hagia Sophia, cannot be sung there today. Instead, we use virtual acoustics: microphones, loudspeakers, acoustical measurements of Hagia Sophia (discussed below), and signal processing to allow Cappella Romana to record and perform this music in a simulated Hagia Sophia.

In the following, we explore the transformation of the singing voice by Hagia Sophia, study the acoustics of Hagia Sophia, and describe the technology behind our simulation of the sound of Hagia Sophia. We also point to features of Hagia Sophia’s acoustics that can be readily heard in our simulations, including in the video above. We start our exploration with the spectrogram images. The first pair of images in Fig. 1 come from the Track 1 on our “Lost Voices of Hagia Sophia” album, “Final (Teleutaion) Antiphon before the Entrance.” The deacon intones “Again and again in peace, let us pray to the Lord,” ( Ἕτι καὶ ἔτι, ἐν εἰρήνη τοῦ Κυρίου δεηθῶμεν / Héti kaì héti, en eirēnē tou Kyríou đeēthōmen) and the choir responds “Lord have mercy” (Κύριε, ἐλέησον / Kýrie, eléison). Each image displays the waveform as a white trace across the top, indicating the pressure variation over time that is sound. The lower portion of the image displays a spectrogram which shows sound energy as a function of frequency across time, where quiet signals are in black to magenta and loud parts are in yellow. The spectrogram shows the melody and harmony, much like a musical score. It also displays acoustical features of the specific performance, such as vibrato, pitch glides, and accents.

The upper image was recorded using microphones placed close to each singer’s mouth. This is to capture only the “dry” voices and not the acoustics of the room. The plot clearly shows word breaks, and an increased amplitude as the choir joins the soloist. The lower image is of the dry recording transformed by the acoustics of Hagia Sophia. It is clear that the reverberation smears the words and phrases in time making them run together and overlap. Phonemes that were distinct, each with their own pitch in time, are now blurred together and heard polyphonically.

In large, reverberant spaces, there is often a perception of motion associated with sound. This is partly a result of the reflection structure of the space, with energy coming from different surfaces about the listener arriving at different times. This effect is hinted at in the pair of images in Fig. 1. In the dry recording, the soloist’s frequency components are fairly solid in color representing relatively constant amplitude. In contrast, the corresponding frequency components in the simulated Hagia Sophia recording are more irregular and scalloped. This is because the reflections arriving at different times are mixing. When reflections overlap in time, they sometimes align and enhance each other, but other times cancel and diminish each other. This natural phenomenon of amplitude variation gives the listener a perceptual sense of movement.

To further explore how Hagia Sophia processes sound, consider a particularly simple sound called an “impulse:” a pulse so short in duration that details of its shape can’t be heard and are irrelevant. Such a pulse produced in Hagia Sophia from the ambo and heard under the edge of the main dome generates the waveform shown in Figure 2, a so- called impulse response. The impulse response is comprised of a direct path at 10 ms, a number of early reflections, including from the main dome near 300 ms, and a lingering, noise-like late-field reverberation. The associated spectrogram shows prominent reflections as vertical features, and the late-field reverberation fading over 10 seconds or so in the mid frequencies, and decaying more quickly at high frequencies.

These impulse response features also appear in the spectrograms of Fig. 1. For example, we see the mid-frequency harmonics significantly extended by Hagia Sophia’s acoustics, while the high-frequency /s/ (the consonant sound “sss”) near 11 seconds is extended by a more modest extent. To simulate the great church in our work, we measured and analyzed impulse responses between locations occupied in the church by privileged listeners and points around the ambo and along a line from the ambo to the apse, similar to where choristers, soloists, deacons, and priests would be during a service.

These measurements were taken using a spatial microphone array having four directional elements in a tetrahedral configuration. Doing so allowed us to infer the arrival direction of the reflections and reverberation impinging the array. In Fig. 3, these arrival directions are displayed in plan and elevation views as lines extending from the listener position in the direction of the arrival, with lengths indicating the energy of arrival and color-coded according to time of arrival. Note how arriving energy is first focused on the source (which is located in front and to the side), and then blooms to surround the listener, enveloping them with reflections from the colonnades and dome, the apse, and eventually the Imperial Gate.

To simulate the acoustics of Hagia Sophia, consider that the how building processes sound doesn’t depend on what the sound is. It generates the same reflections and reverberation, irrespective of what sound is produced. In this way, we can take our understanding of what Hagia Sophia does to an impulse and apply it to chanting voices.

Because Hagia Sophia’s reverberation is so rich and long lasting, and the extent to which it influences performance decisions is so great, the performers of Cappella Romana needed to record this music in the acoustics of Hagia Sophia—they needed to react to and interact with the acoustics of the space while they perform. We used close microphones to record the dry vocal signals and used fast, real-time convolution based on our acoustical measurements of Hagia Sophia to create live auralization signals. These reverberant signals are played out of loudspeakers while the ensemble performs so that performers hear themselves in the simulated Hagia Sophia. In this way, vocalists can tailor their performances to the acoustics of Hagia Sophia even in a recording studio or concert hall.

–– Adapted from Capella Romana/Icons of Sound, Lost Voices of Hagia Sophia, Naxos, 15 November 2019: BYZANTINE CHANT IN THE SIMULATED ACOUSTICS OF HAGIA SOPHIA, by Jonathan S. Abel and Elliot K. Canfield-Dafilou.