| |
ABSTRACT
Data representing physiological variations measured from a musician before and during a musical performance can be used to drive the real-time algorithmic composition of a piece of music. By mapping bioinformatic data to compositional and performance choices, and by rendering and displaying these results into musical notation in real-time, a bioinformatic feedback loop can be created where musical events occuring on varying time-scales are derived from the performer's physiological reactions.
WORKFLOW
Bioinformatic sensors affixed to the skin of the musical performer generate streaming data representative of various physiological states within the body. Using a customized microprocessor, the sensor data is converted into formatted OSC packets and streamed into a computer. The OSC messages are parsed using OSC-compatible Java classes and fed into a custom designed Java-based compositional tool, capable of extracting patterns from the sensor data. Using a cell-based compositional approach, patterns can be applied to pre-composed materials on the micro and macro levels as well as to attack and relative amplitude values.
The resultant modified musical materials are rendered into traditional musical notation using the MusicXML data format and a custom designed XML parser and presented to the performer in real-time for playback, completing the feedback loop. As the performer sight-reads and performs the generated musical structures, the performer's body reacts to the musical gestures and directions on both a conscious as well as subconscious level, feeding the sensors more data.
BIOINFORMATIC SENSORS
In conjunction with the MUS 250a course "Human Controller Interface" (Verplank, Mathews) and the ongoing work with microcontrollers and sensors, an array of bioinformatic sensors are being investigated that would provide both a rich data set to drive the compositional system as well as an elegant and unobtrusive physical design that would in no way hinder a performer's ability to perform music. In addition to being relatively inexpensive and physically small, the desired sensors must be readily available in the short term for testing and integration with the Atmel 16 microprocessor.
INPUT DATA
By tracking changes in various biometric readings given by a performer during a musical performance, changes in musical performance (including changes in pitch, rhythm, amplitude/volume, attack) can be made. All readings must be taken in a non-invasive/non-distracting manner so that the act of gathering data has little to no effect on either the physiological state of the performer or on the musical performance. Possible physiological parameters to use as input include but are not limited to Body temperature, pulse rate, breathing patterns, and CO2 emission; additional parameters can be considered as the project evolves.
CELL BASED COMPOSITIONAL STRUCTURE
The cell-based compositional model proposed in this work takes composed musical cells of pitch, dynamic, and attack values and effects transformations upon these values based on a pre-defined mapping between the musical system and the output from the bioinformatic sensors. For instance, repetitive patterns found in certain bio-data, such as a steady heart-beat from an ECG reading, could be clearly mapped to ostinato patterns of a single repeating cell. Slight variations in ECG intensity or beat-separation could be mimicked by altering dynamic values for individual notes within a cell and by altering the duration and spacing of the notes relative to one-another. Viewing the same data-set from a larger time-scale, patterns of acceleration and decceleration could potentially be mapped to more or less active cells, using larger-scale patterns as determiners of phrase and even movement-level compositional activity.
One advantage of using a cell-based compositional structure over a purely algorithmically-generated structure is that performers are able to familiarize themselves with the basic harmonic and rhythmic materials making up the piece before the actual performance, greatly reducing the chances of missed notes and errors in sight-reading.
DATA/SCORE PRESENTATION
To prevent any stoppage of performance, musical sections must be rendered and presented to the performer in real-time, without any possibility that the performer will complete the currently-presented materials before the next set of materials is generated and presented. To realize these demands, a system of real-time rendering of MusicXML-encoded materials must be devised in conjunction with a single or double-page computer display capable of presenting fully-formed musical notations in a realistic size and clarity. Possible rendering methods currently include JMSL Java classes capable of rendering musical notation as well as PDF page-generation and presentation through the use of either commercial software packages such as Finale or Sibelius, or through the use of open-source notation systems like LilyPond.
|