Reading Response #5

to Artful Design • Chapter 4: “Interface Design”

Oluseyi O.

10/29/2023

Music 256A / CS476a, Stanford University

Reading Response: Trying to Design a Fully Expressive Instrument

From this week’s reading, I’d like to respond to the theoretical “fully expressed interface’s mental model of a human'' Ge created in the chapter, while thinking about the ideas of mutualization, de-mutualization, and re-mutilization brought up in the chapter as well as in the interlude. One of the ideas that immediately stuck out to me during the interlude was focusing on creating an instrument that allows someone to learn it rather than a “smart” instrument that wants to learn the user. The example brought up is the piano, which has an identical interface whether someone is a beginner or an expert. When I see aspects of the mental model such as bio-sensing and the brain, my immediate thought was of some type of “smart” instrument that uses those sensors and the player’s brain waves to attempt to produce the sounds the user is imagining. As an aside, that might sound immediately farfetched, but it is already possible to decode brain waves to predict words someone had heard, for example. The idea of mutualization is another important one; I want this theoretical instrument to have the type of coupling of form and function that is present in acoustic instruments, even though it will be utilizing computer elements.

This theoretical instrument needs to be something that can interface with many parts of the body at the same time, and even takes in things such as skin response and temperature. To that end, it’s going to need to be a pretty complicated device, strapping in to play it will be an experience in and of itself. It might need to be something like a full body motion capture suit. For some reason, perhaps the idea of motion, I’m thinking that this instrument can also be a physical exercise in and of itself. While the typical interaction for feet and toes would be some type of keyboard, what if it was a treadmill or perhaps colored squares one had to run or jump to. A treadmill could control tempo of the music based on how fast one ran, the colored squares could be low bass pedal notes.

Let’s think about how some weirder aspects could be incorporated: the heart, brain, and eyes. When I think about the heart, I think about the heartbeat. A heartbeat could be used to control tempo similar to the treadmill. To slow the music down, someone might have to literally stop for a bit and take a deep breath. As you use the instrument, the music gets faster and faster as you exercise and increase that rate. Something like an EEG setup on the brain can be used to read brainwaves. When calibrated, these setups can track aspects such as eye movement. The direction the user is looking could change the timbre of the sound produced, this could be incorporated with icons in each direction to represent the different qualities.

Ultimately, such an instrument cannot be fully blueprinted in 500 words, but this is a start. At this current point, I’m imagining some sort of all immersive room or perhaps hamster ball coupled with a full motion bodysuit. It would certainly be an unwieldy and expensive “instrument”. “Playing” it might be a test of physical skill as well as musical skill.