Summary

For my final class project, I wanted to play with mappings between the visual world and the audio world. I created a prototype that played sounds as you moused over different parts of an image. I played around with different mappings, but for my demo I mapped each pixel to a single note, where hue mapped to the chroma class of the sound, saturation mapped to the brightness of the note, and value mapped to both the loudness and the octave of the note. I defined the brightness of the sound to be the amount of overtones the sound had, so as the saturation goes from 1.0 to 0.0, I used those values to control the cutoff frequency of a low-pass filter that would progressively dampen more and more of the overtones, making the sound duller.

Throughout the process, I created different prototypes to test different ideas. For example, I created a prototype called InstrumentBrightness which allowed one to control the brightness of a sound with the mouse, and I used this prototype to test how it would actually sound with the low-pass filter technique I described earlier. I also used the prototype to tune the extreme values and interpolation methods that I would ultimately use. I describe the prototypes I made below.

Download

code.zip

Each folder contains a different prototype. I apologize for the fact that the prototypes aren't documented. The two prototypes I demoed are final07 and final08.

Running the prototypes

Each folder contains a different prototype. To run any of the prototypes, simply run the Processing project with the same name as the prototype itself (if it exists) and the chuck script (if it exists).

final07*

(*This was one of the demo prototypes) The final07 prototype plays sounds as you mouse over different parts of an image

To run final07, run both the Processing project final07.pde and the Chuck script final07.ck. The a,s,d,f,g,h,j,k,l keys switch between different images, and the z,x,c keys switch between different keys (x = major key, c = minor key, z = no particular key)

final08*

(*This was one of the demo prototypes) The final08 prototype demonstrates how you can "paint" sounds by playing the greensleeves tune and allowing the user to paint on a canvas using the mouse as it's playing.

To run final08, run both the Processing project final08.pde and the Chuck script final08.ck. The song should start after 3 seconds.

final01

plays notes where pitch and loudness controlled with mouse. This prototype was used to test if osc messaging works.

final02

Convolution test. This prototype was used to see if we could rapidly process sections of the image to be translated to sound. However, this technique wasn't used in the demo prototypes.

final03

plays notes within a single octave based on the hue of pixels under the mouse.

final04

cycles through notes and visualizes the fourier transform of the notes.

final05

plays a sound at different brightnesses based on the number of harmonics present in the gen10 signal. Change the brightness by changing the numOctaves variable at the beginning of the chuck script (keep the number between 1 and 5 inclusive).

final06

plays notes and controls brightness of sound with mouse. Sound brightness controlled by gen10 signals with different numbers of harmonics.

MajorMinorColors

draws gradients of colors controlled by the mouse. This prototype was used to get a sense of which colors look like "major colors" and "minor colors" i.e. "happy colors" and "sad colors." This concept wasn't used in the demo prototypes.

MajorMinorNotes

plays notes within C major or C minor, where the notes are controlled by the mouse. To switch between major and minor press space.