SoundPaint is an audiovisual creation/composition tool. It merges a basic painting application with an additive synthesis / live-input controlled sound generator.
I've always wondered why there aren't more digital art mediums/tools that merge audio and visual, other than video. In those that do exist, it seems like one of the two components is clearly secondary to the other. For example, in film, audio is secondary to visuals; in music videos, visuals are secondary to audio. I have yet to really see a digital art form that treats the two as equals. I hypothesize that a prominent reason for this is the difficulty in composing both at the same time; it is often the case that one must necessarily come before the other. Additionally, there is no obvious direct correlation/mapping between the two. Using one to control the other would be an obvious way to compose both simultaneously, but a good method for doing so is not so obvious. This unity is what I sought to explore through the creation of SoundPaint.
My main goal in the design of SoundPaint was to provide a balance between dependence and independence of the audio and visual components upon one another. I wanted to allow composers/artists to control the two individual or simultaneously based on their preference. Here, I will discuss the design of the two components separately, and then discuss how they are merged to meet this design criteria.
The sound design is really an exercise in additive synthesis and FFT analysis. All sound input is taken in through the microphone and analyzed using an FFT. I thought this would be the simplest, most effective way for sound input, as any sound source can theoretically be used and it does not require any specialized musical knowledge. Each of the points drawn contains one sinusoidal oscillator. The gains and frequencies of these oscillators are determined by the spectrum of the input sound and the additive synthesis algorithm being used. Thus, sounding the oscillators in different quantities and at different speeds will produce different sounds. This dynamism is something else I sought to produce in the design.
The visual design is inspired in part by pointillism and in part by the spray can in Microsoft Paint. I tried to strike a balance between control and chaos in this design. Users can adjust the average size of the points draw, the area in which they are sprayed, and the number of points sprayed at once. However, the actual size of points is determined by the FFT magnitudes, and much like an actual spray paint can, the location of the points within the spray circle is random. In addition, there are a limited number of points that can be drawn in a painting; this was a creative limitation as well as a practical one.
The most important design feature is the playback mode, as it is what allows for the separation/combination dynamic of the audio and visual elements. In normal draw mode, users can choose whether or not they want to record sound as they draw. In playback mode, they can listen back to what they've recorded, record over recorded or unrecorded points, and delete sections of the painting. This ability to manipulate the painting and create the audio and visuals separately but simultaneously is the feature that most embodies my goals for the design of SoundPaint.
In draw mode, click and drag the mouse to draw points. Changing the point size will change the average size of the points drawn and changing the spray radius will change the area in which points are drawn. If record mode is enabled, sound taken from the microphone is used to generate sound using a fast-fourier transform and additive synthesis. Three synthesis modes are available.
FFT sines uses the FFT peaks to resynthesize the sound taken from the microphone.
Saw generate square and saw waves, respectively, using the largest FFT peak as the fundamental frequency. The
# FFT bins/harmonics value determines how many peaks or harmonics are recorded. If record mode is not enabled, hollow points will be drawn but will contain no sounds. Playback mode allows the composer to scrub chronologically through the painting. This can be used to erase parts of the painting or to play the painting. Changing the play speed will change the rate at which the painting can be scrubbed through or played. In addition, with record mode enabled, sound can be recorded over previously recorded points or unrecorded points. Use with caution, as this is permanent! In neutral mode (draw/record/playback modes all off), move the mouse over different sections of the painting to listen to them. Change the size of the mouse head to change the radius of the listening area. Paintings can be saved and loaded as .dat files.