Kandinsky Sonified

Wassily Kandinsky would wanna play with this. Greatly inspired by his aesthetics and synaesthesia on abstract colors/shapes, this is a designing trial to combine the graphics with the acoustics in an interactive way.

Each shape has a timbre, and each color is mapped to a pitch frequency. The volumn of the shape determines the loudness (and certain effects behind). The position determines the panning (audio image).

The interface is pretty intuitive. You can draw squares, lines, and circles by first selecting the corresponding shapes in the toolbar. The color picker is designed to generate a random color with each click. And the eraser can erase a shape by clicking on it. Further interactions include

• Scroll mouse wheel to adjust the playback speed. You can even go reverse!
• Right click switches the sweeping axis.
• When no tool is selected (click again to de-select one), left click on the canvas moves the play head to the mouse position (You can potentially DJ via this).

Codes can be accessed here. Make sure to install the latest ChucK and ChuGL versions. Run the code via chuck kandinsky.ck.

Stories & Details

I love my way towards the final design! (final as for deadline, not final as a design). After the mechanism was established at the milestone, I spent quite some time refactoring my code, puting it into multiple files using the brand new import system. More tweaks happened in visual hints and timbre selection.

Architecture:

• constant.ck: storing constant values across files.
• draw.ck: toolbar design, tool selection, drawing mechanism.
• mouse.ck: mouse class.
• play.ck: timbre design, acoustic system.
• shapes.ck: shapes, playing mechanism.
• kandinsky.ck: main file.

Thanks Kunwoo and Andrew for their support!

Milestone B

Milestone A

Research

Some instant memories came to my mind while Kunwoo was showing this galaxy of sequencer designs. I'm obsessed with an electronic music duo called The Chemical Brothers. Their music videos are always filled with artful surprises. I recalled the music video for Star Guitar:

This is undoubtedly a sequencer! It's such a brilliant idea that the director achieved audio-visual coherence via arranging a crafted reality that synchronizes with the track, and this was made in 2009! A myriads of music games later extend this idea to make the synchronization interactive.

I also recalled a fantastic house music piece called i heard you like polyrhythms by VirtualRiot. Some talented people created a visualization that is insanely brain-satisfying.

Following my research (wandering) of audiovisual music, I discovered a whole new world of Oscilloscope Music. These videos are probably the coolest minimalistic audiovisual pieces I have encountered.

My idea gradually forms: I want to craft a sequencer that exposes matching between the audio and the visual. I have dreamed about this since I read Kandinsky's Point and Line to Plane, where he envisioned potential researches for image-emotion and image-music correlation, starting from basic geometries and their semantic interpretations when juxtaposed with other elements.

My Design: Kandinsky Sonified

Why cannot the sequencer be an abstract painting?

Other Alternatives

Conway's Game of Life Sequencer