220C Progress

Weeks 1 to 3 - Idea Generation and Tinkering

I am taking 220C in conjunction with 250A (instrumental design) for my Symbolic Systems undergrad capstone. At the moment, my only plan is to spend my time designing some sort of instrument or tool.

So far in 250A, we have been studying different sensors. One of my initial ideas is to use Linear Hall Effect Sensors to build an interactive tilt sensor. This is a pretty simple idea, and so it was not hard to make a simple prototype. I used a scrap tube in the Max Lab, a magnet, a Hall Effect Sensor, and my Arduino to make a little instrument that functions like a rain stick. As the magnet gets closer to the sensor, it triggers a MIDI event, and right now, I am simply using the MIDI data to control a synth in Ableton. Here is the tube:

If I pursue this idea further, I will use acryllic to make transparent enclosures. This way, the user can see the magnet moving around. I also think it would be cool to fill the enclosures with liquid to manipulate the speed at which the magnet can move. In general, I think this tilt sensor design could lend itself to some interesting sound sculptures. I have included two sketches of a see-saw and a wind chime that could be interesting to interact with.

For the sake of 250A and this class, I went through the PRL (Product Realization Lab) saftey training. It was quite a cool experience, and honestly, I wish I would have known about the PRL sooner. For this tilt sensor design, my next steps will be to use Adobe Illustrator to design 2D cutouts for laser cutting acryllic.

Week 4 - Video Sonification

This week in 250a, we were tasked with coming up with some instrument ideas. Many of my ideas were built around my hall effect tilt sensor designs. However, one idea was different and stood out to me... I was inspired by the overhead projectors we use in class to project an image onto the screen. I thought it would be interesting to use this image data to control sound in some way. Here was my initial sketch:

I have been interested in stop motion animation for a while, and this quarter, I am in a photography class to improve my still photography skills. This idea was beginning to look like a good opportunity to combine these interests with my computer music explorations. To start, I decided to make a simple stop motion film using hand drawn figures. I was influenced by David Lynch's early film "6 Men Getting Sick 6 Times." In this early film, Lynch used multiple shots of a painting to create an abstract animation. My film is quite bad, but I have included it anyways. In the future, I will use paint, take more snapshots, and a better aspect ratio.

Here is Lynch's film (the only audio he included was a repeating siren sound):

Here is my "film" without sonification:

Week 5 - RGB to MIDI

I decided to direct most of energy to my video sonification idea. Does that mean the majority of my work from weeks 1 to 4 was a waste of time? Kind of, but not necessarily. I will continue to pursue the instrumental design projects more in 250a, but I feel like this idea is better suited for my focus in 220C. I think the creative process is rarely a linear path. For me, it often takes a while to settle on an idea. That said, now that I have a clear path for my project, I will try to lock in and make some concrete progess.

This week, I wrote a python function to convert the average RGB values of a video frame into MIDI CC values. I then used my "Hangman" and Lynch's "6 Men Getting Sick 6 Times" to create some demos of the video sonification I have in mind. First, here is hangman:

I feel like this film did not sonify well because of the lack of color saturation. Yet another reason to approach my next video differently. "6 Men Getting Sick 6 Times" sounds quite nice on the other hand. I left in Lynch's original audio. Here it is:

Week 6 - Coding Work

I primarily spent my time refactoring and improving my python code. I seperated my code into python modules. I primarily did this so that I could test the new HSV code outside the main film_to_midi.py file and consolidate global variables in a config module to be included in the other files.

The rgb_to_hsv function in color_operations.py is pretty simple. I read some articles on how hue, saturation, and value should be calculated and then tested the code to make sure it worked as desired. I also used open cv to make some simple videos to test in the main film_to_midi function.

Week 7 - Cadmium Red

This week, I made one adjustment to the python program. I mainly had to fix how I implemented the ALL_FRAMES flag feature. The MIDIUTIL add controller event function treats time as beats, not seconds. I overlooked this last week, so I had to reimplement the ALL_FRAMES feature so that the time of the MIDI file matched the time of the MP4 upon export.
More importantly, I made a video. I was unsatisfied with "hangman" because the drawing looked pretty bad and the video did not have enough frames and therefore looked choppy. To address this, I decided to do some painting. I ended up settling on a pretty narrow color pallet: cadmium red, black and white oil paints. The whole painting process took around 2 hours and consisted of 533 snapshots. Here is the initial unedited video at a frame rate of 18 fps:

This was a good start; however, I was still not satisfied because the video fit awkwardly within the frame. This was largely because the painting dimensions did not match the aspect ratio of the video. I decided to do some editing, and I came to the creative conclusion to duplicate the video and add a bit of red in between the two duplicates. While this may not be optimal, I think it looks much better. Here is the result:

I think that in the most favorable light, this piece asks the question, "when is a piece of art finished?" When I watch it back, I begin to question when I should have stopped. Next week, I will work on adding the audio through my sonification algorithm and possibly making an even better video.

Week 8 - Short Films

Week 8 already! For Cadmium Red, I mapped the MIDI CC saturation values to the gain of a waveshaper and the MIDI CC brightness values to a the frequency cutoff of a low pass filter. For the audio, I struggled to come up with a creative composition, and instead, decided to use an old song I had on my computer. The song is "I've Got Blood in My Eyes for You" by the Mississippi Sheiks. I felt that the old bluesy quality matched the organic stop motion aesthetic, and furthermore, the lyrics match the blood red colors of the painting. Also, I pitched down the recording a few semitones and compressed it to make the sound a little less harsh. Here is the video file:

Finishing up the audio demo for the Cadmium Red video did not take long, and so, I spent the bulk of the week working on my final project for Music 112 (film scoring). This assignment was great opportunity to try out my video sonification program alongside a more involved musical composition. I used the same MIDI CC mapping that I mentioned above; however, I decided to use granular synthesis to create a metallic / rough texture. I manually muted the texture at one point in the film (when it first fades to darkness as he is leaving the kitchen) because I felt the textural synth was too overpowering at this moment. I also intensified the tracking during the opening and closing credits because these frames were mostly black with only the appearance of the white text. Here is the video:

I think the result is relatively tasteful. To me, the mapping is not too obvious or on-the-nose. Instead, it follows the video just enough to compliment what is going on in the film.

Final Film - Sound of Progress

Here is my final video:

I think I excuted the idea relatively well. I am especially happy with the intro. I think I could improve my video editing skills and clean up the mix a little, but in general, I feel this film represents the culmination of the creative path I have been pursuing.