HW3: Interactive AudioVisual Music Sequencer

Elisse Chow | October 30 — November 13, 2023

Perpetual Motion

This is Perpetual Motion, an pendulum based audio sequencer. Please watch the following video for a demo, and if possible, listen with headphones, there's a panning effect! The narrative is complete at around 2:37, everything after just shows one extra functionality and does not need to be viewed for the in-class critique. Additionally, please note that the keystrokes visualization at the bottom left is not a part of my project's funcionality, but rather the doing of a software called KeyCastr.

Instructions

To try this sequencer, please download here!

Run with: chuck run.ck

This sequencer takes in a variety of keyboard inputs as depicted in the sample video above. Please note that the controls are trigger based and should not react in any way to longer holds/presses.

+/= to add pendulums to the scene

-/_ to take away pendulums from the scene

\ to initiate the pendulum wave

BACKSPACE to delete all shown pendulums

1, 2, 3 to choose different instruments (plucked string, wurley, kick)

The currently selected pendulum will be colored white.

n to cycle through the pendulums

j to lower the pitch & symbol of the pendulums

k to raise the pitch & symbol of the pendulum

o to toggle the active state of the pendulum's sound

Reflection

To describe my feelings around this project in a few words: complicated, relieved, and curious. Complicated in the fact that I had a lot more of a design challenge with this project and the interactivity is still a little bit clunky. I think I found myself stuck in a design trap of wanting to do a lot but having to tone it down and not pursue certain ideas just because I thought I could. I think it was really a time for me to be cognizant of my capabilities with the time that I have and I think that I've turned in a relatively complete project. I'm relieved because I have a bit more freedom after this to not think about pendulums, and to not worry about it as much. As for curious, I think I'm interested in seeing if I can come back to this project way off in the future and improve upon it. There are a lot of small bugs here and there, the systems design is generally clunky, and I'd love to have encapsulated the entire swing in one class. I found myself struggling to prioritize the sound design of it all, and you can hear the awkward jumps when switching from instrument to instrument. Regardless of this all, I am definitely in a much better place mentally with this project, than I was a week ago when I felt things were a bit hopeless.

I think one thing I found a bit frustrating though likely better in the end was the limitation of only having 12 point lights in a scene. I was trying to have around 16 pendulums and was super confused why the last four weren't lighting up. In retrospect, I think it was better to keep it more simple to allow the user to not get even more overwhelmed in regards to the timing of the sound bits.

To speak a bit on the design of the project, the code is extremely messy. But it's generally broken down into several big classes. Each pendulum is it's own GGen, containing a Lightbulb (from light.ck), and the string connecting to (0, 0). It also contains all the state information of whether it's playing, it's inactive, it's selected, etc. Based on a parameter, it will use one of the three instruments it also contains, either a plucked string (chugraph.ck), a wurley, or a kick drum (thanks Tristan!). These pendulums are at the heart of everything. At the beginning, I start with a list of 8 of them but I don't gruck them into the scene until I increment the num_showing using the "+"/"=" key. Using other keyboard controls, I can change the instrument, the note playing, which pendulum I am editing, and the status of the pendulum (if it's allowed to play or not). When the pendulum swings to an area near the maximum amplitude, a note gets played on the instrument. The gain and the panning are also proportional to the current theta.

I think I wanted a non-linear sequencer that could create interesting bits of music, and while it's not always pretty, it creates some fun sounds altogether. I think I'm proud-ish of the work I've done for this project. Not outstandingly proud, but just enough that I don't feel bad about submitting this as the final deliverable.

---

Milestone 2

I say it in the video as well, but I will write here that I'm most definitely not satisfied or really happy with the current state of my work. It doesn't feel like it's a sequencer at this point, the interactivity is relying entirely on the UI, and there are massive bugs in my pendulum simulation. It's barely aesthetically pleasing and it's really a jumbled up skeleton at this point. I'm definitely an example of someone who ran out of time, and my ideas in milestone 1 were either not working as I'd hoped or just too daunting. I hope I'll have something much better for the final deliverable.

To run this project, please download the files here, and run the files with chuck run.ck.



In this next week, I don't want to abandon this idea or this project. But I think it might be possible that this whole idea might take huge shift, and I want to be transparent with that. But to continue with this idea, I think I'd like to make the interaction a lot more streamlined and smoother. The orientation I've placed the camera doesn't exactly lend to a pleasant interaction experience, and I'd like to make it more clear what notes are playing, what things are being interacted with, and how the lengths of the pendulum can impact everything. It's also extremely one dimensional in the sound producing capabilities at the moment, where the notes go off when the pendulum swings back to the other direction. There's only those strings, and there's not a lot you can do in terms of making this act as a "sequencer".

---

Milestone 1

Research + Links

Step Sequencing What it is and how to use it: General intro to step sequencers out there physically and digitally.
Creating a Step Sequencer with Tone.js: Super interesting and gives all of the code they did to build this out in Tone.js. It's fun to play with the sequencer at the bottom of the article.
MIDI controllers, Step Sequencers and Arpeggiators - Sound Design Theory: Informative, and helpful to understand some things.
Designing a simple 5-step sequencer from scratch: Did not watch this through, but saving it for the future.
Learn Push 2: Step Sequencing Beats: Fun to watch and understand how a physical step sequencer works and how all the functionality is packed into so many things. Each button has at least a dual function.

for each of these ideas, I really should have sketched a bit more, but hopefully they still read

Idea #1

For my first idea, I think I was playing around with the basic idea of the step sequencer with three tracks and 8-12 steps set up in a linear manner. Successive interactions with a step on each track would change and map to either different instruments or notes (unsure exactly how this will work). Then as the time bar passes by, the colors on all tracks would somehow combine into the visualization and throw a multicolored paint ball at a white canvas in the distance. This way the history of the entire track as users edit it would be recorded in the canvas image. Input from the user on each of the steps might vary between mouse clicks or keyboard controls (I think the goal is to figure out an intuitive way to get this working and go from there). The loudness of each step might be shown in the tracks by how saturated the color of the step is maybe? Then it's sort of that the step has a lesser or greater contribution to the paintball being created.

Idea #2

I think out of my ideas, I find this the most compelling, though I'm still unsure how to craft the sound design for it and there are a lot of other aspects of the design that are still to be decided. But the essential main factor is, fairy ring and mushrooms! Each ring of mushrooms would correspond to a different layer (ie first layer synths, second layer drums, third layer twinkles?). It would be maybe 16 steps and the size of the mushroom/other aspects such as color would map to volume and pitch. My goal with this is to create pretty, airy music that gives off a magical feeling. Each point on the ring would be selectable with the mouse and mushroom aspects would be able to change in either a GUI on the side or through keyboard controls. The user may add as many tracks as they want to layer extra synths or drums or whatever they prefer. Though, maybe instead of extra layers they'd be able to add other rings in the space with different number of steps.

Idea #3

I'll be honest and say I'm really confused about how this third idea might work. It's a lot more on the experimental side of things and while I love fractals and l-systems, I'm having a hard time trying to translate this into a fullblown sequencer design. I know that if I go with this idea, I want to work in Lindenmayer Systems and Fractals of different types.

Each fractal/l-system would be interactive and choices on earlier generations would propogate to further generations as seen on that sierpinski triangle. As for how this is going to all map to sound, I'm very quite confused. I have found a paper on the musical interpretations of l-systems but that's a bit different from how I understand a sequencer might work. I would love to work in more beautiful math into my designs, but I'm not seeing a way for this at the moment unfortunately.