Étude 2: Wekinate your World

Dominic DeMarco

2/27/23

Project Details

Part 1: Face Synth - with less chaos

FaceOsc + friends = fun!

Deployment Video (ft. Drive)

Watch my friend Drive play around with the system in this video! I explain the basic parameter mapping of eyebrows and jaw position to pitch and the primary lpf in the 5-parameter synth. Warning - video countains lots of loud laughter!

Emergent Properties (Phantom Faces)

FaceOsc liked finding faces in my shirts, and these faces usually tested the extremes of the parameter input space, resulting in some funny noises.

Reflections

Even though this was a technically simple project, it goes to show that basic parameter mapping can create very engaging toys. It also shows that sometimes degrees of freedom can be a curse - by having an equal number of inputs to output and not messing around with the fmosc parameters, it can provide clearer feedback to the person.

Code

Part 2: Catan Placement Strategies

Deciding where to place your starting settlements in the board game Catan can be very difficult - I've played in many games where we took 20 minutes just on placements. This wekinator-powered tool learns good placement strategies (on a scale from 0-100) from a much smaller set of examples (about 50). It then tells you the estimated Placement Value of your selected spot.

Demo Video

Reflections

I love the board game of Catan, and there is so much strategy that goes into placing the initial settlements that certainly cannot be captured by a cubic regression on 50 test inputs. Nevertheless, this does a pretty solid job capturing the structure and patterns that govern what makes a settlement spot good, and it's a reminder that you can get quite expressive results even from a simple model and limited training data. There are some limitations, namely that resources should be entered in a particular order (ore, wheat, sheep, wood/lumber, brick/clay, desert/ocean), and that certain combinations can result in bizzare responses (depending on where on the cubic equation one lands). Still, I'm excited to use this the next time I play Catan (which will be in Vegas)!

Code

Part 3: Hand Synth

I found this really cool HandPose-Osc tool on the web that can send out OSC messages for 21 control points on the hand. I ran them through a short hand-relay program that shifts the coordinate space with respect to the palm of the hand, and passes on the 21 translated coordinates to Wekinator. I then moved my hand into a bunch of different positions and generated different sounds using the 5-paramter synth, and I trained the model on about 750 examples. Training took a little while (about 2 mins) but it worked like a charm!

Demo Video

Reflections

I really enjoy playing around with the 5-parameter synth. It is deceptively expressive and can create a wide variety of sound. In class, we've talked about the power of "small data" for enabling music expressivity. In addition, I think the wacky and unpredictable behavior of the FM synth augments the power of "small data". Even simple neural network and interpolation models can create unexpected sounds, simply because it's hard to predict how interior points correlate to the output in the FM model. I'm very happy to have found the HandPose system, because I found the overall hardest part of this project to be collecting input data. When you have a face/hand recognition model to do that for you, suddenly it becomes a lot easier. The Catan project took me the longest to make end-to-end, and this took the shortest amount of time because I already had experience connecting the components together. All in all, this was a fun programming étude!.

Code