Etude #3

James Zheng

One

Video
Code

Two

Video
Code

Three

Video
Code

For these Wekinator projects, I explored the capabilities of FaceOSC, VisionOSC, and Wekinator. I knew I wanted to make use of computer vision by mapping different facial expressions and hand positions to different audio. I wanted to test out what expressions Wekinator could differentiate through FaceOSC. Although FaceOSC has Pose and orientation (which direction you're facing) as datapoints, I found that Wekinator was not accurate in determining the class based on which direction my head was facing (straight, left, or right). I also tried winking with each eye, but found that Wekinator also could not differentiate this very accurately. I tried a lot of experiments such as these, and found that the mouth was the feature that Wekinator seemed to focus on the most. As I found that FaceOSC did not offer sufficient variety for Wekinator to recognize different classes, I decided to try VisionOSC, which not only offered face tracking, but also hand tracking (among others, like body, text, and even animal!). This proved to work better for me - the model was able to differentiate between hand positions much more easily (20 points per hand plus the increased dexterity are likely good reasons for this). The challenge with this was training the regression model so that I could control both pitch and volume. There seems to be limitations here, as even with extensive training, it was difficult to get the sound to be what I wanted. From these projects, I had fun with my creativity while also acknowledging the limitations of these easy-to-use tools. It makes me curious what I can do with more advanced tools and techniques.