Description
This is my final project for CS 470: Music and AI. The assignment description can be found here.
Demo Video
Code and Data Files
Brief Report
This system uses a python program that is able to track the x and y positions of a yellow object and a green object in the webcam. It outputs these positions as OSC, which is then fed into Wekinator. Wekinator is trained to send 4 outputs based on this data. These outputs are read by a ChucK program that creates a theremin-like sound and a drum beat. The x and y positions of the yellow object control the pitch and gain of the theremin sound, and the x and y positions of the green object control the gains of the hihat and snare sounds.
Reflection
I was not expecting to write my reflection on this, but through the process of working on this final project, I learned a lot about the capabilities and limitations of Bing Chat and ChatGPT. For my milestone, I had used a Max/MSP example from the Wekinator website for tracking a colored object in the webcam. I needed to expand on this, which I hoped to do by adding tracking for a second colored object. However, I did not have time to figure out how to edit the example code, especially given that it was written in a programming language I was unfamiliar with and required additional downloads. I ended up asking Bing Chat to write Python code for me that would be able to track the x and y positions of two colored objects using a webcam. I was really surprised by the accuracy of the code it came up with. It was also really good at altering the code when given additional suggestions. For example, I asked Bing Chat how the code would be able to send this information to Wekinator, and it automatically filled in the parameters in the code with Wekinator’s defaults. I also asked it to open up a window with a mirrored image of what was being tracked and to display the x and y positions in the window, and it was able to do this accurately as well. On the other hand, both Bing Chat and ChatGPT were really bad at writing code in ChucK. ChatGPT would write nuanced-looking code, but it was completely incapable of writing code that was even syntactically correct and clearly seemed to be making stuff up when it came to the names of classes. Bing Chat would at least write code that would run, so it was more accurate, but it was usually extremely simple and didn’t produce sounds that sounded very good.
Acknowledgments
This example helped me make the drum beat.
Bing Chat helped me make the python code for tracking the objects using the webcam.
ChatGPT helped with writing the code for this webpage.