Revision as of 04:19, 19 February 2023

Programming Etude #3: "Wekinate Your World"

Music and AI (Music356/CS470) | Winter 2023 | by Ge Wang

In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is probably good! Playful is wonderful!

Due Dates

• Final Deliverable: webpage due Monday (2/27, 11:59pm)
• In-class Presentation: Tuesday (2/28)

Discord Is Our Friend

• direct any questions, rumination, outputs/interesting mistakes to our class Discord

Things to Think With

• some relevant readings (including from your Critical Response #2 assignment)
  • "Power to the People: The Role of Humans in Interactive Machine Learning"
  • "Humans in the Loop: The Design of Interactive AI Systems"
  • Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance

Tools to Play With

• get and play with Wekinator
• get the latest bleeding edge secret chuck build (2023.01.23 or later!)
  • macOS this will install both command line chuck and the graphical IDE miniAudicle, and replace any previous ChucK installation.
  • Windows you will need to download and use the bleeding-edge command line chuck (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default cmd command prompt, or might consider downloading a terminal emulator.
  • Linux you will need to build from source, provided in the linux directory
  • all platforms for this project, you will be using the command line version of chuck.
• NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest official ChucK release

Getting Started with Wekinator

• Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms:
  • the Wekinator application is designed to be used in conjunction with other software (ChucK, Pure Data, etc.), connected via the Open Sound Control (OSC) communication protocol
    • the system architecture looks like INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC:
    • INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models
    • OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm)
    • in this approach, Wekinator app's TRAIN mode iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's RUN mode predicts output based on new input
  • the Wekinator object in ChucK allows you work with Wekinator without the need for network communication
    • Wekinator can TRAIN (from input-output observations) and/or RUN (predict output from new input)
    • or use Wekinator.loadData() to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved using Wekinator.saveData()
    • helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the Wekinator app, save the data; (step 2) load the data in the Wekinator object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC)
• for this assignment, we recommend trying both before deciding what to use for each of your three system designs
• Resources for learning and working with Wekinator
  • check out some example projects people have created using Wekinator
  • for a general overview of how Wekinator works, here are basic instructions
  • get started with using Wekinator walkthrough
  • for a more detailed reference, check out Wekinator detailed instructions
  • here are additional resources for working with Wekinator examples
  • sample ChucK code for Approach A (using OSC): Wekinator -> OSC -> ChucK code
  • sample ChucK code for Approach B (using Wekinator object within ChucK): Wekinator object in ChucK

Your Task

• observe and identify three activities or tasks you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small)
• experiment (a lot) with different ideas (possibly more than the three you end up with)!
• of your three systems, at least one should be an expressive musical instrument using Wekinator for regression
• once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are the virtues here
• deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems!
  • "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..."
• for each system, create a short video showing the system in action

A Few Bad Ideas!

• Using camera input and sound output, create a day / night classifier (or sonifier)—I mean, who has time to look out the window?
• An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—try playing this for your roommate, or your cat (and film the performance and audience reception)
• using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice command system that does meaningless things—inject a little AI-mediated chaos into your day
• for more ideas, check out these projects (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)!

Reflections

• write ~300 words of reflection on your etude. It can be about your process or the products. Tell us about your attempt to deploy them.

Final Deliverables

• create a CCRMA webpage for this etude
  • the URL should live at https://ccrma.stanford.edu/~YOURUSERID/356/hw3 or https://ccrma.stanford.edu/~YOURUSERID/470/hw3
  • alternately, you may use Medium or another publishing platform (but please still link to that page from your CCRMA webpage)
• your webpage is to include
  • a title and description of what you made (free free to link to this wiki page)
  • three video recordings corresponding to your three interactive AI systems in action
  • all relevant code and data files for all three systems
  • brief report on how you created these
  • your 300-word reflection
  • any acknowledgements (people, code, or other things that helped you through this)
• submit to Canvas only your webpage URL