Difference between revisions of "356-winter-2023/hw3"
From CCRMA Wiki
(→Getting Started with Wekinator) |
|||
(31 intermediate revisions by the same user not shown) | |||
Line 4: | Line 4: | ||
<div style="text-align: left;">[[Image:Wekinator-wavebot.jpg|400px]]</div> | <div style="text-align: left;">[[Image:Wekinator-wavebot.jpg|400px]]</div> | ||
− | In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is | + | In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is good! Playful is wonderful! |
=== Due Dates === | === Due Dates === | ||
Line 21: | Line 21: | ||
=== Tools to Play With === | === Tools to Play With === | ||
* get and play with [http://www.wekinator.org/ '''Wekinator'''] | * get and play with [http://www.wekinator.org/ '''Wekinator'''] | ||
− | * get the latest [https://ccrma.stanford.edu/courses/356/bin/chuck/ '''bleeding edge secret <code>chuck</code> build'''] (2023. | + | * get the latest [https://ccrma.stanford.edu/courses/356/bin/chuck/ '''bleeding edge secret <code>chuck</code> build'''] (2023.02.20 or later!) |
** '''macOS''' this will install both command line <code>chuck</code> and the graphical IDE miniAudicle, and replace any previous ChucK installation. | ** '''macOS''' this will install both command line <code>chuck</code> and the graphical IDE miniAudicle, and replace any previous ChucK installation. | ||
** '''Windows''' you will need to download and use the bleeding-edge command line <code>chuck</code> (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default <code>cmd</code> command prompt, or might consider downloading a [https://www.puttygen.com/windows-terminal-emulators terminal emulator]. | ** '''Windows''' you will need to download and use the bleeding-edge command line <code>chuck</code> (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default <code>cmd</code> command prompt, or might consider downloading a [https://www.puttygen.com/windows-terminal-emulators terminal emulator]. | ||
** '''Linux''' you will need to build from source, provided in the <code>linux</code> directory | ** '''Linux''' you will need to build from source, provided in the <code>linux</code> directory | ||
− | |||
* NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest [https://chuck.stanford.edu/ official ChucK release] | * NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest [https://chuck.stanford.edu/ official ChucK release] | ||
=== Getting Started with Wekinator === | === Getting Started with Wekinator === | ||
− | * Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms | + | * a terrific online course on Kadenze from Dr. Rebecca Fiebrink on interactive ML for music using Wekinator: |
− | ** | + | ** '''[https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info "Machine Learning for Musicians and Artists"]''' |
− | *** the system architecture | + | ** in particular, the videos for the following [https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v/sessions sessions] can be particularly helpful for this assignment: |
+ | *** Session 5: Sensors and Features: Generating Useful Inputs for Machine Learning | ||
+ | *** Session 6: Working with Time | ||
+ | *** Session 7: Session 7: Developing a Machine Learning Practice; Wrap-up | ||
+ | ** These videos are clear, concise, and practical! We highly recommend taking a look as you explore Wekinator and this homework! | ||
+ | ** NOTE: to view these videos, you will need to 1) sign up for a Kadenze account, 2) upgrade for the (free) 7-day premium membership trial; for anyone wanting to reference the course beyond the trial, Ge will reimbursement one-month of premium membership. | ||
+ | * Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms: | ||
+ | ** (primary approach) the '''Wekinator application (GUI)''' is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the <code>Open Sound Control (OSC)</code> communication protocol | ||
+ | *** the system architecture looks like <code>'''INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT'''</code> where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC: | ||
*** INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models | *** INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models | ||
− | *** OUTPUT: based the output parameters of Wekinator, | + | *** OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm) |
− | *** in this approach, Wekinator app's '''TRAIN mode''' iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's '''RUN mode''' predicts output based on new input | + | *** in this approach, '''Wekinator app''''s '''TRAIN mode''' iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's '''RUN mode''' predicts output based on new input |
− | ** | + | ** (secondary approach) the '''<code>Wekinator</code>''' object in <code>ChucK</code> allows you to work with Wekinator using code and without the need for network communication |
− | *** | + | *** '''<code>Wekinator</code>''' can TRAIN (from input-output observations) and/or RUN (predict output from new input) |
− | *** | + | *** or use <code>'''Wekinator'''.loadData()</code> to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved using <code>'''Wekinator'''.saveData()</code> |
− | *** | + | *** helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the '''Wekinator app''', save the data; (step 2) load the data in the '''<code>Wekinator</code>''' object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC) |
− | + | * for this assignment, we recommend trying both before deciding what to use for each of your three system designs | |
− | * for this assignment, we recommend trying | + | |
* Resources for learning and working with Wekinator | * Resources for learning and working with Wekinator | ||
** check out some [http://www.wekinator.org/example-projects/ '''example projects'''] people have created using Wekinator | ** check out some [http://www.wekinator.org/example-projects/ '''example projects'''] people have created using Wekinator | ||
Line 46: | Line 52: | ||
** get started with using [http://www.wekinator.org/walkthrough/ '''Wekinator walkthrough'''] | ** get started with using [http://www.wekinator.org/walkthrough/ '''Wekinator walkthrough'''] | ||
** for a more detailed reference, check out [http://www.wekinator.org/detailed-instructions/ '''Wekinator detailed instructions'''] | ** for a more detailed reference, check out [http://www.wekinator.org/detailed-instructions/ '''Wekinator detailed instructions'''] | ||
− | ** | + | ** example code for working with [https://ccrma.stanford.edu/courses/356/code/wekinate/app/ '''Wekinator app, OSC, and ChucK'''] |
− | + | ** example code for working with the [https://ccrma.stanford.edu/courses/356/code/wekinate/object/ '''<code>Wekinator</code> object within ChucK'''] | |
− | ** | + | ** here are a [http://www.wekinator.org/examples/ '''trove of examples'''] for working with Wekinator across many languages |
=== Your Task === | === Your Task === | ||
− | * observe and identify three activities or tasks you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small) | + | * observe and identify '''three activities or tasks''' you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small) |
+ | * of your three systems, '''at least one should be an interactive and expressive musical instrument designed with Wekinator for regression''' | ||
* experiment (a lot) with different ideas (possibly more than the three you end up with)! | * experiment (a lot) with different ideas (possibly more than the three you end up with)! | ||
− | + | * once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are virtues here | |
− | * once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are | + | * '''deploy''' these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems! |
− | * deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems! | + | |
** "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..." | ** "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..." | ||
− | * for each system, create a short video showing the system in action | + | * for each system, create a '''short video''' showing the system in action |
=== A Few Bad Ideas! === | === A Few Bad Ideas! === | ||
* Using camera input and sound output, create a day / night classifier (or sonifier)—''I mean, who has time to look out the window?'' | * Using camera input and sound output, create a day / night classifier (or sonifier)—''I mean, who has time to look out the window?'' | ||
* An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—''try playing this for your roommate, or your cat (and film the performance and audience reception)'' | * An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—''try playing this for your roommate, or your cat (and film the performance and audience reception)'' | ||
− | * using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice command system that does meaningless things—''inject a little AI-mediated chaos into your day'' | + | * using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice or gesture command system that does meaningless things—''inject a little AI-mediated chaos into your day'' |
+ | * Use Wekinator to train an instrument for controlling/exploring your audio mosaics (the output would be an N-dimensional/dimensionally-reduced vector) | ||
* for more ideas, check out [http://www.wekinator.org/example-projects/ these projects] (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)! | * for more ideas, check out [http://www.wekinator.org/example-projects/ these projects] (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)! | ||
Latest revision as of 20:33, 25 February 2023
Contents
Programming Etude #3: "Wekinate Your World"
Music and AI (Music356/CS470) | Winter 2023 | by Ge Wang
In this programming etude, you are using Wekinator to create three interactive AI utilities/toys in your everyday life! These don't have to be useful; in fact, whimsical is good! Absurd is good! Playful is wonderful!
Due Dates
- Final Deliverable: webpage due Monday (2/27, 11:59pm)
- In-class Presentation: Tuesday (2/28)
Discord Is Our Friend
- direct any questions, rumination, outputs/interesting mistakes to our class Discord
Things to Think With
- some relevant readings (including from your Critical Response #2 assignment)
Tools to Play With
- get and play with Wekinator
- get the latest bleeding edge secret
chuck
build (2023.02.20 or later!)- macOS this will install both command line
chuck
and the graphical IDE miniAudicle, and replace any previous ChucK installation. - Windows you will need to download and use the bleeding-edge command line
chuck
(for now, there is no bleeding-edge miniAudicle for Windows); can either use the defaultcmd
command prompt, or might consider downloading a terminal emulator. - Linux you will need to build from source, provided in the
linux
directory
- macOS this will install both command line
- NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest official ChucK release
Getting Started with Wekinator
- a terrific online course on Kadenze from Dr. Rebecca Fiebrink on interactive ML for music using Wekinator:
- "Machine Learning for Musicians and Artists"
- in particular, the videos for the following sessions can be particularly helpful for this assignment:
- Session 5: Sensors and Features: Generating Useful Inputs for Machine Learning
- Session 6: Working with Time
- Session 7: Session 7: Developing a Machine Learning Practice; Wrap-up
- These videos are clear, concise, and practical! We highly recommend taking a look as you explore Wekinator and this homework!
- NOTE: to view these videos, you will need to 1) sign up for a Kadenze account, 2) upgrade for the (free) 7-day premium membership trial; for anyone wanting to reference the course beyond the trial, Ge will reimbursement one-month of premium membership.
- Wekinator (and its underlying idea of creating expressive mapping interactively and by examples to continually refine the model) can be used in many different forms:
- (primary approach) the Wekinator application (GUI) is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the
Open Sound Control (OSC)
communication protocol- the system architecture looks like
INPUT >--(OSC)--> Wekinator app >--(OSC)--> OUTPUT
where INPUT and OUTPUT can be different softwares that communicate with Wekinator app using OSC: - INPUT: provides the input data (e.g., mouse locations, sensor data, gamepad/gametrak input, video frames) for training, updating, and playing with Wekinator models
- OUTPUT: based the output parameters of Wekinator, renders in real-time (e.g., sound synthesis, graphics rendering, parameters to control a game or hardware like waving a robot arm)
- in this approach, Wekinator app's TRAIN mode iterates between adding new observations (specific input to output examples) to training the model; Wekinator app's RUN mode predicts output based on new input
- the system architecture looks like
- (secondary approach) the
Wekinator
object inChucK
allows you to work with Wekinator using code and without the need for network communication-
Wekinator
can TRAIN (from input-output observations) and/or RUN (predict output from new input) - or use
Wekinator.loadData()
to load input-output observations from an ARFF file, either saved in a Wekinator project (currentData.arff) or saved usingWekinator.saveData()
- helpful in deploying a Wekinator system without using OSC; e.g., (step 1) capture observations in the Wekinator app, save the data; (step 2) load the data in the
Wekinator
object in ChucK, train, and run without OSC. Also maybe helpful for those wanting to deploy using WebChucK (which doesn't yet support OSC)
-
- (primary approach) the Wekinator application (GUI) is designed to be used in conjunction with other software (ChucK, Pure Data, Processing, Arduino etc.), connected via the
- for this assignment, we recommend trying both before deciding what to use for each of your three system designs
- Resources for learning and working with Wekinator
- check out some example projects people have created using Wekinator
- for a general overview of how Wekinator works, here are basic instructions
- get started with using Wekinator walkthrough
- for a more detailed reference, check out Wekinator detailed instructions
- example code for working with Wekinator app, OSC, and ChucK
- example code for working with the
Wekinator
object within ChucK - here are a trove of examples for working with Wekinator across many languages
Your Task
- observe and identify three activities or tasks you like to "wekinate" (i.e., to design a interactive AI system for—for this assignment, keep them small)
- of your three systems, at least one should be an interactive and expressive musical instrument designed with Wekinator for regression
- experiment (a lot) with different ideas (possibly more than the three you end up with)!
- once again, these don't have to be useful; instead, playfulness, whimsicality, and/or expressiveness are virtues here
- deploy these in some everyday context; feel free to involve those around you (e.g., roommates! friends! enemies!) in testing out your AI systems!
- "hey I have to do this thing for a class; would you kindly try/play/listen-to this thing..."
- for each system, create a short video showing the system in action
A Few Bad Ideas!
- Using camera input and sound output, create a day / night classifier (or sonifier)—I mean, who has time to look out the window?
- An instrument using one or more continuous input (such as mouse, gamepad, gametrak, Kinect, wiimote, VR controller with or without the VR) controlling a multi-parameter synthesis (like one of the STK instruments like Bowed)—try playing this for your roommate, or your cat (and film the performance and audience reception)
- using Wekinator's dynamic time warping, train a few audio or other gestures, and create a voice or gesture command system that does meaningless things—inject a little AI-mediated chaos into your day
- Use Wekinator to train an instrument for controlling/exploring your audio mosaics (the output would be an N-dimensional/dimensionally-reduced vector)
- for more ideas, check out these projects (some of these are large-scale projects; for this programming etude, remember to keep the scope of your Wekinate creations small)!
Reflections
- write ~300 words of reflection on your etude. It can be about your process or the products. Tell us about your attempt to deploy them.
Final Deliverables
- create a CCRMA webpage for this etude
- the URL should live at https://ccrma.stanford.edu/~YOURUSERID/356/hw3 or https://ccrma.stanford.edu/~YOURUSERID/470/hw3
- alternately, you may use Medium or another publishing platform (but please still link to that page from your CCRMA webpage)
- your webpage is to include
- a title and description of what you made (free free to link to this wiki page)
- three video recordings corresponding to your three interactive AI systems in action
- all relevant code and data files for all three systems
- brief report on how you created these
- your 300-word reflection
- any acknowledgements (people, code, or other things that helped you through this)
- submit to Canvas only your webpage URL