356-winter-2023/hw2

Programming Project #2: "Featured Artist"

Music and AI (Music356/CS470) | Winter 2023 | by Ge Wang

In this programming project, we will learn to work with audio features for both supervised and unsupervised tasks. These include a real-time genre-classifier and a feature-based audio mosaic tool using similarity retrieval. Create a feature-driven musical statement or performance!

• Phase One (Colab + ChucK):
  • understanding audio, FFT, feature extraction
  • extract different sets of audio features from GTZAN dataset
  • run real-time classifier using different feature sets
  • run cross-validation to evaluate the quality of classifier based different features

• Phase Two:
  • curate your own set of sounds and songs
  • build a database mapping sound <=> feature vectors
  • using similarity retrieval, build a feature-based sound explorer
  • using your database and retrieval tool, design an interactive audio mosaic tool
    • real-time
    • feature-based
    • used for performance
    • takes any audio input (mic or any unit generator)
  • (optional) do this in the audiovisual domain

• Phase Three
  • use your prototype from Phase Two to create a musical statement
  • (optional) do this in the audiovisual domain

Due Dates

• Milestone 1: webpage due Monday (1/30, 11:59pm) | in-class check-in Tuesday (1/31)
• Final Deliverable: webpage due Wednesday (2/8, 11:59pm)
• In-class Presentation: Thursday (2/9)

Discord Is Our Friend

• direct any questions, rumination, outputs/interesting mistakes to our class Discord

Things to Think With

• read/skim the classic article "Musical Genre Classification of Audio Signals" (Tzanetakis and Cook, 2002)
  • don't worry about the details yet (you will get to them later); it's important to get a general sense what audio features and how they can be used

Tools to Play With

• get the latest bleeding edge secret chuck build (2023.01.23 or later!)
  • macOS this will install both command line chuck and the graphical IDE miniAudicle, and replace any previous ChucK installation.
  • Windows you will need to download and use the bleeding-edge command line chuck (for now, there is no bleeding-edge miniAudicle for Windows); can either use the default cmd command prompt, or might consider downloading a terminal emulator.
  • Linux you will need to build from source, provided in the linux directory
  • all platforms for this project, you will be using the command line version of chuck.
• NOTE: to return your chuck back to a pre-bleeding-edge state, you can always install the latest official ChucK release

GTZAN Dataset

• next, you'll need to download the GTZAN dataset
  • 1000 30-second music clips, labeled by humans into ten genre categories

HW2 Sample Code

• you can find sample code here
• start playing with these, and reading through these to get a sense of what the code is doing
  • word2vec-basic.ck -- basic example that...
    • loads a word vector
    • prints # of words and # of dimensions in the model
    • shows how to get a vector associated with a word (using getVector())
    • shows how to retrieve K most similar words using a word (using getSimilar())
    • shows how to retrieve K most similar words using a vector (using getSimilar())
    • uses the W2V helper class to evaluate an expression like "puppy - dog + cat" (using W2V.eval())
    • uses the W2V helper class to evaluate a logical analog like dog:puppy::cat:?? (using W2V.analog())
  • word2vec-prompt.ck -- interactive prompt word2vec explorer
    • this keeps a model loaded while allowing you to play with it
    • type help to get started
  • starter-prompt.ck -- minimal starter code for those wishing to include an interactive prompt in chuck, with sound
• example of poems
  • "i feel" -- a stream of unconsciousness poem (dependency: glove-wiki-gigaword-50-tsne-2 or any other model)
    • usage: chuck poem-i-feel.ck
  • "Random Walk" -- another stream of unconsciousness poem
    • usage: chuck poem-randomwalk.ck or chuck poem-randomwalk.START_WORD (to provide the starting word)
  • "Spew" -- yet another stream of unconsciousness poem
    • usage: chuck poem-spew.ck or chuck poem-spew.ck:START_WORD (to provide the starting word)
  • "Degenerate" -- a prompt-based example (run in command line chuck)
    • usage: chuck poem-ungenerate.ck

Express Yourself!

• play with the ChucK/ChAI starter/example code for Word2Vec (feel free to use any part of these as starter code)
• write code to help you create some experimental poetry involving text, sound, and time.
  • text: use the Word2Vec object in ChucK and one of the datasets to help you generate some poetry
  • sound: use sound synthesis and map the words (e.g., using their vectors to control parameters such as pitch, volume, timbre, etc.) to sound
  • time: don't forget about time! make words appear when you want them to; synchronize words with sound; visually and sonically "rap" the words in time! make a song!
• create two poetic programs / works / readings:
  • make them as different as possible
  • for example one poem can be fully generated (you only need to run the chuck code, and the poem starts) and the other one interactive (incorporates input from the user, either through a a text prompt, or another means of input such as mouse / keypresses)
• In class, we will have a poetry reading where we will run your code

Some Prompts and Bad Ideas

• a poem can be about anything; hint: try starting with how you feel about a particular thing, person, or event
• starting with an existing poem and use word2vec to morph it over time
• an experimental love poem
• stream of consciousness
• remember "Jabberwocky" by Lewis Carroll? maybe your poem doesn't need to make sense to everyone
• HINT: try to take advantage of the medium: in addition to printing out text to a terminal (both a limitation and a creative constraint/opportunity) you have control over sound and time at your disposal
• HINT: experiment with the medium to embody your message -- for example, a poem about chaos where the words gradually become disjointed and nonsensical
• HINT: can use a mixture of lines from existing poems and machine generated/processed output

Reflections

• write ~250 words of reflection on your etude. It can be about your process, or the product, or the medium, or anything else. For example, how did your poems make you feel? Did you show them to a friend? What was their reaction? What makes something "poetry" (versus, say, "prose")?

Deliverables

• create a CCRMA webpage for this etude
  • the URL should live at https://ccrma.stanford.edu/~YOURUSERID/356/etude1 or https://ccrma.stanford.edu/~YOURUSERID/470/etude1
  • alternately, you may use Medium or another publishing platform (but please still link to that page from your CCRMA webpage)
• your Etude #1 webpage is to include
  • a title and short description of the exercise (free free to link to this wiki page)
  • all relevant ChucK code for your experimental poetry tool
  • your two poems in some (video recording with audio) form; this will depend on what you chose to do; since sound and time are involved, you might include a screen capture with high-quality audio capture
  • your 250-word reflection
  • any acknowledgements (people, code, or other things that helped you through this)
• submit to Canvas only your webpage URL