Project 2: "Featured Artist"

Kunwoo Kim

Title: WTFF14 (SKETCH VIDEO)

---- Phase 1 ----

Genre Classifier

I have tested changing the following features.

1) Number of MFCCs (20 vs 40)
=> 40 seems to give worse results

2) Different FFT Window sizes (512 vs 1024 vs 2048)
=> Not much of a difference

3) Length of analysis (5, 10, 20, 30 seconds)
=> 20 gave a significantly better result than 5 or 10
=> 20 and 30 did not show too much of a difference

4) Adding ZeroX
=> ZeroX significantly worsened the result (the average went down to 0.1)

5) Adding Chroma, kurtosis, rolloff
=> They did not seem to influence the classifier too much.

6) kNN
=> 2 and 3 signifiantly improved the result.

* Best Result: 30 seconds of audio, 20 MFCCs, 2048 FFT window, kNN = 3

# of data points: 1000 dimensions: 23 
fold 0 accuracy: 0.5033
fold 1 accuracy: 0.5016
fold 2 accuracy: 0.5212
fold 3 accuracy: 0.5343
fold 4 accuracy: 0.5180

---- Phase 2 ----

I wondered what kind of music would my character in Final Fantasy 14 generate with a given input, so I decided to use my "Bard Ensemble". In particular, I used my arrangement of a song called, "Dedicated to Moonlight".

I extracted the features from "Dedicated to Moonlight - Bard Ensemble Version" and started playing with my microphone inputs. It seemed to be sensitive to different sounds such as a hiss, whistle, or singing. I tweaked around the kNN values as well. I had different sets of feature extraction, but the default sounded the best. I also played around with NUM_FRAMES and NUM_VOICES.

---- Phase 3 ----

I use Centroid, Flux, RMS, and 20 MFCCs. Also 4096 fft size and 4 NUM_FRAMES.
Currently, you can double click on the Unity build file, but there is no interaction implemented yet. The sounds will be different, but it's overall very similar to the video.

In Final Fantasy 14, I had my character play each intstrumental part in front of a wall painted in green for a Chroma Key effect. The 8 Bards are being "mosaic-ed" on top of a video that only has SFX and Voice audio turned on.

---- Acknowledgement ----
- Masayoshi Soken for composing "Dedicated to Moonlight".
- Videos are all recorded and edited by myself, but the source comes from "Final Fantasy 14".
- I used Chunity. Thank you to Andrew Zhu and Ge for creating the template!