MusicFlow

MusicFlow Preview

MusicFlow is a Realtime Music Improvisation Tool with audio visualization.
A short demo/teaser video is available here: MusicFlow

Idea and Premise

Connecting the Audio and Viusal together is always a asthetic way to desplay either music or art. I pretty enjoy the first homework to make an audio-visualizer, which shows the dynamic visual impact according to the changes of music featrues, including volume, frequency and beats, etc. Actually, when I play the piano, there is a scene in my mind sometimes because music is kind of expression, which shows what you are thinking about. Then, I am thinking about why not make a real-time audio-visualizer based on a scene while you are able to generate the music at the meantime; when you generate the music, visualizalization will be triggered.

When I was thinking about what scene can I build, the works by TeamLab caught my eyes, which is a japanese Digital Art Museum. I watched the following video and amazed by the flowing particle effect. So I decided to build a space with Flowing stars according to the music displaying. The video below is the TeamLab video gave me the inspiration.

Motivation

As I mentioned in the previous part, my main idea is to make a real-time audio-visualizer based on a scene while you are able to generate the music. Then I thought the instrumental-version karaoke with auido-visualization could be a good choice. The motivation for this is that when I play the mobile karaoke, I always think that what if there is a piano/guitar karaoke so that I can play the piano or other instruments with listening the background music. And if there could be audio-visualization, that could be awesome. So, for this project, I just help satisfy myself.

Details of the Product

1. Final Product
This is a Instrument-karaoke application with beautiful audio-visualization. You can load any your favorite music, pop, jazz or clasical as a background music, and then you can improvise the music by using the keyboard typing. At the mean time, you can enjoy the audio-visualization on the screen.

2. Interface and User guide
There are two modes in this application: a. Play Mode b. Explore Mode
In the Play Mode, three views can be chosen. The following pictures or videos show the camera setting1, setting2 and setting3. Basically, the stars are visualized by the BGM. When the BGM is faster, the stars moving forward faster, when the volume is larger, theere is glowing effect.
diagram
There are interesting modes: "black hole" and "white hole" mode.In the black hole mode, the pitch of BGM will be faster and also the view will be in the faster navigation effect. On the contract, the white mode is in low pitch. To trigger that: [Z]Black hole mode
[M]White hole mode

You can use Mouse cursot to change the view in the space. Also you can zoom in and zoom out by using the scrolling the mouse. The mouse also controls the sound panning(when your mouse curcor is going left, the paning is ranging from -1 to 1). Also, you can freeze the mouse control:
[left shift]Lock the view movement conntrol
[right shift]Lock the sound panning conntrol

diagram
In the middle of top scrren, the chords will be shown according to the timing of the background music. Then you can play improvisation based on the real-time chord. The keybord corresponds the notes:
[A]C note in 4th octave.
[W]C sharp note in 4th octave.
[S]D note in 4th octave.
[E]D sharp note in 4th octave.
[D]E note in 4th octave.
[F]F note in 4th octave.
[T]F sharp note in 4th octave.
[G]G note in 4th octave.
[Y]G sharp note in 4th octave.
[H]A note in 4th octave.
[U]A sharp note in 4th octave.
[J]B note in 4th octave.
[K]C note in 5th octave.
[O]C sharp note in 5th octave.
[L]D note in 5th octave.
[P]D sharp note in 5th octave.
[micolon]E note in 5th octave.
[Quote]F note in 5th octave.

As you play the music, the navigating direction of boat will change.
In the left corner, there is a tool bar we can control. There are three camera settings, which shows three different views of the boat navigations. Then you can also connect the microphone(you can choose the microphone you connect to the computer) to control the boat.Your voice is louder, the boat will oscillate obviously. Then you can change the Pitch, BGM volume, string volume and vocal volume.
diagram

Lets see inside the Explore Mode
There are also three camera settings. Select three seetings from the tool bar mentioned before.

Under the Hood

MusicFlow was implemented in Unity, with some of sound multipulation by the plug-in ChUnity. During the develpment, following gameobjects are essential:
GameObject FlowfieldThe flowfield model to set random particle moving direction and generate particles
GameObject AudioFlowfieldUsing FFT to extract the spectrum and volume of music and then linked to color and glowing effect flowfield particles
GameObject KeyboardGnerate the note sounds
GameObject CanvasUser Interface
GameObject Cameracan be mulnitulated by mouse and sound
For this software, the main challange is to make music-learning visualization. Need to linked the GameObject Camera, GameObject Keyboard, etc with the audio input. The sound features(amplitude and spectrum) is written in the C# file, called audio peer, which can be called in every Gameobject.

Metrics for Success

The main Metrics for Success of this product is based on the level "how users can enjoy or having fun by playing music with audio-viusalization". It is kind of hard to make every stuff in my application is followed by the music and the scene building could be more dynamic or real. I could improve more later. But at least, I have fun when i play this toy by myself and as well as my friends.