Connecting the Audio and Viusal together is always a asthetic way to desplay either music or art. I pretty enjoy the first homework to make an audio-visualizer, which shows the dynamic visual impact according to the changes of music featrues, including volume, frequency and beats, etc. Actually, when I play the piano, there is a scene in my mind sometimes because music is kind of expression, which shows what you are thinking about. Then, I am thinking about why not make a real-time audio-visualizer based on a scene while you are able to generate the music at the meantime; when you generate the music, visualizalization will be triggered.
When I was thinking about what scene can I build, the works by TeamLab caught my eyes, which is a japanese Digital Art Museum. I watched the following video and amazed by the flowing particle effect. So I decided to build a space with Flowing stars according to the music displaying. The video below is the TeamLab video gave me the inspiration.
As I mentioned in the previous part, my main idea is to make a real-time audio-visualizer based on a scene while you are able to generate the music. Then I thought the instrumental-version karaoke with auido-visualization could be a good choice. The motivation for this is that when I play the mobile karaoke, I always think that what if there is a piano/guitar karaoke so that I can play the piano or other instruments with listening the background music. And if there could be audio-visualization, that could be awesome. So, for this project, I just help satisfy myself.
1. Final Product
This is a Instrument-karaoke application with beautiful audio-visualization. You can load any your favorite music, pop, jazz or clasical as a background music, and then you can improvise the music by using the keyboard typing. At the mean time, you can enjoy the audio-visualization on the screen.
2. Interface and User guide
There are two modes in this application: a. Play Mode b. Explore Mode
In the Play Mode, three views can be chosen. The following pictures or videos show the camera setting1, setting2 and setting3. Basically, the stars are visualized by the BGM. When the BGM is faster, the stars moving forward faster, when the volume is larger, theere is glowing effect.
There are interesting modes: "black hole" and "white hole" mode.In the black hole mode, the pitch of BGM will be faster and also the view will be in the faster navigation effect. On the contract, the white mode is in low pitch. To trigger that:
[Z]
Black hole mode
[M]
White hole mode
[left shift]
Lock the view movement conntrol [right shift]
Lock the sound panning conntrol [A]
C note in 4th octave. [W]
C sharp note in 4th octave. [S]
D note in 4th octave. [E]
D sharp note in 4th octave. [D]
E note in 4th octave. [F]
F note in 4th octave. [T]
F sharp note in 4th octave. [G]
G note in 4th octave. [Y]
G sharp note in 4th octave. [H]
A note in 4th octave. [U]
A sharp note in 4th octave. [J]
B note in 4th octave. [K]
C note in 5th octave. [O]
C sharp note in 5th octave. [L]
D note in 5th octave. [P]
D sharp note in 5th octave. [micolon]
E note in 5th octave. [Quote]
F note in 5th octave. GameObject Flowfield
The flowfield model to set random particle moving direction and generate particles GameObject AudioFlowfield
Using FFT to extract the spectrum and volume of music and then linked to color and glowing effect flowfield particles GameObject Keyboard
Gnerate the note sounds GameObject Canvas
User Interface GameObject Camera
can be mulnitulated by mouse and sound GameObject Camera
, GameObject Keyboard
, etc with the audio input. The sound features(amplitude and spectrum) is written in the C# file, called audio peer, which can be called in every Gameobject.