Guitar effects control by face expressions and gestures

Music 356 Etude 3

Soohyun Kim

Demo Video with Explanation: https://youtu.be/--Lp2WFTOwU

Try Googling "guitarist face" or "guitar face". You can see tons of photos of guitarists cannot help expressing the sound of their guitar performance with their face; Their face expression reacts to their guitar performance. For instance, the most typical guitar face expression happens when they use a wah effect pedal; Guitarists including Jimmi Hendrix mimics the wah- wah- sound with their mouth shape while playing the guitar!

The idea of this project is to reverse the reaction between the guitar performance and face expression. The face expressions and gestures are detected by FaceOSC and Wekinator, and then the output system generates the guitar effects, so that guitarist can control their sound effects using their face expressions and gestures.

The conventional way to control such guitar sound effects is pressing expression pedals with a guitarist's foot. But through this project, I found that face expression and gesture detection can provide more intuitive and expressive control for guitar sound effects. I think this is because face expressions and gestures have more emotional correlation with the guitar sound than pressing pedal with a foot.

While this intuitive and expressive control gives plus points to this system as a good instrument, the drawback of this system is the stability. The control of sound effects is quite noisy which makes it hard to control smoothly sometimes (I think I just overpowered this stability issue by my 12-year guitar experience for this project.). I think I have to make better parameter smoothing process to stabilize the system.

Wah effect

adc => Gain g => ResonZ f => Envelope vol => dac;

adc => Gain g_ => dac;

MOUTH_WIDTH and MOUTH_HEIGHT control f.freq.

Pitch Shift (Vibrato & Bending)

adc => Gain g => PitShift p => Envelope vol => dac;

POSE_ORIENTATION controls p.shift.

Volume Effect

adc => Gain g => Envelope vol => dac;

POSE_ORIENTATION controls g.gain.

Acknowledgements

I started from the code face-relay.ck by Prof. Ge Wang.