Difference between revisions of "GuitarFace"

From CCRMA Wiki
Jump to: navigation, search
(Assumptions)
Line 1: Line 1:
 
Welcome to the GuitarFace Wiki! Check out the GitHub repository for GuitarFace here: https://github.com/ginacollecchia/GuitarFace.
 
Welcome to the GuitarFace Wiki! Check out the GitHub repository for GuitarFace here: https://github.com/ginacollecchia/GuitarFace.
  
Why is this project called '''guitar face'''? It started with a feeling:
+
Why is this project called '''guitar face'''? It started with a feeling. Which felt a lot like this.
  
 
[[File:Brianmay_guitar_face.jpg]]
 
[[File:Brianmay_guitar_face.jpg]]
Line 27: Line 27:
 
Joe Satriani, stunned
 
Joe Satriani, stunned
  
 +
''(source for most of these images: http://www.guitarburn.com/2009/09/the-10-most-disturbing-guitar-faces/)''
  
 
'''Similarities''':  
 
'''Similarities''':  

Revision as of 00:10, 14 November 2013

Welcome to the GuitarFace Wiki! Check out the GitHub repository for GuitarFace here: https://github.com/ginacollecchia/GuitarFace.

Why is this project called guitar face? It started with a feeling. Which felt a lot like this.

Brianmay guitar face.jpg

Oh hey there Brian May! I bet you're playing some sweet jams. Here are some other amazing guitar faces. Try to notice similarities between them.

Gary Moore guitar.jpg

Gary Moore, amazing

Ritchie guitar.jpg

Ritchie Sambora, squeezing one out

Metheny guitar.jpg

Pat Metheny is trying, people.

Malmsteen guitar.jpg

Yngwie! So many nooootes.

Joe Satriani guitar.jpg

Joe Satriani, stunned

(source for most of these images: http://www.guitarburn.com/2009/09/the-10-most-disturbing-guitar-faces/)

Similarities:

  • open mouth, maybe even some tongue action, OR pursed lips
  • closed eyes OR buggy eyes
  • scrunched face OR long face
  • raised head?

With these observations, we believe there might be extractable features of a guitar face. GuitarFace seeks to pair musical data from a MIDI guitar input with computer vision in order to trigger events in a visual environment, so that guitarists (and other musicians alike) can get real time feedback of their playing, and fun rewards for their practice sessions.

Goals

  • To detect the facial features of the guitar face using OpenCV
  • To make a visualizer of musical MIDI data that records and rewards user input in real time
  • To be able to compare sessions against one another and track individual progress

Assumptions

In our design of the musical data that we'll be providing to players, we make a few assumptions about what musicians want. Our model is similar to an exercise routine, where the user could set goals / thresholds for their practice sessions. We are interested in tracking the following variables, namely in the context of solo guitar playing and improvisation:

  • duration of the session / number of notes played
  • count of pitches in the user-defined key
  • count of notes on or off the user-defined beats, e.g. tempo and meter (and an optional metronome feature)
  • count of vibratos
  • count of pitch bends
  • count of slides
  • count of power chords
  • count of musical intervals (m2, M2, P4, etc.)
  • count of "big jumps" between pitch (>P8)
  • chord labeling
  • dynamic range (moving average / smart)
  • pitch range
  • stage presence, guitar face
  • fretboard heat-map: where on the guitar neck are you playing most frequently?
  • repetition of pitch sequences
  • clarity and consistency of dynamics and intonation (streaks)
  • mixture of divisions of the beat (i.e., all quarter notes = bad)
  • pace of playing
  • extra-musical moments, such as palm mutes and artificial harmonics, if detectable
  • the distribution of these features over the specific beats of a measure

These comprise the analytic structure of our software. We can abstract musical functions such as measure and chord naming. Each of these variables will trigger an event, whether it's a histogram/graph, numerical count, graphical object (a squiggly for vibrato, changing with time, fading away completely after some time lapse), or graphical environment change. Major changes will happen when certain (imaginary or user-set) goals are reached--for example, "100 notes!"->the tunnel is blue now!->(thought bubble) whoaaa!

Storyboard

Environment.jpg

Image: One idea for the graphical environment of GuitarFace. Facial tracking shown in the top left. Moving tunnel where the present is closest to the eye and past is at the center. MIDI events shown within the tunnel: 2 squiggly lines for vibrato, a star for a recent something or other. Tunnel texture looks here a lot like an FFT waterfall, but should rather represent pitches or just be pot-smokin'. Data and records shown along the bottom (first iteration ideas). Notifications of rewards in the top right.

MIDI data is output in real time from a MIDI guitar pedal unit, the Roland GR55. The data isn't perfect, so hopefully, we won't have to spend much time or energy scrubbing the data in order to get decent, guitar-like musical data into the system. We don't need to play it back, anyways. We could also augment the USB MIDI data from the GR55 with 3 other forms of data: the audio signal from the guitar, MIDI out from the GR55, and the audio signal from the GR55, which sounds significantly better than the USB out.

There are two forms of input into the software: MIDI input, and user input. The user input consists of things like "goals" for the current session, key, BPM, metronome on/off, and more options. These data are passed through functions to compute the aforementioned measures, such as number of notes.

These functions are wired up to the graphical display and the graphical events.

Finally, a summary of the session is output and capable of being saved, for comparison with previous sessions.

Therefore, there are 4 main screens:

1. Set user options: how many notes do you want to play? Other goals? Initialization options.

2. Play! See what you're playing appear on screen!

3. End session, save report? Restart?

4. Compare session to previous sessions.

User

Musicians, specifically guitar players, interested in knowing more about their practice sessions. However, this software can be used by any instrument capable of producing MIDI output. The output can serve as a score, though we don't really think of it primarily that way; we're more interested in the statistics and progress one makes toward set goals in their practice time, such as the change in duration from session to session, or pitch and volume content.

Libraries and previous work

Relevant papers

Sensors / Accessories

  • Godin MIDI guitar
  • Roland GR55 guitar MIDI interface / pedals
  • Computer camera to detect guitar face
  • Lighting for the face?
  • Accelerometer / Game-Track to track hip gyration?

Testing

Team

  • Roshan Vidyashankar
  • Gina Collecchia

Milestones

  • Week 1: OpenCV compilation, research; most of MIDI code; graphics setup
  • Week 2: OpenCV feature detection progress; MIDI + graphics integration and design; make the thing work, basically
  • Week 3: UI/UX tweaks
  • Week 4: Code code code

Scratchwork