Difference between revisions of "Gesture Signal Processing and Music"

From CCRMA Wiki
Jump to: navigation, search
(Created page with '== Gesture and Measuring Movement == * What is a gesture? Physical. Bodily: hands, face, posture. Non-verbal. What do gestures communicate? Is a gesture more like a button or…')
 
(Accelerometers: How to analyze data)
 
(5 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
** 3D Motion Capture using many cameras
 
** 3D Motion Capture using many cameras
 
* Goniometry: measuring joint angles with physically attached sensors
 
* Goniometry: measuring joint angles with physically attached sensors
** Laetitia Sonami's Lady's Glove:
+
** Laetitia Sonami's Lady's Glove: http://www.youtube.com/watch?v=HYTrNOmSRSo&feature=player_embedded
 +
* Electromyography: measures muscle activation by measuring electrical activity
 +
** Pamela Z's controller (which I believe uses emg): http://www.pamelaz.com/VociMov.html
 +
* With accelerometers attached to the body (e.g. wiimote, iphone)
 +
 
 +
===Really basic physics===
 +
* Position: x
 +
* Velocity: v = dx/dt
 +
* Acceleration: a = dv/dt
 +
** Proportional to force (Newton's 2nd law of motion: F = ma )
 +
* Jerk: j = da/dt
 +
 
 +
== Interlude #1: Open Sound Control ==
 +
* Open Sound Control: http://opensoundcontrol.org/
 +
* iPod Touch has 3 axis accelerometer
 +
* We will use TouchOSC to transmit accelerometer data as OSC messages over wifi to Max/Pd
 +
** http://hexler.net/software/touchosc
 +
 
 +
== Interlude #2: Filtering of Audio Signals ==
 +
* Can view any sound as being composed of sinusiodal waves at different frequencies
 +
* A filter removes energy at only certain frequencies:
 +
**A low-pass filter removes high freqs.
 +
**A high-pass filter removes low freqs. 
 +
** A band-pass only lets certain middle frequencies through.
 +
* Is differentiation a low-pass or high-pass filter?  What about integration?
 +
 
 +
== Back to movement and music ==
 +
=== Accelerometers: How to analyze data ===
 +
* How to distinguish orientation vs movement?
 +
* Can get jerk via difference (as approximation of differentiation)
 +
* How to get velocity from acceleration (or position from velocity)?
 +
** True integrator (will eventually overflow): <pre>y = y_prev + x.</pre>
 +
** Leaky integrator as approximation of integration:  <pre>y = a*y_prev + (1-a)*x</pre>
 +
* Thresholding - detecting specific events
 +
** max: past. pd: threshold
 +
* Classification through training of Machine Learning algorithms
 +
**e.g. Rebecca Fiebrink's Wekinator: http://wekinator.cs.princeton.edu/
 +
 
 +
=== Mapping to Sound ===
 +
* How do we create a musically expressive instrument from sensor data?
 +
* We need to map the data from one domain to another appropriately:
 +
** linear: to map [a b] to [c d] use <pre>z = (s-a)*(d-c)/(b-a) + c</pre>
 +
** exponential: more appropriate in many situations since perceptual qualities are often logarthmic.
 +
** any arbitrary function
 +
* max: scale, pd: ???
 +
* How do we decide the best mapping? (i.e. How do we select what physical parameter controls which musical parameter?)
 +
**this is an art, not a science!

Latest revision as of 00:43, 12 October 2009

Gesture and Measuring Movement

  • What is a gesture? Physical. Bodily: hands, face, posture. Non-verbal. What do gestures communicate? Is a gesture more like a button or a handle?
  • What is a musical gesture?

Techniques for measuring human movement

Really basic physics

  • Position: x
  • Velocity: v = dx/dt
  • Acceleration: a = dv/dt
    • Proportional to force (Newton's 2nd law of motion: F = ma )
  • Jerk: j = da/dt

Interlude #1: Open Sound Control

Interlude #2: Filtering of Audio Signals

  • Can view any sound as being composed of sinusiodal waves at different frequencies
  • A filter removes energy at only certain frequencies:
    • A low-pass filter removes high freqs.
    • A high-pass filter removes low freqs.
    • A band-pass only lets certain middle frequencies through.
  • Is differentiation a low-pass or high-pass filter? What about integration?

Back to movement and music

Accelerometers: How to analyze data

  • How to distinguish orientation vs movement?
  • Can get jerk via difference (as approximation of differentiation)
  • How to get velocity from acceleration (or position from velocity)?
    • True integrator (will eventually overflow):
      y = y_prev + x.
    • Leaky integrator as approximation of integration:
      y = a*y_prev + (1-a)*x
  • Thresholding - detecting specific events
    • max: past. pd: threshold
  • Classification through training of Machine Learning algorithms

Mapping to Sound

  • How do we create a musically expressive instrument from sensor data?
  • We need to map the data from one domain to another appropriately:
    • linear: to map [a b] to [c d] use
      z = (s-a)*(d-c)/(b-a) + c
    • exponential: more appropriate in many situations since perceptual qualities are often logarthmic.
    • any arbitrary function
  • max: scale, pd: ???
  • How do we decide the best mapping? (i.e. How do we select what physical parameter controls which musical parameter?)
    • this is an art, not a science!