PID2008 Lab 4

Lab 4: Accelerometers and Gesture Tracking

For this lab you need a 3D accelerometer board, plus a cable to get from that board to the PORTC header of your AVRmini.

Setup

• Save lab5.tar.gz into your personal ('clarnece') directory.

            ~> cd ~/clarence
   ~/clarence> tar xzf lab5.tar.gz

Accelerometer Basics

• Connect your 3D acclerometer board to PORTC of your AVRmini board, making sure that you orient the ribbon cable consistently between the two ports (so that pin 1 connects to pin 1, pin 2 to pin 2, etc., instead of backwards).

• Compile and load lis3l02osc.c in the lis3l02osc directory

                   ~/clarence> cd lab4/lis3l02osc
   ~/clarence/lab4/lis3l02osc> make load

• Open simple-view-accel.pd (in ~/clarence/lab4/pd/) and make sure you're getting incoming X, Y, and Z acceleration values.

• What are the "default" acceleration values when the accelerometer board isn't moving?

• Turn the accelerometer upside down. Now what values do you get?

• Shake the accelerometer back and forth and see what numbers appear in the numbers boxes.

• Note how unsatisfactory it is to look at the output of the accelerometers just with number boxes, seeing only the current values.

Accelerometer Plotting

• To get a better sense visually of what's coming out of the 3D accelerometer, we're going to plot the three acceleration values as functions of time.

• Open accelplot.pd and don't be afraid. Ignore the fairly untidy tangle of cables and just pay attention to the comments that are surrounded by double brackets. Those show you what you can do with this patch.

• Again, turn the accelerometer slowly upside down, but this time while plotting the output. What do you see?

• Do the same for shaking the accelerometer back and forth. If you had to write a program to differentiate this from the slow turn, what would your program look for?

• What other gestures can you make with the accelerometers? Which ones do you think you could recognize visually just from the plots? Which ones do you think you could write a program to detect?

Naive Gesture Detection in PD

• One obvious difference between fast jerky movements and slow gradual movements is sudden jumps in the acceleration values. We will detect these jumps in Pd with a simple threshold test.

• Take a look at help-delta.pd, the help patch for the delta abstraction. (By the way, note that you can make your own help patches for Pd objects that you write just by following the help-objectname.pd naming convention.) The delta abstraction simply returns the difference between subsequent input values. (Technically, this is a "one-zero highpass filter.")

• Connect a delta object to one or more acceleration values, pick a threshold that corresponds to a satisfying level of jerkiness, and cause Pd to make a sound when you exceed the threshold. For extra credit, give the user additional control of the sound based on the direction and/or the magnitude of the jerk.

• Congratulations; you have now written a jerk detector.

Gesture Detection on the AVR

• Implement the same kind of simple threshold-based "jerk detection" in C on the AVR. Instead of sending every acceleration value, have the AVR send an OSC message only when a jerk is detected. Now your Pd patch only has to make a sound each time it receives an OSC message.

• Once you do this, you are now sampling the accelerometers at a much higher sampling rate, because you don't have to wait for the (slow) transmission of an OSC message between every time you check the current values of the accelrometers. This means two things:

1. You should have lower latency and lower jitter in the overall jerk-to-sound system, and
2. Your thresholds may have to change: since there's so much less time between successive measuresments of the acceleration values, the values can't change as much between successive samples as they did with the slower sampling rate.

Audio Filtering

• The purpose of this part of the lab is to get a sense for the effect of different kinds of filters, and to start thinking about (audio) signals as being comprised of frequency components. Don't worry; we'll come back to the accelerometers later.

• Open the pd patch audio-filters/filter-demo.pd

• This patch allows you to select one of four input sources (white noise, a sine wave, a pair of sine waves, or Monday's collection of oud samples) and pass the sound through one of seven possible filters:

• • No filtering
  • High pass filtering with Pd's (one-pole) hip~ object
  • High pass filtering with a "cascade" of four hip~ objects
  • Low pass filtering with Pd's (one-pole) lop~ object
  • Low pass filtering with a cascade of four lop~ objects
  • Band pass filtering with Pd's bp~ object
  • Band pass filtering with a cascade of Pd's bp~ objects

• Play with this patch to get a feeling of the effect of different kinds of filters on different input sounds.

• Be very careful with the output gain! White noise is extremely loud (per unit of amplitude)!

• Start with the white noise source. This is the best input for hearing the differences between different kinds of filters because it contains all frequencies. (It's called "white" noise by analogy to white light, which contains all frequencies, i.e., all colors of light.) Turn the master volume and/or your headphones way down, then select input source zero (white noise) and filter type zero (unfiltered). Beautiful, huh?

• Now step through the other six filter types, playing with the parameters of each. Sweep the high-pass cutoff frequency. Sweep the cascaded high pass cutoff frequency and note that the four filters have "four times as much" effect on the sound as the single hip~ object. Ditto for the low pass objects. For the band pass, start with the default Q factor of 1 and sweep the center frequency. Then make the Q factor small and sweep the frequency again. Then make the Q factor large and sweep the frequency again. Now you know what these filters do.

• Repeat all of the above on the single sine wave. Note that no matter what filtering you do, all you change is the gain (and phase) of the sine wave. (Geek moment: the reason is because all of these filters are "linear and time-invariant".) This is very important: filters don't add anything; they just change the balance of what's already there. Note that lowpass filtering reduces the volume of high frequency sine waves but has less effect on the volume of low frequency sine waves, etc.

• Now try this on a pair of sine waves spaced pretty widely apart in frequency (for example, 100 Hz and 2000 Hz). Hear how the different filters affect the relative volumes of the two sine waves.

• Finally, play some of the oud samples (via the same QWERTY keyboard triggering mechanism) through various filters. Experiment with transposition and how it interacts with filtering. In particular, transpose the samples down by a large amount and see how highpass cuts all the sound (as with a low-frequency sine wave), while lowpass emphasizes the "bassiness" of the sound.

Filtering Acceleration Data to Distinguish Tilt from Sudden Motion

• Reconnect your accelerometer and open the Pd patch pd/guppy-help.pd. Follow the three numbered steps, including the calibration. To calibrate, return the accelerometer board to its "upright and locked" position, press the "calibrate" button, and leave it alone for about a second. Now move around the accelerometer and note that the tilt appears pretty much exclusively in the "tilt" outputs, and that the sudden motion appears pretty much exclusively in the "sudden motion" outputs. Amazing! How do they do that?

• The answer is with filtering. Caveat: although we believe filtering is the best way to solve this gesture discrimination problem, this particular implementation is somewhat of a hack. The reason is that all of Pd's filtering tools work only on audio signals, so the guppy patch (in particular, the accel-xover subpatch) converts the incoming OSC messages into audio signals, smooths them out, then lowpasses and highpasses them (at 5 and 20 Hertz, respectively) to differentiate tilt (the low frequency component) from sudden movements (which have lots of high frequency components).

• The moral of the story is that control signals have frequency components too, just like audio signals, and you've got a lot more power if you can think about them in the "frequency domain", just like it's powerful to think about audio signals in the frequency domain. Great. Now go read Julius Smith's books.

Put everything together

Using the pd/guppy-help.pd and the audio-filters/filter-demo.pd as starting points, create your own pd patch that make some sound based on the acceleration tracking. For example, you can use the tilt to control cutoff frequencies, and the suden motion to control the pitch (or any other parameter or combination that you like). Think of how you can combine groups of parameters into single gesture in a musical way.

Feel free to add buttons (to control the filter type and/or the sound type for example) into your patch.

Optional: Tweak the filters

• Can you find settings for the filters that better discriminate between tilt and sudden motion?

Optional: Implement the filtering in Pd's event domain

• For all you DSP wizards out there, you don't need to rely on Pd's built-in one-pole audio filters. Instead, you can implement the same filters in the Pd event domain, thereby saving a huge amount of CPU time. First you'll need to figure out the sampling rate you're getting from the AVR. Then it's a simple matter of filter design. Have fun!

NEW! Convert xyz accel data to spherical coordinates with this patch.

Optional: Implement the filtering in the AVR

• As you saw in part 5, you can sample at a much faster data rate if you don't send every single accelerometer measurement over the serial connection. That means you could have a more complete view of the higher frequency components of the accelerometer signals. Wouldn't that be great?

PID 2008