Difference between revisions of "250a Graphics Lab"

Revision as of 12:02, 1 November 2012

Lab 6: Graphics
See this quarter's schedule for due dates.

In this lab we will explore different possibilities for computer graphics in musical interaction using Satellite CCRMA, Pd, and a custom graphics system called SimpleGraphics. You need your MaxKit, with Satellite CCRMA.

Choose A Place To Sit in the Ballroom

• It is a little bit trickier to use the ViewSonic monitors because you will need to run the following extra commands on your kit before it works (so you might prefer to sit at a SAMSUNG monitor for example because then that won't be an issue):

cd /boot/uboot

sudo pico boot.cmd

• change line that starts 'setenv dvimode' to 'setenv dvimode 640x480@60'

sudo mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n "Ubuntu" -d ./boot.cmd ./boot.scr

These commands change the default HDMI video resolution to one that works with the ViewSonic monitors.

Setting Up Software

• Use the usual procedure (see described before) to power up Satellite CCRMA and login as the user ccrma with the password temppwd.

• Download the lab files to your laptop.

• Copy the lab files to your home directory ~ using CyberDuck.

• Unzip the files directly on your board: change into your ~ directory and unzip the archive by running the following two commands:

cd ~

unzip graphicslab2012.zip

• Remove the archive file by typing

rm graphicslab2012.zip

• Change to the ~/graphicslab2012 directory by typing

cd ~/graphicslab2012

• Install the graphics software by running a script (this will take a while and require entering the password temppwd):

./install-SGX.sh

• If the installation was successful, then remove the graphics software archive:

rm GFX.zip

and then reboot your kit by running the following command (after this you will of course need to login again):

sudo reboot

SimpleGraphics

SimpleGraphics is a small program that can receive information about graphics primitives and then render these using OpenGLES. The result is a quick and easy visualization tool for environments that otherwise might have difficulty rendering real-time graphics, such as Pd on Satellite CCRMA.

To get started with SimpleGraphics, connect your Satellite CCRMA kit to a monitor and run the SimpleGraphics program on your kit:

   cd SimpleGraphics/
   ./SimpleGraphics &

At this point you should just see a blank screen, because we have not given SimpleGraphics anything to draw yet.

Open up "simplegfx-primitives.pd" in pd:

   pd simplegfx-primitives.pd &

Follow the instructions in the patch.

You can see that each drawing "command" consists of

   send /sg/somecommand object-name a-bunch-of-numbers

The command either creates or modifies an object that has that name, according to the parameters in the remainder of the command. For the ellipse command, these numbers correspond to

   x-position y-position width height color-red color-green color-blue color-alpha

So

   send /sg/rect myrect 0 0 0.1 0.1 1 0 0 1

will create a rectangle a position (0, 0) with size (0.1, 0.1), with a color of (1, 0, 0, 1), i.e. pure red.

Play around with these primitives a bit. What command will make a circle that fills the whole screen? Try combining multiple primitives to create some more complicated image, like a keyboard or a face. What happens when you change a shape's alpha to less than 1? What if you draw multiple overlapping primitives with alpha < 1?

Animation

Close simplegfx-primitives.pd, restart SimpleGraphics, and open simplegfx-animation.pd. Here you will see a more complicated patch that sends out the same command periodically with parameters that vary according to the value of a sine wave. Note that osc~ is used to create the sine wave, just like for a regular audio signal, but we have to use snapshot~ to convert the value into something we can stick in a SimpleGraphics message.

Animation in computer graphics is useful for all sorts of reasons, but two stand out for use in interactive computer music. Firstly, obviously entities in the graphical space will change position and other parameters in response to human input, such as a circle that bounces around the screen when you shake an accelerometer, which we can accomplish using animation. More subtly, animation can be used to make static objects like "lively" and "natural". If in the previous example the circle's size modulates a little bit over time, it becomes more clear that this circle might be something we can interact with.

Come up with your own animation using SimpleGraphics and Pd. You might try varying the color of an object with noise~ -- do you want to vary the color across the full spectrum, or just add a little "crackle" to a static color?

Combining Sound and Graphics

Again restart SimpleGraphics, and now run simplegfx-sound.pd. You'll see this plays back a short audio file, mapping volume to the size of a graphical object and pitch to its color.

The issue of mapping is crucial when combining audio and visual media, as individually music and visual arts don't always have a natural correspondence.

Try changing this patch to map volume and pitch to different parameters of a different type of object. Does this make more or less sense to you than the original mapping?

Make A Musical Interaction

As the final main deliverable of the lab, create an interaction with the Satellite CCRMA kit that produces both sound and graphics in response to sensory input. Think about what you want to design before you get started, and especially consider the relationship between the generated sound and graphics.