CS247 Project 3: Gesture Controller

Hongchan Choi, Sukwon Chung, Tyler Crimm, Jieun Oh, Matt Sencenbaugh
{ hongchan, quantasw, tman7000, jieun5, msencenb } @stanford

Jaunary 26, 2012

Jump to: Brainstorming | Controller 1 | Controller 2 | Videos | Source Code

Initial Selection-Gesture Brainstorming

We brainstormed various gestures that can be used for selection. Some of our favorites included:

Gesture Controller 1: Multi-Select Drag and Drop

Design Sketch

Description & Rationale

In this Multi-Select Drag and Drop controller, there are several selectable items floating around the user on screen. Users can select an item by grabbing it using both hands. When grabbing begins, the color of the item changes from light gray to dark gray. To finalize selection, the item should be dragged into the "like" bin.The color of selected items change to match the color of the bin, a visual cue that the item has been categorized accordingly.Users can repeat these actions to add more items into a bin; in this manner, this controller allows for selecting multiple items (and even categorizing them to multiple bins). To undo selection, re-drag the unwanted item into the "dislike" bin.

Context of Use: when, where, why

Using this gesture controller to select multiple items can be a great metaphor when the items selected, combined, create new meaning. For example, in an application for ear training, we can select a musical note, one at a time, and add them to the selection bin to make a chord.

Or we can select multiple numbers to that sum up to some number, in an application for arithmetic education.

Also, users can categorize selected items into different groups if we use several, distinct selection bins.


We minimize false-positives by activating dragging only when users have both hands on an item, and by visually displaying the drop-boxes in which users must release the items. (It is very easy for a single hand to accidentally reach over items displayed on screen, but more difficult to have both hands over an item unless intended. Also, by having to bring the item into the bin with both hands, we make the selection gesture very explicit.

This design is ideal when there are several ( 2 <= n <= approximately 10) possible items to be selected. Users may experience fatigue if there are too many (e.g. >10) items to be selected or categorized.

Ability to undo: We had originally considered "drag-and-pop" controller, used to select and pop-away selected items. In this previous design, users would use a single-hand touch gesture to drag desired item, and use the second hand to clap the first hand to confirm selection, which would pop away the chosen item. However, this design made it difficult to undo, as the selected item would disappear from the screen. By keeping selected items color-coded on screen, users can undo the selections by re-dragging undesirable items into the "dislike" bin.

Other Potential Application Scenarios

(1) Two people are playing a game together and would like to select game characters. There are several game characters available and the two players have their own preferences. Among the possible characters, each player will select top three chocies and put it to the three boxes: "first choice", "second choice", and "third choice". Based on the selections made, the computer will randomly (but with different probability) assign a character to the players, such that more preferable choice will have higher chance of being picked. This will add more fun to players.

(2) Parents want to play videos for their kids. They have many available videos and want their kids to choose the one they want. Kids can choose the videos they prefer from among the different titles by using this controller. Using this, kids will not only see the videos but also have some physical activities with more fun.

Gesture Controller 2: Elbow-Pivot Browser

Design Sketch

Description & Rationale

This controller allows for easy scroll-through by "moving" your hand with respect to your elbow:

Since this controller is implemented based on vectors and angles, gestures will be recognized irrespective of user's position in space or distance from camera, provided that the user is facing the camera.

Note: This controller was implemented on XNA framework 4.0

Context of Use: when, where, why

While this controller can be applied to a wide range of context, it would work best when users need to scroll through a long list of items, such as choosing a movie from a large database. From small hand gestures, users can control the direction and speed of scroll, and signal selection.

It would be possible (and easy) to adopt this controller for a binary selection task (true | false): users can simply move the hand left or right to signal true or false. For selection from linearly ordered display items, as described above, holding the hand at the 11 o'clock position continues the scroll backward, and holding the hand at the 1 o'clock position continues the scroll forward. To increase the speed of scroll, increase the angle of rotation, such as by holding at 10 o'clock or 2 o'clock position.

In summary, this design leverages our ability to have fine control over our hand position, to be interpreted as the scroll-speed.


Speed: as described above, this controller allows the user to have fine control over the speed of scroll. The controller leverages our ability to move our hand and arm quickly, without requiring a large space.

Accuracy: a small range of angles close to 12 o'clock position should be reserved for holding the item still in the scroll menu. By lowering sensitivity (e.g. larger angle movement of hand required for scrolling at a given speed), we can improve accuracy for users. (This results in a direct trade-off between speed and accuracy of control.)

False Positives: We make our gestures explicit by having a high-wave gesture to get Kinect's attention (described in Step 1 above). We minimize false positives by using right hand for browse (Step 2) and left hand for select (Step 3). Visual feedback of which item is currently in the center aids users to not make mistakes.

Fatigue: users can relax the upperarm and not hold it up so high to use this controller.

Demo Videos

Controller 1: Multi-Select Drag and Drop

Controller 2: Elbow-Pivot Browser

Source Code

Note: We have two separate zip files, one for each controller, because the first controller was implemented as a WPF application while the second controller was implemented using the XNA framework. We checked with Elliot that this would be okay. To compile and run the second controller, you need Microsoft XNA Game Studio 4.0 and Micorsoft Visual Studio SP1; refer to the README file in the zip for more info

Note 2: When compiling the Elbow-Pivot Browser, if you're getting "Error loading pipeline assembly (some .dll file)", go through all the dll files in the /lib directory, right click, go to properties, and then "unblock" them; it seems like Windows 7 takes extra security precautions on .dll files downloaded from the web…

Controller 1: download Multi-Select Drag and Drop (WPF)

Controller 2: download Elbow-Pivot Browser (XNA)