We brainstormed various gestures that can be used for selection. Some of our favorites included:

• move both arms left (or right) to scroll through linear tiles, then push both arms forward to select (like cover-flow)
• make an "X" or "O" with your arms to answer "no" and "yes"
• type with your body with custom alphabet sign-language that requires whole body movement
• clap (gesture and/or sound recognition): number of times clapped indicates the choice made
• turn (virtual) knob clockwise or counter-clockwise, then clap to select
• kick on the object to get rid of it; throw at it to destroy it; slash with a blade, like fruit ninja...
• selection based on the distance user is away from the Kinect sensor
• two-user cooperation methods: high-five and low-five to agree on a selection; turn backs to partner to disagree.
• other two-user gesture potentials: piggy back. hug.
• physical workout gestures: jumping jacks, squat, push-ups, sit-ups, side-step

Design Sketch

Description & Rationale

In this Multi-Select Drag and Drop controller, there are several selectable items floating around the user on screen. Users can select an item by grabbing it using both hands. When grabbing begins, the color of the item changes from light gray to dark gray. To finalize selection, the item should be dragged into the "like" bin.The color of selected items change to match the color of the bin, a visual cue that the item has been categorized accordingly.Users can repeat these actions to add more items into a bin; in this manner, this controller allows for selecting multiple items (and even categorizing them to multiple bins). To undo selection, re-drag the unwanted item into the "dislike" bin.

Context of Use: when, where, why

Evaluation

We minimize false-positives by activating dragging only when users have both hands on an item, and by visually displaying the drop-boxes in which users must release the items. (It is very easy for a single hand to accidentally reach over items displayed on screen, but more difficult to have both hands over an item unless intended. Also, by having to bring the item into the bin with both hands, we make the selection gesture very explicit.

This design is ideal when there are several ( 2 <= n <= approximately 10) possible items to be selected. Users may experience fatigue if there are too many (e.g. >10) items to be selected or categorized.

Ability to undo: We had originally considered "drag-and-pop" controller, used to select and pop-away selected items. In this previous design, users would use a single-hand touch gesture to drag desired item, and use the second hand to clap the first hand to confirm selection, which would pop away the chosen item. However, this design made it difficult to undo, as the selected item would disappear from the screen. By keeping selected items color-coded on screen, users can undo the selections by re-dragging undesirable items into the "dislike" bin.

Other Potential Application Scenarios

(1) Two people are playing a game together and would like to select game characters. There are several game characters available and the two players have their own preferences. Among the possible characters, each player will select top three chocies and put it to the three boxes: "first choice", "second choice", and "third choice". Based on the selections made, the computer will randomly (but with different probability) assign a character to the players, such that more preferable choice will have higher chance of being picked. This will add more fun to players.

(2) Parents want to play videos for their kids. They have many available videos and want their kids to choose the one they want. Kids can choose the videos they prefer from among the different titles by using this controller. Using this, kids will not only see the videos but also have some physical activities with more fun.

Design Sketch

Description & Rationale

This controller allows for easy scroll-through by "moving" your hand with respect to your elbow:

• Step 0: Stand in the sweet spot in front of the Kinect. This causes a glowing object to show on screen, as a visual feedback. The closer the user is to the sweet spot, the bigger the glow. The glow indicates that the system is ready to receive gesture commands from the user.
• Step 1: Raise your left hand (over your head) to bring up the coverflow menu. The menu is a linear list of items to be selected. Once a menu is brought up, user can put the hand down.
• Step 2: Browse with your right hand: either move the hand left (approximately 11 o'clock) or right (approximately 1 o'clock), with respect to the elbow, to browse forward or backward. The right arm does not need to be held up high; it can be at any height.
• Step 3: Once the desired item is shown in the center of list, select using your left hand: do a simple "up" gesture by raising the left hand at least to the height of the left elbow. Upon selection, the menu automatically hides from view.

Since this controller is implemented based on vectors and angles, gestures will be recognized irrespective of user's position in space or distance from camera, provided that the user is facing the camera.

Note: This controller was implemented on XNA framework 4.0

Context of Use: when, where, why

While this controller can be applied to a wide range of context, it would work best when users need to scroll through a long list of items, such as choosing a movie from a large database. From small hand gestures, users can control the direction and speed of scroll, and signal selection.

It would be possible (and easy) to adopt this controller for a binary selection task (true | false): users can simply move the hand left or right to signal true or false. For selection from linearly ordered display items, as described above, holding the hand at the 11 o'clock position continues the scroll backward, and holding the hand at the 1 o'clock position continues the scroll forward. To increase the speed of scroll, increase the angle of rotation, such as by holding at 10 o'clock or 2 o'clock position.

In summary, this design leverages our ability to have fine control over our hand position, to be interpreted as the scroll-speed.

Evaluation

Speed: as described above, this controller allows the user to have fine control over the speed of scroll. The controller leverages our ability to move our hand and arm quickly, without requiring a large space.

Accuracy: a small range of angles close to 12 o'clock position should be reserved for holding the item still in the scroll menu. By lowering sensitivity (e.g. larger angle movement of hand required for scrolling at a given speed), we can improve accuracy for users. (This results in a direct trade-off between speed and accuracy of control.)

False Positives: We make our gestures explicit by having a high-wave gesture to get Kinect's attention (described in Step 1 above). We minimize false positives by using right hand for browse (Step 2) and left hand for select (Step 3). Visual feedback of which item is currently in the center aids users to not make mistakes.

Fatigue: users can relax the upperarm and not hold it up so high to use this controller.

Controller 1: Multi-Select Drag and Drop

Controller 2: Elbow-Pivot Browser

Note: We have two separate zip files, one for each controller, because the first controller was implemented as a WPF application while the second controller was implemented using the XNA framework. We checked with Elliot that this would be okay. To compile and run the second controller, you need Microsoft XNA Game Studio 4.0 and Micorsoft Visual Studio SP1; refer to the README file in the zip for more info

Note 2: When compiling the Elbow-Pivot Browser, if you're getting "Error loading pipeline assembly (some .dll file)", go through all the dll files in the /lib directory, right click, go to properties, and then "unblock" them; it seems like Windows 7 takes extra security precautions on .dll files downloaded from the web…

Controller 1: download Multi-Select Drag and Drop (WPF)

Controller 2: download Elbow-Pivot Browser (XNA)