Meaningful Gestures

Kinect, Microsoft’s video-game controller that registers a user’s intentions from his gestures, will be the shape of things to come. Researchers at the Human-Computer Interaction Institute at Carnegie Mellon University, in Pittsburgh, think the Kinect’s basic principles could be used to make a technological panopticon that monitors people’s movements and gives them what they want, wherever they want it. Someone in a shopping mall, for example, might hold up his hand and see a map appear instantly at his fingertips. This image might then be locked in place by the user sticking his thumb out. A visitor to a museum, armed with a suitable earpiece, could get the lowdown on what he was looking at simply by pointing at it. And a person who wanted to send a text message could tap it out with one hand on a keyboard projected onto the other, and then send it by flipping his hand over. In each case, sensors in the wall or ceiling would be watching what he was up to, looking for significant gestures and reacting accordingly.

An older project, OmniTouch, combined a Kinect-like array of sensors with a small, shoulder-mounted projector to project interactive displays onto nearby surfaces, including the user’s body. This prototype, Armura takes the idea a stage further by mounting both sensors and projector in the ceiling. This frees the user from the need to carry anything, and also provides a convenient place from which to spot his gestures. The actual detection is done by infra-red light, which reflects off the user’s skin and clothes. A camera records the various shapes made by the user’s hands and arms. Software then identifies different arrangements of the user’s arms, hands and fingers, such as arms-crossed, thumbs-in, thumbs-out, book, palms-up, palms-down and so on. The hands alone are capable of tens of thousands of interactions and gestures. The trick is to distinguish between them, matching the gesturer’s intention to his pose precisely enough that the correct consequence follows, but not so precisely that slightly non-standard gestures are ignored.

More information:

http://www.economist.com/node/21548486