Ghost Fingers

Multi-touch was spreading from phones to every screen, including screens across the room: TVs and large monitors. You cannot touch a display beyond arm's reach, and a mouse throws away the directness that makes touch good. Keyboards were still the best way to type and the worst way to point.

Seamless switching between keyboard input and multi-touch gestures on a display beyond reach. The system shows a translucent, real-time image of the user's fingers and hands over the remote UI, closing the interaction loop so direct manipulation works at a distance. Text input and touch can even happen at the same time, one hand each.

I developed the concept and the prototypes at HP/Palm and wrote the two patent applications that protected the novel elements; the system was also documented in a published research paper.

A wireless keyboard carries a micro camera facing the keys. The camera tracks hand and finger positions and generates a live, translucent overlay of the fingers on the remote UI. In text-input mode the keyboard behaves normally. Pressing a designated key, for example CTRL, switches to multi-touch mode with finger tracking, and users press icons or manipulate elements on screen without touching the display. Only hands in touch mode are visualized; a typing hand needs no overlay.

The prototypes delivered an excellent user experience and the innovations were filed as two patent applications and described in a published paper. The patents were later sold to Qualcomm as part of a larger IP transfer and subsequently abandoned by Qualcomm as a business decision, which makes the concept publicly available for any company that wants to bring it to life.