Bare-Hand Gesture Interactive Sound Panorama

Spatial audio and Individual Sound Zones were making it possible to place sounds anywhere in a cabin. The controls for doing so were menus on a screen. Placing a sound in space with a two-dimensional touch UI is indirect, distracting for a driver, and throws away the intuition everyone already has about where things are.

Let the hand do it. Users grab, move and release sound sources such as music, phone calls and alerts, and the surround panorama follows. In a car the driver can hand a call or a song to a passenger with a gesture, complementing HARMAN's Individual Sound Zones. The central design principle was input-output coincidence: the gesture is the manipulation, so the interaction is immediately understood.

As project lead I contributed to the ideation and refined the concept, built the first prototype myself, and oversaw the iterations that followed, including the wearable version. A preliminary automotive product was defined as GESS, Gesture Enabled Sound Space.

The first prototype used six JBL Pulse speakers in a mock cabin, chosen because they light up when playing, which gives visual feedback about where a sound currently lives. An Xbox Kinect tracks the hand in 3D and recognizes grab and release, so a sound can be picked up, carried to another seat, and dropped or thrown.

The headphone prototype mounts a Leap Motion controller on the headband to track the hands, and uses off-the-shelf 3D audio tools for rendering. Users move individual sounds left-right, forward-back and, within limits, up-down. It works without any display, which is exactly what headphones need.

Two granted patents, one for the in-vehicle system and one for the wearable system, an early automotive product definition (GESS), and a line of thinking that carried into HARMAN's later spatial audio work and the pseudo-holographic instrument cluster.