Mid-Air Haptic Feedback System

Ultrasonic transducer arrays that let drivers feel gestures in mid-air, from a JBL speaker at CES 2016 to a fully integrated test vehicle at CES 2017.

Company: HARMAN International
Year: 2014 — 2017
Role: Project lead · concept, prototype program, in-vehicle integration, OEM and executive demonstrations
Team: Future Experience Team, with Ultrahaptics (now Ultraleap)
Location: Palo Alto, CA

Technology

Ultrasonic phased transducer arrays
Hand tracking
Automotive gesture GUI
Acoustically transparent trim
Custom in-vehicle integration

01The opportunity

Gesture control in cars had a feedback problem. A wave in the air that nobody acknowledges feels like talking to a wall, and drivers glance at the screen to check whether the system understood. Ultrasonic arrays could project a tactile sensation onto the hand in mid-air. If gestures could be felt, they could be trusted, and eyes could stay on the road.

02The idea

Use phased arrays of ultrasonic transducers to add a tactile dimension to gestures and spatial displays: a click you feel when a gesture is recognized, a texture that tells you which control your hand is over. The domain is wide, from automotive to AR/VR/XR, 3D interfaces, wearables and gaming. We started with the car, where the safety case is strongest.

03My role

I led the collaboration with Ultrahaptics and the prototype program. I showed the first prototype, a JBL speaker with gesture and mid-air haptic controls, at CES 2016. I then spearheaded a standalone tabletop prototype for automotive use cases and a fully integrated system in a test vehicle, presented at CES 2017, and demonstrated the technology to HARMAN executives, including board chairman Young Sohn, and to major OEMs in Silicon Valley and Detroit.

04How it worked

An array of ultrasonic transducers focuses acoustic pressure at points in space to create sensations on the skin. A hand tracker locates the fingers; the system renders tactile feedback synchronized with a gesture-driven infotainment GUI. In the vehicle prototype the array sits behind custom trim and transmits through acoustically transparent material, so the technology can be hidden inside a grille rather than exposed on the dashboard.

05Prototype / product

Mid-air haptics transducer array integrated into custom vehicle trim behind an acoustically transparent grille — Fig. 01PrototypeThe in-vehicle prototype. The transducer array is embedded in custom trim and drives a fully functional GUI for automotive gestures. The system transmits through acoustically transparent materials such as this grille.

Mid-air haptics system installed in the test vehicle's center console — Fig. 02PrototypeInstalled in the test vehicle and presented at CES 2017.

Tabletop mid-air haptics prototype with display showing infotainment applications — Fig. 03PrototypeThe tabletop prototype, exhibited worldwide, in China, Korea, Japan, Germany, Italy, the UK and the U.S., to show mid-air haptics across infotainment applications.

Tabletop mid-air haptics demonstrator with transducer array and hand tracker — Fig. 04PrototypeThe standalone tabletop demonstrator for automotive use cases.

JBL Authentics L16 speaker with mid-air haptic gesture controls at CES 2016 — Fig. 05PrototypeCES 2016: a JBL Authentics L16 speaker with gesture controls and mid-air haptics, developed with Ultrahaptics. A natural fit for studio tools like 3D mixing consoles, and eventually for touchless feedback on headphones.

06Outcome

The CES 2016 speaker drew coverage from The Verge; the CES 2017 in-vehicle system was covered by ExtremeTech, and HARMAN announced a partnership with Ultrahaptics to bring custom haptic sensations to the connected car. The prototypes positioned HARMAN as an early mover in touchless haptics and opened discussions with leading automotive manufacturers about integrating the technology into future vehicles.

Shown at CES 2016 and CES 2017
HARMAN / Ultrahaptics partnership announced
Demonstrated to OEMs in Silicon Valley and Detroit

07Patents, press, links

Press

Links