Audio Augmented Reality

Headphones had exactly two settings for the world: let it all in, or block it all out. Noise cancellation was a blunt instrument, and it made people less safe on the street and less present with the people around them. Hearing is the one sense we cannot close, and nobody had given it an interface.

Audio Augmented Reality (AAR): a head-worn system that processes everything the ears would hear and selectively cancels, attenuates, enhances, adds or replaces individual auditory events. Mute the traffic, keep the voices. Amplify the kettle. Swap the din of the street for ocean waves. Always let sirens and your own name through.

The result is better hearing, a more pleasant auditory experience, and more safety in noisy environments, all under the user's control by gesture or voice.

I was part of the ideation, patenting and refinement of the concept, and I directly managed the creation of the first three proof-of-concept prototypes. I also managed the production of the UX film shown here, which was presented at CES and other trade shows.

External microphones capture the acoustic scene; onboard processing classifies sound sources and events, then applies per-source treatment before rendering to the ears: cancellation, attenuation, enhancement, or replacement with a chosen sound. Users adjust the prominence of sound categories with gestures, for example pointing to mute traffic, or voice commands. Critical sounds such as sirens or the user's name bypass the filters.

The same idea moves into the car cabin, where existing microphones, sound systems and voice agents make an in-cabin AAR simpler to implement and less computationally constrained than a head-worn system.

The team's work directly influenced HARMAN products. The Ambient Aware feature that debuted in the JBL Everest Elite 700 and JBL Reflect Aware headphones originated from this concept, a first step toward a family of products with enhanced auditory UX, with an estimated five million units sold to date. Multiple patents were granted from the project.