FANCI Consortium Project

Individual sensing technologies for the car were maturing separately: gaze tracking, gesture, emotion recognition, facial recognition. Each vendor had a demo; none of them added up to a car that understands its driver. The European Union funded FANCI, Face and body Analysis Natural Computer Interaction, to find out what happens when they are fused.

A two-year, €2.5 million Horizon 2020 project bringing together leading companies in eye-gaze tracking, bare-hand gesture interaction, emotion sensing, facial recognition and embedded DSP to build next-generation automotive interaction. Sensor fusion produces a human-like interface: music, lighting and climate adapt to the driver's mood; display brightness follows where the driver is looking; the system understands and responds to the driver's needs in real time.

As HARMAN's project lead I contributed to the UX and HMI design, the system architecture, the API design and future-proofing, and the integration of emerging technologies into the platform. I presented the first demonstrator at CES 2016 to OEM executives, partners and journalists, which earned positive coverage for HARMAN and the consortium, including from CNET.

I co-authored the IEEE paper “Sensor Fusion Paradigm for Smart Interactions Between Driver and Vehicle”, nominated for Best Industrial Paper at IEEE Sensors 2016 in Orlando.

Each partner's sensing module feeds a shared fusion layer through a common API. Facial recognition identifies the driver, emotion sensing estimates state, gaze tracking provides attention and gesture provides intent. The fused model drives the cabin: adapting media, lighting and climate to improve mood, adjusting display brightness based on gaze, and enabling gesture and gaze interactions that reduce distraction.

A working sensor-fusion demonstrator presented at CES 2016, positive press for HARMAN and the consortium, and an IEEE Sensors 2016 paper nominated for Best Industrial Paper. Project results are documented on the European Commission's CORDIS portal.