Could emotion detectors make driving safer?

March 17, 2014

(Credit: EPFL/Jamani Caillet)

Researchers in EPFL’s Signal Processing 5 Laboratory (LTS5), working with PSA Peugeot Citroën, have developed an emotion detector based on the analysis of facial expressions in a car, using an infrared camera placed behind the steering wheel.

3D facial analysis (credit: NVISO, an LTS5 spin-off company)

The researchers say they can read facial expressions and identify which of the seven universal emotions a person is feeling: fear, anger, joy, sadness, disgust, surprise, or suspicion.

The problem was to get the device to recognize irritation on the face of a driver. Everyone expresses this state somewhat differently — a kick, an epithet, a nervous tic or an impassive face.

To simplify the task at this stage of the project, Hua Gao and Anil Yüce, who led the research, chose to track only two expressions: anger and disgust.

How to detect irritation

  1. The system “learned” to identify the two emotions using a series of photos of subjects expressing them.
  2. Then the same exercise was carried out using videos. When the test failed, it was usually because this state is very variable from individual to individual — given the diversity of how we express anger.

(Credit: EPFL)

Additional research aims to explore updating the research system in real time, using a self-taught human-machine interface or a more advanced facial monitoring algorithm, says Gao.

Detecting emotions is only one indicator for improving driver safety and comfort. In this project, it was coupled with a fatigue detector that measures the percentage of eyelid closure.

The LTS5 is also working on detecting other states on drivers’ faces, such as distraction, and they are studying lip reading for use in vocal recognition (“OK car, cruise at 50 mph”).

So how would facial detection data showing irritation could be used in the real world? A calm voice (Scarlett Johansson from Her?) suggesting the driver calm down, or pull over and take a stress pill? A live display advising the driver?

(Credit: KurzweilAI)

Or maybe a future connected car system could display the moods of all drivers to avoid road rage?

EPFL’s Thiran Jean-Philippe suggested, in an email response to KurzweilAI, that emotional states like stress or anger could be handled with soft music or a soft light on the dashboard. “Similarly, detecting fatigue could lead to launching energetic music or more aggressive lighting of the dashboard. In the longer term, in the context of semi-autonomous vehicles, it will be increasingly important for the car to be aware of the emotional state of the driver, in order to correctly decide which tasks have to be handled automatically by the car and which ones can be/have to be transferred back to the driver.”

‘Driver State Estimation’ — when the car takes over

Volvo wants to take all this a big step further. It announced today it’s conducting research into driver sensors to create cars that get to know their drivers. The idea is to check to see if the driver is focused, and if the driver’s attention is directed elsewhere, and use that data to inform Volvo’s semi-automated Lane Keeping Aid, Collision warning with full auto brake, and Adaptive Cruise Control with Queue Assist.

A sensor would be mounted on the dashboard in front of the driver and measure between different points on the driver’s face*. Small LEDs illuminate the driver with invisible infrared light, which is then monitored by the sensor. The system is already installed in test vehicles.

By monitoring eye movements, the car would be able to adjust both interior and exterior lighting to follow the direction in which the driver is looking. The car would also be able to adjust seat settings, for instance, simply by recognising the person sitting behind the wheel.

“The analysis of the driver’s state, known as Driver State Estimation, in which driver sensors play an important role, is a field that may be key to self-driving cars in the future,” says Volvo. “The car will need to be able to determine for itself whether the driver is capable of taking control when the conditions for driving autonomously are no longer present. A driver sensor could be of assistance in this.”

“This technology is one of the many initiatives bringing Volvo Cars closer to its goal for 2020: that no one shall be killed or seriously injured in a new Volvo,” advised the company.

* “The car doesn’t save any pictures and nor does it have a driver surveillance function,” according to Volvo’s  Per Landfors.