Is this person happy or frustrated?
May 28, 2012
Can you tell which of these smiles is showing happiness? Or which one is the result of frustration? (See bottom of post.) A computer system developed at MIT can. (Credit: Hoque et al.)
Do you smile when you’re frustrated? Most people think they don’t — but they actually do, a new study from MIT has found.
What’s more, it turns out that computers programmed with the latest information from this research do a better job of differentiating smiles of delight and frustration than human observers do.
The research could pave the way for computers that better assess the emotional states of their users and respond accordingly. It could also help train those who have difficulty interpreting expressions, such as people with autism, to more accurately gauge the expressions they see.
“The goal is to help people with face-to-face communication,” says Ehsan Hoque, a graduate student in the Affective Computing Group of MIT’s Media Lab who is lead author of a paper just published in the IEEE Transactions on Affective Computing. Hoque’s co-authors are Rosalind Picard, a professor of media arts and sciences, and Media Lab graduate student Daniel McDuff.
In experiments conducted at the Media Lab, people were first asked to act out expressions of delight or frustration, as webcams recorded their expressions. Then, they were either asked to fill out an online form designed to cause frustration or invited to watch a video designed to elicit a delighted response — also while being recorded.
Still images showed little difference between these frustrated smiles and the delighted smiles elicited by a video of a cute baby, but video analysis showed that the progression of the two kinds of smiles was quite different: Often, the happy smiles built up gradually, while frustrated smiles appeared quickly but faded fast.
In addition to providing training for people who have difficulty with expressions, the findings may be of interest to marketers, Hoque says. “Just because a customer is smiling, that doesn’t necessarily mean they’re satisfied,” he says. And knowing the difference could be important in gauging how best to respond to the customer, he says: “The underlying meaning behind the smile is crucial.”
The analysis could also be useful in creating computers that respond in ways appropriate to the moods of their users. One goal of the research of Affective Computing Group is to “make a computer that’s more intelligent and respectful,” Hoque says.
The work was supported by Media Lab consortium sponsors and by Procter & Gamble Co.
* The smile on the right is from frustration.
Ref.: Mohammed E. Hoque, Daniel J. McDuff, Rosalind W. Picard, Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles, IEEE Transactions on Affective Computing, 2012, DOI: 10.1109/T-AFFC.2012.11
Update 5/28/2012: title changed for accuracy (was: “Is this smile real or fake?”) and clue moved to the bottom. — Ed.
Comments (6)
by Simon van Rysewyk
To Peter Simmons.
You wrote: ‘We do this sort of thing unconsciously all the time, another superiority to robots that won’t ever change. Just don’t think too hard about it and it is obvious.’ Actually, let’s think about it. It is far more complex than you indicate.
Consider the use of AI in pain assessment. Human beings are quite limited in their ability to detect changes in the intensity of pain facial actions over the relatively brief time frames involved in pain facial expression (movement dynamics). Currently, the complexity of pain facial expression is available only to human beings with skill in decoding pain faces. But, the reliability of human observation – even in trained health care professionals – is far from perfect. This is where IT can enhance pain assessment. IT pain assessment can provide precise assessment of changes in pain actions over time, and can discriminate genuine and controlled facial displays (useful if a patient is faking pain).
And, in case you were wondering: AI does not aim to displace the doctor-patient relationship anymore than Mozart or Rembrandt are in competition with molecular genetics. AI and IT in such settings are used to supplement human pain assessment, not displace it.
by Peter Simmons
It was obvious, and easy; I knew immediately. We do this sort of thing unconsciously all the time, another superiority to robots that won’t ever change. Just don’t think too hard about it and it is obvious.
by Cybernettr
You were just lucky. After all, you had a 50/50 chance.
by Ralph Dratman
I looked at the photos and compared the sensations in my face when trying out a smile of frustration and one of happiness. I could feel my forehead and eyes staying flat while acting frustrated — I used a “you’ve got to be kidding” expression — but crinkling up when acting happy. Looking at the two photos, I could see that in the one on the right, the area around the eyes and forehead was smooth. That turned out to be right, but of course I had a 50% chance of being right even without any knowledge.
by eric g
sorry, the post title is misleading. According to the story, the computer gets to see more, like the duration of the smile as well as the preceding the snapshot. Given these two screenshots, the computer would be unable to ascertain as easily.. and I feel if we were shown 5-10 videos like the computers, we (humans) could as well.
by Editor
Eric: you’re right. I just changed the title. The smile in frustration is not faked (if we accept the validity of the data), just confusing, which is the central point of the research, if I understand it correctly. The giveaway is the lack of expression in the upper part of the face in the image on the right.