By combining audio and visual data, Yongjin Wang from the University of Toronto and Ling Guan from Ryerson University in Toronto have developed a system that recognizes six human emotional states: happiness, sadness, anger, fear, surprise, and disgust.

Their system can recognize emotions in people from different cultures and who speak different languages with a success rate of 82%.

Emotion recognition systems help the computer to understand the affective state of the user, and hence the computer can respond accordingly based on that perception.

Some vocal and facial features may play an important role in characterizing certain emotions, but a very minimal role in other emotions. As a general example, happiness is detected better using certain visual features (e.g. in smiling), while anger is detected better using audio features (e.g. in yelling).

The researchers suggest that a computer that can recognize human emotion could one day have applications in customer service, computer games, security/surveillance, and educational software.