Client-Side Real-Time Emotion Recognition

This asset is responsible for integrating multiple emotion recognition modalities and combining them into a single classification. In its final version it will incorporate 5 different modalities: facial expression recognition, text recognition, speech recognition, touch recognition and GSR/EDA recognition for arousal. The client-side version is designed to run directly in the client application and communicate using web-services with server-side Emotion Recognition Assets (e.g Speech and Text Emotion Recognition Assets). This asset is prepared to run with any number of emotion recognition modalities, and their configuration and incorporation into the classification fusion process is straightforward.

Current status

For this first release, three emotion recognition modalities were integrated (the ones available during this release): the Sentiment analysis asset from UPB, the speech emotion recognition asset from INESC-ID, and an internally developed asset for EDA recognition which will be later replaced by the Real-Time Arousal Detection asset from TUSofia. Three classifier fusion policies were implemented for this first release: max, weighted average, and kalman fusion. The max policy just uses the maximum value of its classifiers for predicting a particular emotion, while the weighted average policy takes into account a weighted average from all classifiers contributing to a particular emotion. The kalman fusion policy is the most complex, and it is based on the system proposed in [1] using a kalman filter to combine multiple sensors. The main advantage of this policy is that it can more easily deal with situations where one of the sensors is not currently active, by using information from past observations and explicitly assuming a higher uncertainty about these observations.

[1] Glodeck, M., Reuter, S., Schels, M., Dietmayer, K. & Schwenker, F. (2013). Kalman Filter Based Classifier Fusion for Affective State Recognition. In Multiple Classifier Systems, Lecture Notes in Computer Science 7872. Springer.