By: Dr. Zerrin Yumak

Virtual Human Controller is a RAGE asset developed by Utrecht University. It is a collection of modules that allows to create virtual characters that can talk, perform facial expressions and gestures. The asset provides a quick pipeline to build conversational 3D non-player characters from scratch.

Why is it useful?

If you are a game developer that likes to include virtual characters with realistic appearance and social behaviors, this is the asset for you. For example, you might want to develop a job skills training game where a virtual character takes the role of an interviewer. Or you want to train police officers where they interact with non-player characters in a game that takes the role of citizens or victims . You could also imagine a game to help people with social anxiety to safely experience social situations. It is up to your imagination! Role-playing characters are required in almost every application area including business, health, education, security and military.

What does it actually do?

The asset is built on top of the Unity 3D Game Engine. It includes the following functionalities:

  • Creating an animatable 3D avatar: We defined a pipeline where 3D virtual characters can be animated easily. We export skinned 3D models from DAZ3D Editor together with the static expressions for speech and facial animation and import them to Unity. That requires no designer effort in terms of animation.
  • Individual Animation Controllers: We have individual animation controllers for speech, gaze, facial expressions and gestures. For speech (1) and gaze (2), we are currently using 3rd party low-cost or free assets from Unity Asset store. For text-to-speech, we use CereVoice (3), which is currently the world’s most realistic and affordable text-to-speech engine that also supports a variety of languages. For the facial expressions and gestures, Unity’s animation blending and blendshapes features are used. We also provide a database of conversational gesture animations database captured with a Vicon Motion Capture device that exists at our university.
  • Multi-modal animation generation: The asset helps to generate multi-modal synchronized animations thanks to the Behavior Mark-Up Language (BML) (4). We developed a BML Realizer for Unity. Why is BML useful? It has a human-readable notation that allows to semantically combine any facial expressions, gestures, speech and gaze animation together. To our knowledge, a BML Realizer for Unity does not exist or is not publicly available.

Does that sound complex? Not really, it took for our partners just a few days to animate a new virtual character with our asset. And that is even becoming faster based on feedback we receive. We provide a simpler pipeline in comparison to other publicly available virtual human controllers such as VHToolkit (5). Playing animations is also very easy: a few lines of BML input and you have your character talking and waving to you.

rage-2

When and on which platforms can we use it?

The above described functionalities are currently available. Our asset is developed based on Unity game engine. That was a deliberate choice as Unity provides a fast pipeline with its fundamental character animation capabilities. That allows us to focus on more complex social animations. Since Unity is multi-platform, our asset can be used on different platforms including web and mobile applications. Our colleagues from INESC-ID already used it for a web application.

Where is it used in RAGE currently?

We successfully integrated our asset with Communicate! (6) dialogue manager from Utrecht University and with the Emotion Appraisal (7) module from INESC-ID from Portugal. In addition to inter-asset integration, our asset is currently being used by the game developers at BipMedia in Paris. It will take part in the interviews skills training game for Randstad (8).

What are the other projects that use this asset?

We are interested in naturally behaving interactive virtual characters in games and in virtual reality applications. Our asset was recently used for a case study where the virtual character takes the role of a Virtual Receptionist. The set-up includes a microphone to capture people’s speech and a Kinect camera to capture their behaviors. Beyond the functionalities of the Virtual Human Controller, we added Google speech recognition and chatting functionalities using AIML Pandorabots. Furthermore, we developed a novel autonomous gaze control module based on Kinect to drive the “look at” behavior of the virtual character in group-based interactions. This is our new research and will be available for interested asset and game developers.

We showed our results as a live interactive demo in two public events recently: one in May for the visit of EU ambassadors to Utrecht and the other one in June during the INTETAIN 2016 conference. Take a look at the video:

What is next?

Our next goal is to develop a module for physiologically-driven socio-emotional gaze behavior. What does that mean for the game developers? It means we will provide them an easy tool to generate expressive gaze behaviors by only tweaking a few parameters. For example, gaze behavior for an angry or friendly character will be automatically generated by changing the low-level physiological parameters (saccade duration, blinking rate etc). We will also explore how gaze is synchronized with other modalities such as gestures and facial expressions.

Contact Information

Our asset is available to RAGE game developers and other interested parties. Currently, asset information can be accessed from the RAGE portal. We will soon provide a public website for our asset and licensing information. For access to the software repository and documentation or for further questions and feedback, please email to Zerrin Yumak.

References
(1) http://lipsync.rogodigital.com/
(2) https://www.assetstore.unity3d.com/en/#!/content/4
(3) https://www.cereproc.com/
(4) http://www.mindmakers.org/projects/bml-1-0/wiki
(5) https://vhtoolkit.ict.usc.edu/
(6) http://rageproject.eu/developing-scenarios-for-training-communication-skills/
(7) http://rageproject.eu/playing-with-emotions/
(8) http://rageproject.eu/on-our-way/
Posts are written and signed by its authors. RAGE Project does not responsabilize for the opinions and comments published in them.