Skip to content

Authoring UnityDeploy

Jump to bottom
Fabrizio Nunnari edited this page Mar 8, 2022 · 8 revisions

Setting up the character in Unity

This document explains how import in Unity a character created in the main Authoring procedure.

Prerequisite

Download the YALLAH SDK from https://cloud.dfki.de/owncloud/index.php/s/WxjR8X4MH7qGtxn, extract the zip, and import the YALLAH_Unity-yymmdd.unitypackage package into your Unity project.

The package will add the following folder to your Assets:

/YALLAH/
  Characters/
    MBLab-F_CA01/
    MBLab-M_CA01/
  Scenes/
    TestMaleBase.unity
    ...
  Scripts/

The Characters folder contains two basic, naked, characters; one male and one female, generated using the default body proportions.

The Scenes folder contains Test scenes that put both characters in a minimalistic demo stage. They are stress tests activating all functionalities at the same time. Don't be surprised if the characters look psychotic.

The Scripts folder contains the Motion Controllers of the characters, i.e., the scripts enabling their interactive animation.

The rest of this document explains the procedure to apply the scripts to a freshly imported character.

Add the FBX Scene to your Assets

  • Copy the character scene (e.g., Anna.fbx) into a dedicated directory, e.g.: MyUnityProject/Assets/Characters/Anna/

Configure Import settings

Select the prefab model /Assets/YALLAH/Characters/Anna/Anna.blend and in the inspector, configure:

  • Import Settings -> Model

    Leave the default values

    Unity Import Mode

  • Import Settings -> Rig

    • Animation Type Generic
    • Avatar Definition: Create From This Model
    • Root node: XXXArmature (the parent of the root node)

    Unity Import Rig

  • Import Settings -> Animation

    • Leave the global import settings at default

      Unity Import Animation

    • Configure each animation (Important for locomotion and other looping animations)

    • Adjust the name. Unfortunately, the Blender FBX exported prepend the armature name to each animation. You might want to remove such prexif to have an easier life later when addressing animations by name;

    • Check Loop Time for each animation that has to loop (e..g, WalkCycle, TurnLeft and TurnRight)

    • Check the flag Root Transform Rotation for every animation that is NOT supposed to rotate the character (e.g., Idle, WalkCycle)

      • This avoid for example that a character WalkCycle steers on a side because of accumulation of errors.

    • Check the flag Root Transform Position (Y) for every animation that is NOT supposed shift the character vertically (e.g., Idle, WalkCycle, TurnLeft/Right)

      • This avoids for example that an error accumulation lifts the character in the sky while looping an Idle animation

    • Check the flag Root Transform Position (XY) for every animation that is NOT supposed translate on the horizontal plane (e.g., Idle, Wave Hello)

      • This avoids for example that an error accumulation slides the character on the floor.

    • Please, see the Unity manual on Root Motion to understand the details.

    Unity Import Animation

  • Import Settings -> Materials

    Unity Import Materials

    • For each of the character's material: drag-n-drop the pre-packed material definition from:
      • for male character: YALLAH/Characters/MBLab-M_CA01/Materials/
      • for female character: YALLAH/Characters/MBLab-F_CA01/Materials/
    • For clothes, see the following section.

Fix the clothes materials

Note, you can do this operation also later, when the chracter is visible in the scene.

Note 2, this is subject to change in the future, as we plan to distribute pre-configure Unity materials together with the assets.

Open the Assets's Avatar (e.g.: Anna) and go through the meshes of the clothes (e.g., Top, Shoes, Bottom, ...). For each of them, select the materisl and fix it:

  • Albedo color: (255,255,255,255)
  • Albedo: set the diffuse texture
    • (drag it from the "textures" directory)
  • Normal Map: set the normal texture

Unity Fix Clothes Material

Fix the Hair material

Note, this is subject to change in the future, as we plan to distribute pre-configure Unity materials together with the assets.

  • Select the ..._DiffuseWithAlpha.png texture:
  • In the inspector, enable Alpha Is Transparency
  • Select the Hair Mesh.
  • Set the Albedo texture to ..._DiffuseWithAlpha.png.

Now, according to the hair model, you have two options to fix the material properties.

Option 1: Nice transparency, bad visibility.

  • Set Rendering Mode to Fade

The transparency will be nice, but if you look the hair from inside the head, the backface culling will let you see outside the head. With some hair model and camera angles, this is not acceptable.

Option 2: Bad transparency, good visibility.

  • Set Rendering Mode to Cutout
  • Set Alpha Cutoff to a vale as low as 0.01
    • This will at least adjust the rendering of hair tips
  • After putting the character in the scene:
    • Add Component -> DuplicateMeshFaces.cs
    • It will enable the visibility of the polygons from both sides after starting the game.
    • If you use Unity 2019 you can get: Not allowed to access triangles/indices on mesh 'Hair(Clone)' (isReadable is false; Read/Write must be enabled in import settings).
      • To solve, go back to the Model import panel and check Read/Write Enabled.

The DuplicateMeshFaces.cs doesn't work properly with transparent materials.

Add a character to a scene

Put a character in the scene, e.g., by dragging the Anna.blend asset in the Hierarchy Window.

Unity Hierarchy and Character

Create a prefab?

This is a good moment to save the character as a prefab, in order to have all the manual configuration done, ready to be exported as package, but before adding scripts.

Since Unity 2018.3 you can do this using Prefab Variants (Before that, creating a prefab from a character would break the original prefab defined by the Blender import. See: https://docs.unity3d.com/Manual/PrefabVariants.html). Simply drag the character from the scene layout back into the asset view and when prompted choose Prefab Variant.

Scripts for interaction with MaryTTS

  • Select the Mesh (e.g., AnnaMesh)

  • In the inspector Add Component: MaryTTSController.cs.

    Configure MaryTTS

The script YALLAH/Scripts/tts/MaryTTSController.cs uses MaryTTS as web service.

The controller sends the sentence text and gets back a WAV file and the realized durations (a table with timecode-phoneme pairs). This architecture might introduce delays of the server is slow to respond. It is advised to install MaryTTS on your local machine if quick reactions to speech orders must be guaranteed.

You can also specify Additional HTTP Request Parameters. For example:

&effect_Volume_selected=on&effect_Volume_parameters=amount:1.5;

See: http://mary.dfki.de:59125/documentation.html, section "Audio effects", for the full documentation.

Eye Blinking

  • Select the Mesh (e.g., AnnaMesh)
  • In the inspector Add Component: Eye Blink Controller.

No configuration is needed. The character will blink at irregular intervals.

Eye gaze

  • Select the Mesh (e.g., AnnaMesh)
  • In the inspector Add Component: Eye Head Gaze Controller.

Select/drag the following bones from the Armature hierarchy (e.g., expand AnnaArmature) in the configuration panel:

  • Left Eye
  • Right Eye
  • Neck

Configure Eye Gaze

Facial Expressions

  • Select the Mesh (e.g., AnnaMesh)
  • In the inspector Add Component: Facial Expression Controller.

The script YALLAH/Scripts/facial_expressions/FacialExpressionsController.cs scans the list of BlendShapes (aka ShapeKeys, in Blender terminology) and makes a list of all blend shapes beginning with the fe_ prefix.

The Basic set of expressions is automatically extracted from the MB-Lab default character. If you want to add more expressions to this controller, simply create your own expressions in Blender and name them with a fe_ prefix.

Additionally, the controller adds a "Normal" facial expression to the list of the fe_xxx. The normal expression is indexed as 0, all the remaining are numbered from 1 onwards.

At run time, you can manually input the facial expression number in the Current Expression Index.

Facial Expression Panel

Animation playback

  • Select the avatar root GameObject (e.g., Anna)
  • In the inspector, locate the Animator component and select Controller: /YALLAH/Scripts/Animation/CharAnimationController.controller. Avatar Animator Panel
  • In the inspector Add Component: Animation Controller.

The Animation Controller script (YALLAH/Scripts/animation/CharAnimationController.cs) centralizes a set of animation and their role, avoiding the need to have a Animation Overrider or the need to create a different controller (State Machine) for each character.

Configuration:

  • Enable Ambient at start: the ambient animation will be enabled when the game starts (Suggested option).
  • Idle Ambient Animation Clip: select a clip where the character is standing and slightly moving, in order to remove the "puppet" effect.
  • Static Pose Animation Clip: select a clip where the character is just standing in a static neutral position, like the reference A-pose.
  • AnimationClips: insert the number of clips you want to support and drag them into the editor panel.
    • For each slot, drag an animation clip, e.g., from the asset /YALLAH/Characters/Anna/Anna(.blend)/Salsa.
    • At runtime, you can click on Play Random Anim to play a random animation from this list.
  • Locomotion (You can skip these if you don't add the locomotion controller. See next section):
    • Walk Cycle: A loopable animation with a walking cycle.
    • Turn Right: A loopable animation stepping and rotating to the right.
    • Turn Left: A loopable animation stepping and rotating left.
    • The three animations must be ideally synchronized in time, i.e., the animations must have the same duration and the feet must touch the floor at the same timestamp.

Animation Controller Panel

Restriction: the animation clips for a character must be picked from the same blender scene prefab. At the moment, it is not possible to use the animation clips of a charater across other characters, even if the skeletal structure is the same. This causes a waste of memory. We will investigate how to optimize the management of the animations and reduce memory footprint.

Script for Locomotion

  • Select the avatar root GameObject (e.g., Anna)
  • In the inspector Add Component: Locomotion Controller.

Locomotion Controller Panel

The script YALLAH/Scripts/animation/LocomotionController.cs allows the character to walk to a target destination.

Configure:

  • Target Position shows where the character would like to walk to.
  • Force Zero Y is needed to force the avatar game object to y=0, in case the animations have a vertical drift.
  • Distance Threshold the character will stop walking when he reaches this distance from the target. Too small values might lead to an infinite attempt to reach the target.
  • Rotation Threshold Degs tolerance, in degrees, in the alignment with the target. Too small values might lead to infinite attempts to orient towards the target.

To test the locomotion, at runtime, manually insert the coordinates of the Target Position and click on the Force Start.

If you get the Animator CharAnimationController from the source instead than from a Unity package, you will need to configure the Animation States to point to a set of DUMMY animations. The name of this dummies is used to set on-the-fly (programmatically) the animation in the state.

Select the avatar and open an Animator tab, and set the states to use the speficied dummies:

  • Base Layer:
    • Idle state: DUMMY_STATIC_POSE_ANIMATION
    • CurrentActionState: DUMMY_CURRENT_ANIMATION
    • AmbientState: DUMMY: IDLE_AMBIENT_ANIMATION
  • Locomotion Layer:
    • Inactive: leave None (Motion)
    • Set the four motions of the WalkBlendTree to IDLE_AMBIENT_ANIMATION DUMMY_WALK_CYCLE_ANIMATION, DUMMY_TURN_RIGHT_ANIMATION, DUMMY_TURN_LEFT_ANIMATION, as shown below
    • Animator Locomotion Configuration
Clone this wiki locally