Authoring CharacterCreation

Project YALLAH - Authoring a Virtual Character

This document is a chapter of the YALLAH Manual series. In this chapter we describe how you can use the YALLAH system to create a new virtual character.

This chapter assumes that the author has a basic knowledge of the Blender 3D authoring software, in particular how to configure the layout of the GUI, select objects, investigate properties, and load and execute scripts. To start with the use of Blender you can watch the Official Blender tutorials on Youtube.

YALLAH consists of a selection of open-source tools and a set of scripts that, altogether, enable for the generation of a new virtual character in less than 1 hour. The produced character will be ready to be embedded in real-time engines.

The process is divided in two steps.

1. The first step, Character Generation, consists of creating a new character using the ManuelBastioniLAB add-on for Blender:

   • This includes editing the character's proportions.

2. The second step is the Character Preparation for Unity:

   • Setup the character for real-time execution (fixing materials and adding blend shapes and/or bones needed at runtime for extra functionalities.);
   • Adding clothes and animations.

Character Creation

The first step consists of creating a new character with the software MB-Lab, which is a Blender plugin. Then, a set of scripts must be executed. Each script fixes a specific aspect of the character data (skeletal structure, Shape Keys, ...) as required by the various functionalities (e.g., eye-gaze, talking, ...).

Install the prerequisites

• Install the Blender 3D authoring software (tested with v2.93.5).

• Install the MB-Lab addon for Blender and enable it (tested with [v1.7.8.5].

  • (To learn how to install Blender addons: https://docs.blender.org/manual/en/latest/editors/preferences/addons.html)

• Download the YALLAH SDK YALLAH_SDK-yymmdd.zip from https://cloud.dfki.de/owncloud/index.php/s/WxjR8X4MH7qGtxn and extract it;

• Install and enable the yallah Blender add-on.

  

• Download the YALLAH-Assets-YYMMDD.zip if you want to provide your character with clothes and animations.

Choose a name

Choose a name for your character. If this example we will follow with the name Anna.

Generate the character

• Create a character using ManuelBastioniLab.
  • 
  • In the 3D View, locate the tab MB-Lab in the (left) tool panel
  • Select either Cuacasian female (F_CA01) or Caucasian male (M_CA01)
  • Uncheck the option Use Cycles engine
  • Click Create character

Customize your character

At this point, you can take advantage of the MB-Lab tool to customize the size and proportions of your character. You can use the Random Generator or the Body, face and measure parameters to customize your character. The GUI is easy asn self-explanatory.

This is an artistic job. It's your job. How to do it is out of the scope of this document. Please, refer to the MB-Lab Documentation to learn how to customize your character.

Finalize the character

• Go to the Finalize tools tab
• Uncheck Save images and backup character (Well, we don't need tha images, but if you want to store your character, it won't be anyway needed later)
• Uncheck Remove modifirers
• Type The name of your character as prefix, e.g.: Prefix: Anna
• Click Finalize
• 

P.S.: It is very important that you do not save the Blender scene with the same name of the Armature object. If you name, e.g., the scene Anna_armature.blend and the armature Anna_armature, Unity is going to mess up with the animation system!

At this point you can save the scene (e.g., as Anna.blend) and re-open it later if you want to apply different clothes and/or animations.

Character Preparation

After MB-Lab finalization, you proceed with further steps to make the character ready for Unity.

Setup the character

The YALLAH Setup consists of adding bones, ShapeKeys, blendshapes, and adjusting material definitions according to the neededs of the motion controllers and other features used in real-time rendering. The character setup steps consist of several sub-steps. Some are mandatory, some others are optional -- needed only if you plan enable specific functionalities. For example, in order to enable the Text-to-speech functionality, you need to customize the character (run a Blender script) to create the visemes of the character. Afterwards, in Unity, you need to attach a C# script as component to the character Mesh.

We offer a Setup a MBLab character Operator that applies all of the sub-steps at once.

• Go to the Yallah tools panel
• Select the mesh object (e.g., Anna_body)
• Click on Setup a MBLab character
• 

If you need to skip some of the preparation steps, or if you want to have an idea of what happens to your character, the following subsections document all the steps performed by the Setup script. If you don't need special customizations or optimization, just skip to the next section. (However, if you want to contribute to the YALLAH project, you better read here first.)

(Setup) Create the visemes for integration with MaryTTS

Creates the ShapeKeys for the phonemes.

• Select the mesh object (child of the armature) and stay in Object Mode.
• Load and Run the script Features/MaryTTS/Setup.py
  • The phonemes are taken from MBLab 1.6 and defined as mix of the basic ShapeKeys.
  • The definition of the phonemes is stored in file Features/MaryTTS/PhonemesMBLab1_6.json.
  • This is done through the CreateShapeKeys operator: bpy.ops.object.create_shape_keys(shape_keys_filename="Features/MaryTTS/PhonemesMBLab1_6.json")

(Setup) Create the Facial Expressions

Create the ShapeKeys for the Facial Expressions, each starting with 'fe_'.

• Select the mesh object (child of the armature) and stay in Object Mode.
• Load and Run the script Features/FacialExpressions/Setup.py
  • The expressions are taken from MBLab 1.6 and defined as mix of the basic ShapeKeys.
  • This is done through the Create ShapeKeys from File operator: bpy.ops.object.create_shape_keys(shape_keys_filename="Features/FacialExpressions/FacialExpressionsMBLab1_6.json")

(Setup) Eye bones

The following script will setup bones to pilot the eyes of the character. Actually, the bones will not be used to rotate the eyes (MBLab does it using ShapeKeys), but rather to query the position of the center of the eyeballs, which is needed to compute the direction of the gaze.

• Select the mesh object (child of the armature) and stay in Object Mode.
• Load and Run the script Features/EyeGaze/Setup.py

(Setup) Camera requirements

The following script will setup a bone at top of the character's head. It is useful to adjust the distance and position of a camera according to a requested shot type (e.g., full shot or extreme close-up).

• Select the mesh object (child of the armature) and stay in Object Mode.
• Load and Run the script Features/Camera/Setup.py

(Setup) Real Time Requirements

Blender modifiers prevent the importation of the ShapeKeys in Unity. The code essentially removes all of the object modifiers, except the armature modifier for skeletal animation mbastlab_armature.

• Select the mesh object (child of the armature) and stay in Object Mode.
• Load and Run the script Features/RealTimeRequirements/Setup.py

(Setup) Create an A-Pose action

We create an action named A-Pose, which is composed of a single keyframe at position 1, where all the transformations and cleared (no rotation, translation, or scaling). This is needed to configure the real-time animation controller in Unity.

• Select the armature object and stay in Object Mode.
• Invoke the operator bpy.ops.yallah.create_apose_action()

Import clothes (Optional)

Clothes are available in external blender scenes. You can find those scenes in the YallahAssets-yymmdd.zip archive under YallahAssets/Clothing.

Import the clothes by appending the objects from the Blender scene. For each piece of clothes you want to add:

• Import the clothes:
  • From the menu Select File --> Append
  • Locate and Select the scene containing the clothes (e.g.: WomanClothes-02-CasualJeansAndShirt.blend)
  • During the selection, enter the scene (remember, a .blend file is seen as a directory), enter the Object directory
  • Select the piece of clothes you want to add (e.g., Bottoms, Tops, and Shoes. Shift+click for multiple selection).
  • Click Append from Library
  • 
• Configure the clothes:
  • Set the parent of the clothes object (e.g., Bottoms) to the Armature object (e.g., Anna_armature)
  • 
• Fit the clothes to the character:
  • Open the ManuelBastionLIB tab and the panel AFTER-CREATION TOOLS -> PROXY FITTING
  • Select the Character mesh (e.g., Anna_body)
  • Select the piece of clothes as Proxy (e.g., Bottoms)
  • Leave Offset: 0 and Influence: 20
  • Select Fit Proxy
    • This will also automatically add an Armature modifier to the fitted object (e..g, Bottoms).
  • 
• Repeat for each piece of clothes you want to add.

By selecting the armature in Pose mode, you can now verify that the clothes move together with the body.

Import Hair (Optional)

It works exactly as per the clothes. In short:

• From the menu Select File --> Append
• Locate, e.g., YallahAssets/Hair/WomanHair-02-MiddlePart.blend
• Select Object/Hair and Append from Library
• Fit the Hair to the character (Fit Proxy, exactly as you did for the clothes)

Remove unneeded vertices

While animating the character, some skin might get out of the clothes. In this case it is better to remove the vertices under the clothes. It has the double advantage to avoid skin-clothes compenetration and increases the rendering performances.

• Select the mesh object (e.g., Anna_body)
• Go to the Yallah panel and Click Load Clothes Vertex Groups
• 
• Enter edit mode and use the vertex groups to quickly select the vertex type that you want to delete.
• ATTENTION! According to the type of clothes that you use, more or less vertices must be deleted. Very likely, you MUST manually customize the vertex selection.

(It is also possible to take advantage of MBLab automatic vertex masking system, which automatically detects the vertices under the clothes. But in this case you will delete only a limited set of vertices.)

Import animations (Optional)

We have a number of animations that can be imported into the newly created character. These animations are stored in separated Blender scenes and can be imported by appending Actions:

• Idle animation:
  • From the menu Select File --> Append
  • Open an animation file, e.g., Woman_Idle.blend
  • Enter the Action folder
  • Select, e.g., IdleSlowBreath
  • Click Append
• Other animations animation:
  • From the menu Select File --> Append
  • Open an animation file, e.g., Woman_AnimationSet1.blend
  • Enter the Action folder
  • Select, e.g., Salsa, Salute, and Waving
  • Click Append
• Locomotion animation:
  • From the menu Select File --> Append
  • Open an animation file, e.g., Woman_WalkSet2.blend
  • Enter the Action folder
  • Select:
    • 1 animation to walk forward, e.g. Walk;
    • 2 animations, to turn left and right , e.g., LeftTurn and RightTurn
  • Click Append

For each Action you must force the Action to stay in the scene. We have a comfortable Button for this:

• Select the armature.
• Click on the Set 'F' (dumy user) to all actions
• 

Pack and Save

• Bring all textures inside the Blend file.

  • Pack data: File -> External Data -> Pack All Into .blend

• Save the scene with a filename corresponding to the name you chose for the character.

  • E.g.: Save to Anna.blend

Export as FBX

The most reliable way to use a Blender character in Unity is to first export it in the FBX format.

• Export the FBX file File -> Export -> FBX (.fbx)
• Configure the export option:
  • Apply modifiers option prevent exporting shape keys. This is the most important option that a user should untick in FBX export menu;
  • Leaf bones are not needed for Unity;
  • (Optional) Copy path will copy all materials to the path since we are exporting a binary FBX.
  • 
  • 

Final Remarks

At this point the FBX scene in ready to be imported in a Game Engine. See the next guide Manual - Authoring - Unity Deploy to use the created character in the Unity Game Engine.