Drag[en]gine Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{tag>motioncapture democap}}
<WRAP youarehere>
[[:start|Start Page]] >> [[democap:main|DEMoCap: Drag[en]gine Motion Capture]] >> ** Motion Transfer Lip Sync **
</WRAP>

====== Motion Transfer Lip Sync ======

{{democap:motion_transfer_lipsync.png}}

Uses audio analyzis of user speech to apply lip synchronization without the use of face tracking hardware. The result of the lip synchronization provides weights for visemes. These weights are used as vertex position set weights (blend shapes or morphs in other applications) to perform mouth animation. Each viseme is assigned a vertex position set.

Three operating modes are supported. The first applicable mode is used:

  - **Full Viseme**: At least one viseme of the full viseme group is defined. The lip synchronizer provides the weights for each viseme and applies it to the corresponding vertex position set.
  - **Viseme subset**: At least one viseme of the viseme subset group is defined. This provides a subset of mouth poses consisting of 4 basic shapes. This is sufficient for many situations since games tend to have voice acting in one language but the user often reads the closed captions in a different language.
  - **Open mouth move**: An animation move is blended based on the mouth openness value of the lip synchronizer. This is fast to do since it requires only a single animation move. This is best used for characters with elongated snouts or mouth shapes where complex mouth animations are possible to do in a reasonable way.
  - **Jaw bone**: A jaw bone is rotated based on the the mouth openness value of the lip synchronizer. This is the fastest solution not requiring any upfront modification of the character. It has the same use case as //Open mouth move//.

The visemes line up with the visemes defined by VRChat and similar applications.

====== Motion Transfer Menu ======

Click the button ''...'' to show the motion transfer menu. It contains these entries:

===== Auto Rig =====

Automatically fills in vertex position set names using predefined auto-rig definitions. Right-click the motion transfer panel to access the auto-rig options in the context menu.

  * **Auto Rig...**: Opens a dialog to select from a list of predefined auto-rig definitions. The selected definition is applied to fill in the vertex position set names matching the character model.
  * **Auto Rig Best Matching**: Automatically tries all available auto-rig definitions and applies the one that matches the most vertex position sets in the character model. A dialog is shown if no matching definition is found.

Auto-rig definitions are XML files (*.deard) loaded from ''/content/autorig''. The following definitions are provided out of the box for viseme vertex position set assignment:

  * VRChat (VRCFT Unified Expressions)
  * VRoid (VRM)
  * Drag[en]gine

More definitions can be added for example by creating a modification using Mod.IO or creating a feature issue on GitHub and attaching the new file.

====== Name ======

Name of motion transfer to identify it in the list. Name is not required to be unique.

====== Open Mouth ======

Configures how the mouth is driven if no viseme vertex position sets are assigned. The open mouth move and jaw bone parameters are exclusive. The open mouth move takes precedence over the jaw bone if both are set.

===== Open Mouth Move =====

Animation move from the character animation file to blend for opening the mouth. The blend factor of the animation move is driven by the detected mouth openness ranging from fully closed to fully open.

===== Jaw Bone =====

Bone to rotate as an alternative to an open mouth animation move for driving jaw flap. Click on the label or the combo box to activate bone selection mode. Click in the preview window on a bone to select it. The selected bone is marked in red.

===== Jaw Rotation =====

Rotation axis of the fully open mouth along the X axis (red axis). Click on the label or the text widgets to activate coordinate system mode. A coordinate system gizmo is shown in the preview window at the position of the jaw bone. Rotate the coordinate system until the X axis (red arrow) aligns with the jaw rotation axis.

===== Jaw Angle =====

Rotation angle in degrees of the fully open jaw bone.

====== Viseme Simple ======

Assigns vertex position sets for the basic subset of four visemes. These four visemes are sufficient for basic lip sync animation when a complete set of mouth shapes is not available in the character model. Each combo box shows the vertex position sets defined in the character model. Assign the vertex position set matching the mouth shape for the respective viseme.

===== Silence =====

Neutral mouth position used when no sound is produced. Assign the vertex position set representing the default closed or resting mouth shape. If no //All Visemes// viseme is set this is also used for closed mouth sounds like ''m'' or ''p''.

===== AA =====

Open mouth shape. Used for back open vowels, for example ''a'' in "father" or ''o'' in "hot".

===== OH =====

Rounded open mouth shape. Used for back rounded vowels, for example ''o'' in "go" or ''aw'' in "saw".

===== E =====

Slightly open mouth shape. Used for mid front vowels, for example ''e'' in "red" or ''eh'' in "bed".

====== All Visemes ======

Assigns vertex position sets for all visemes not covered by the simple subset. Using the full set of visemes produces more nuanced and natural looking lip sync animation. The character model needs vertex position sets matching each assigned viseme. Each combo box shows the vertex position sets defined in the character model. Assign the vertex position set matching the mouth shape for the respective viseme. Visemes left unassigned are ignored.

===== IH =====

Narrow horizontal mouth shape. Used for close front vowels, for example ''i'' in "bit" or ''ee'' in "bee".

===== OU =====

Rounded and nearly closed mouth shape. Used for close back rounded vowels, for example ''u'' in "you" or ''oo'' in "too".

===== PP =====

Closed lips shape. Used for bilabial consonants, for example ''p'', ''b'' and ''m''.

===== FF =====

Lower lip touching upper teeth. Used for labiodental consonants, for example ''f'' and ''v''.

===== TH =====

Tongue near or between the teeth. Used for dental consonants, for example ''th'' in "think" and ''th'' in "the".

===== DD =====

Tongue tip raised near the upper teeth and alveolar ridge. Used for alveolar consonants, for example ''d'', ''t'' and ''l''.

===== KK =====

Back of tongue raised toward the velum. Used for velar consonants, for example ''k'', ''g'' and ''ng''.

===== CH =====

Rounded mouth with tongue near the hard palate. Used for palato-alveolar consonants, for example ''ch'', ''j'' and ''sh''.

===== SS =====

Teeth nearly closed with air passing between them. Used for alveolar fricatives, for example ''s'' and ''z''.

===== NN =====

Near-closed mouth with air flowing through the nose. Used for nasal consonants, for example ''n'' and ''ng''.

===== RR =====

Slightly open and rounded mouth. Used for rhotic consonants, for example ''r''.
Drag[en]gine Wiki

User Tools

Site Tools

Page Tools
