Skip to main content

Lip Sync

Lip sync makes an agent's mouth move while a voice line plays. The dialogue system does not analyse audio itself -- it drives a third party lip sync package through a small provider component, and starts and stops that package around dialogue audio.

Providers are shipped for uLipSync and OVRLipSync, and you can write your own.

How it fits together​

When an audio beat plays, the chain is:

  1. ChatBeatPlaybackService starts the clip on the speaker's AudioSource and calls StartLipSync(audioSource) on that agent's presentation service
  2. DefaultDialoguePresentationService forwards the call to the ILipSyncProvider it found on the agent
  3. the provider enables the lip sync component, which reads the audio and moves the mouth
  4. when the beat finishes or the conversation stops, StopLipSync() disables it again

The provider is found by LipSyncResolver, which looks for any MonoBehaviour implementing ILipSyncProvider on the agent's own GameObject first, then on its parents. Nothing is registered in module settings -- lip sync is per agent, and an agent with no provider simply plays audio with a still face.

The AudioSource is created on the agent's root GameObject when it joins a conversation. This matters: the uLipSync component reads audio through OnAudioFilterRead, so it has to sit on that same root GameObject to hear anything.

Installing uLipSync​

uLipSync is not bundled with CIVIL-DIALOGUE-SYSTEM -- it is a separate MIT licensed package that you install yourself.

  1. open Window > Package Manager
  2. choose + > Add package from git URL
  3. enter https://github.com/hecomi/uLipSync.git#upm

The dialogue assemblies detect the package by name (com.hecomi.ulipsync) and define DIALOGUE_ULIPSYNC automatically. There is no scripting define to set by hand. Once the package finishes importing, the provider components and the setup menu items appear.

If the menu items under BardTreeLtd/Dialogue/Setup/ are missing, the package has not been picked up -- check that it is listed in Package Manager and let the editor finish recompiling.

Setting up an agent (blend shape)​

Use this route when your character has mouth blend shapes on a SkinnedMeshRenderer. It gives the best results.

  1. select the root GameObject of the agent -- the same object that has the agent and dialogue components, not the face mesh child
  2. run BardTreeLtd/Dialogue/Setup/Add uLipSync (Blend Shape) to Selected
  3. on the uLipSync component, assign a Profile. The package ships samples at Packages/uLipSync/Assets/Profiles/ -- uLipSync-Profile-Sample-Male and uLipSync-Profile-Sample-Female are good starting points
  4. on the uLipSyncBlendShape component, assign your face SkinnedMeshRenderer
  5. map the phonemes A, I, U, E, O to the matching mouth blend shapes on that renderer

The menu item adds uLipSync, uLipSyncBlendShape and ULipSyncProvider, and wires the provider's lipSync field for you. Steps 3 to 5 are per character and cannot be automated -- blend shape names differ between models.

Setting up an agent (jaw bone)​

Use this route for characters with no mouth blend shapes. It rotates a jaw bone by audio volume, which is cruder but works on any rigged head.

  1. select the root GameObject of the agent
  2. run BardTreeLtd/Dialogue/Setup/Add uLipSync (Jaw Bone) to Selected
  3. on the uLipSync component, assign a Profile
  4. check the jaw bone assignment on ULipSyncJawBoneProvider

The setup tool searches the rig for a bone named exactly jaw and assigns it if found. Most rigs use another name (Bip01 Jaw, mixamorig:Jaw, DEF-jaw), so expect to drag the bone in yourself -- the Console message tells you which happened.

Then tune the movement on ULipSyncJawBoneProvider:

  • Open Rotation -- the euler rotation applied at full volume, default (0, 0, -15). If the jaw opens sideways or backwards, this is the axis to change
  • Volume Scale -- how loud a line has to be to open the jaw fully. Lower it for quiet voice lines
  • Smoothing -- how quickly the jaw follows the audio. Higher is softer and laggier

Using OVRLipSync​

OVRLipSyncProvider targets Meta's Oculus Lipsync package. Because that is a manual download rather than a registry package, it cannot be auto detected the way uLipSync is, so this route needs two manual steps:

  1. import the Oculus Lipsync package into the project
  2. add DIALOGUE_OVRLIPSYNC to Scripting Define Symbols in Project Settings > Player, and add the Oculus Lipsync assembly to the BardTreeLtd.Dialogue assembly definition under Assembly Definition References

Then add OVRLipSyncProvider to the agent root alongside an OVRLipSyncContext, and assign the context on the provider. There is no setup menu item for this route.

Writing your own provider​

The provider contract is deliberately small:

csharp
public interface ILipSyncProvider
{
    void OnDialogueAudioStarted(AudioSource audioSource);
    void OnDialogueAudioStopped();
}

Implement it on a MonoBehaviour, put that component on the agent root, and the resolver will find it. The shipped providers just toggle a component on and off, but you are free to do anything -- trigger a viseme animation, drive a shader parameter, or play a generic talking animation with no audio analysis at all.

A provider on a parent object is also found, which is useful when the agent is a child of a larger rig.

Troubleshooting​

The mouth never moves. Confirm all three components are on the same root GameObject as the AudioSource. DefaultDialoguePresentationService caches the provider in Awake, so a provider added to a live agent mid-play is not picked up until the object is re-created.

The provider component is missing from the Add Component menu. The uLipSync package is not installed or has not compiled. The provider files are compiled out entirely without it, so they will not appear.

Audio plays but the uLipSync component stays disabled. The provider disables the lip sync component in Awake by design and only enables it while a line is playing. If it never enables, the presentation service is not finding the provider -- check the object hierarchy.

The mouth moves for the wrong character. Each participant gets its own AudioSource and provider. A shared AudioSource parented above several agents will drive whichever provider the resolver reaches first.

Blend shapes twitch or barely open. This is profile and mapping work rather than a dialogue system issue -- calibrate the profile against your own voice lines using the uLipSync package's own tooling.

Next step​

Continue to UI and Theming, or see Beats for how audio beats are authored.

Last updated on