Skip to main content

Lip-sync limitations and workarounds

Learn where Lip-Sync struggles, from hidden faces to speakers who are off screen, what Dubly handles automatically, and practical workarounds to save credits.

Lip-Sync re-animates the speaker's mouth to match the dubbed audio — but it doesn't work on every shot and knowing its limits saves you credits and frustration. This article covers where Lip-Sync struggles and what you can do about it.


What Lip-Sync can't do

Three things Lip-Sync is often expected to do, but doesn't:

  • It can't sync to audio you supply in the standard workflow. Lip-Sync always uses the dubbed audio Dubly generated for that subdub. Audio files (MP3, WAV, M4A) can't be uploaded at all — see Upload a video: supported formats.

  • It can't run without a translation. Lip-Sync is a step inside a dub, so you always create a dub from a source language into a target language first. There is no lip-sync-only flow and no way to skip the language selection.

  • Merging your own audio into the video first doesn't get you there either. Dubly would treat that track as the source to transcribe and translate — Lip-Sync would then match the mouth to the newly dubbed audio, not to your track.

So if what you need is an existing video matched to a voice track you already have, that isn't part of the standard workflow today. It may still be possible for your project: tell Dubby in the chat what you want to achieve and what audio you have, and our team will look at your case individually and get back to you. For specific projects we can check on request what we can make possible.


Where Lip-Sync struggles

Lip-Sync depends on the AI seeing a face clearly on screen. It tends to produce weak results in these situations:

  • Insufficient footage of the face: The video lacks at least 10 seconds of continuous, clear facial coverage, preventing the AI from properly analyzing and animating the lip movements.

  • The speaker isn't facing the camera (profile shots, quick turns away). Face tracking can lose the mouth.

  • The face is small in the frame (wide shots, crowd scenes, group photos). Detail is too low for realistic mouth movement.

  • The mouth is partially hidden — microphones held close, hands in front of the face, hoodies, masks, heavy beards.

  • Very fast cuts or rapid head motion. The model has less time to re-animate between frames.


What Dubly does automatically

You don't need to mark anything. The pipeline:

  • Detects where a face is visible. Passages with no face visible are passed through unchanged — Lip-Sync is applied only where a face is on screen. You'll later see these passages as blocks on the lip-sync track of the Visual Editor; they don't line up with the spoken segments.

  • Normalizes the video to H.264 MP4 at up to 1920px width and 30 FPS before lip-syncing, so mild encoding quirks in the source don't break the pipeline.

  • Retries segments that fail to process automatically. If a segment still fails, Lip-Sync fails for that whole language — your dubbed video without Lip-Sync stays available.


What you can do

Before you run Lip-Sync:

  • Crop tighter on the speaker's face if it's small in the frame — a tighter composition gives the model more to work with.

  • Cut out non-speaking B-roll — if your video has long stretches where nobody is talking on camera, Lip-Sync isn't adding value there anyway.

  • Avoid strong face occlusion or extreme angles where possible.


Two speakers, only one on screen

In interviews and conversations, someone is often speaking who isn't visible at that moment. Lip-Sync may then move the visible person's mouth to the off-screen speaker's lines as well — it looks as if one person is speaking both voices. Setting the right number of speakers when you create the dub keeps the voices apart, but doesn't solve this on its own.

Here is how to switch lip sync off for those passages. This only works in the Visual Editor, not in the Advanced Editor:

  1. Open your finished dub from My Dubs and go to the Edit Translation tab.

  2. Set the Editor mode to Visual.

  3. Below the segments is the lip-sync track. Click the blocks where the off-screen person is speaking and switch them from Lip sync to Off.

  4. Click Re-render at the bottom to apply the change.

Important: the lip-sync blocks follow the passages where Dubly detected a face, not the spoken segments. Their number and boundaries therefore usually differ (for example 3 blocks for 5 segments), and a block can only be switched as a whole. If a block covers lines from both speakers, switching it off turns lip sync off for both. If that can't be separated cleanly, message Dubby in the chat (bottom right) with the dub link and the timestamps of the passages concerned.

Full details on the lip-sync track: Visual Editor: adjust timing and split segments.


When Lip-Sync fails completely

If Lip-Sync produces a "Failed" status on the dub page, the most common cause is no face detected anywhere in the video. Animation, motion graphics, product videos without presenters, or voice-over-only footage can't be lip-synced.

Lip Sync


Cost reminder

Lip-Sync is billed separately from dubbing at 1 credit per minute per subdub. If a shot clearly won't benefit from Lip-Sync (wide shots, no speaker on camera), skipping Lip-Sync for that language saves you that credit spend — see When should you turn on lip-sync? for the decision framework.


Still problems?

If your source video meets the quality bar above — clear mouth, good framing, stable lighting, consistent frame rate — and you're still facing problems or your lip-sync failed completely, ask Dubby in the chat (bottom right) and include:

  • The dub link

  • A timestamp of the problem segment

  • A short description of the problem

This helps us investigate whether it's something we need to tune on our side.

Did this answer your question?