Lip sync for cartoons: making mouths match Vietnamese speech

The simplest way to lip sync a cartoon is to drive the mouth from the loudness of the audio: when the voice is loud, the mouth opens; when it's quiet, it closes. That's how the mouths work in Duma Meow, my Vietnamese cartoon about four cats, and it suits Vietnamese well because the language is built from short, clearly separated syllables.

Here's how loudness-driven lip sync works, where it shines, where it falls short, and the habits that keep it looking right.

Two main approaches to lip sync

Animators usually choose between two methods:

• Phoneme or viseme sync. Each speech sound gets a matching mouth shape, like a wide mouth for "ah" or closed lips for "m." It's detailed and expressive but takes much more work.
• Loudness-driven mouth flaps. The mouth opens and closes based on how loud the audio is at each moment. It's quick, consistent and easy to automate.

For short comedy clips, mouth flaps are often enough. Viewers' eyes go to the face and the reaction, and a mouth that moves on the beat of the voice reads as talking.

How loudness-driven flaps work

The idea is simple. The voice track is measured frame by frame. If the volume is above a quiet threshold, the mouth shows an open shape; louder sound can mean a wider mouth. Below the threshold, the mouth closes.

A few settings make a big difference:

• The threshold. Too low, and breathing or background noise flaps the mouth. Too high, and soft words get skipped.
• The number of mouth shapes. Two shapes (open and closed) look classic and cartoony. Three or four (closed, half, open, wide) look smoother.
• Smoothing. A little smoothing stops the mouth from flickering open and shut on every tiny bump in the audio.

Why Vietnamese suits mouth flaps

Vietnamese is often described as a monosyllabic language at the level of sound: speech is made of distinct syllables, and each syllable carries a tone. Listen to a line like "Ối giời ơi..." or "Rồi thì, kiểu là, nói chung là..." and you can hear the beats.

That rhythm maps neatly to open-close mouth movement. Each syllable tends to produce its own burst of loudness, so the mouth naturally opens once per syllable. English, with its consonant clusters and unstressed syllables, can blur together more.

Tones don't change mouth shape much, so loudness-based sync doesn't need to "know" the tone. What matters is the timing of the syllables.

Short lines, big reactions

Duma Meow leans on short lines: "Ủa?" (Huh?), "Mười điểm." (Ten points.), "Chờ chút." (Wait a sec.). With lines this short, the mouth might only open two or three times. That puts more weight on the pose and expression around the mouth.

Some lines barely need lip sync at all. "Haizz." is a sigh, so a slow open and a slump does the job. A shout like "Á á á á!" is one long, loud sound, so the mouth simply stays wide. See writing jokes for cats who barely talk for how short lines shape the comedy.

Why trim, never stretch

Every voice line in Duma Meow is trimmed to fit, never time-stretched. Time-stretching changes how long the audio lasts, and it can warp the voice and smear the syllables. Since tones in Vietnamese carry meaning, distorting the pitch contour is a real risk.

Trimming keeps the voice natural. If a line is too long for a shot, the fix is to cut silence, adjust the edit or rethink the shot, not to squeeze the audio. Clean, unstretched audio also gives the loudness-based mouth a clearer signal to follow.

Tips for cleaner lip sync

• Trim silence at the start and end of each line so the mouth doesn't open early.
• Remove noise before the sync step; background hiss can trigger flaps.
• Close the mouth on pauses. Commas and ellipses should show a closed mouth.
• Add a hold after a big shout so the face can react.
• Animate the head and eyes too. A small head bob on stressed words sells the speech.
• Check at phone size. In a vertical 1080×1920 Short, small mouth details disappear, so exaggerate.

When to go beyond flaps

If a character sings, speaks long sentences in close-up, or needs precise expressions, a phoneme-based approach is worth the extra effort. For quick comedy beats, the time saved with flaps is better spent on poses and timing.

Getting started

Record or generate a short line, trim the silence, and try a simple two-shape mouth driven by loudness. Watch it at full speed rather than frame by frame; lip sync is judged by the eye in motion. For more on the voices themselves, see making a Vietnamese cartoon with AI voices.

Start with one three-syllable line and two mouth shapes; if it reads as talking at full speed, you're done.