Dubbing

Dubbing is an AI dubbing studio built into the console. It takes a finished video in one language and carries it to broadcast-ready cuts in other languages — transcript, translation, synthetic voices, the audio mix, on-screen graphics, and optional lip-sync — in one editor, with a human in control of every step.

A traditional dubbing chain is scattered across specialists: a transcription house, a translator working against timecodes, voice talent in a booth, an audio engineer rebuilding the mix around the surviving music bed, a graphics operator re-versioning the lower thirds, and weeks of round trips between all of them. Dubbing folds that chain into a single surface where the source transcript, the timed translation, the generated voices, the stems, and the final picture are all views of the same project. The AI does the heavy lifting at each stage; nothing advances to the next stage until an editor has had the chance to review and correct it.

This document is a brief on how Dubbing works. The complete reference — the data model, segment semantics, mixing rules, and version schemas — lives in Dubbing Reference. For authoring the HTML templates that render translated lower thirds, see Designing Lower-Third Templates.

From Source to Dubbed Cut

A project starts from a video already in the media library and ends with versioned, downloadable dubbed cuts. Every long-running stage runs in the background with live progress — leave the page mid-render and the editor re-attaches to the running work when you return.

One Source, Many Languages

A dubbing project is one source item — the original video plus everything learned about it — and any number of language versions built on top of it. Creating the first language version locks the source transcript permanently: every translation downstream is anchored to the source segment timing, so the foundation stops moving the moment something is built on it.

Each language version then lives its own life — its own translations, voice assignments, generated audio, translated graphics, and rendered cuts — fully independent of its siblings. Current target languages: Arabic, English, French, and Farsi.

The Transcript

Setup runs three AI passes over the source video, together or individually:

• Transcription with speaker diarization — the spoken word becomes timed segments, each attributed to a speaker (labeled A, B, C and renameable), with obvious transcription slips auto-corrected and logged.
• Audio separation — the soundtrack splits into a clean speech stem and a background stem (music, effects, ambience). The background survives into every dubbed mix; the speech stem feeds segments that keep their original voice.
• On-screen text detection — a vision pass inventories burned-in text, covered below.

The transcript is then an editable working document: fix text, retime segments with frame-accurate timecode fields, split a segment at the cursor, merge neighbors, insert into gaps, and switch to a per-speaker timeline view that makes crosstalk and overlaps visible lane by lane.

Every segment carries a type — the dubbing treatment it will receive:

Speaking segments are assigned a voice from the project's voices library — per segment, or for every segment of a speaker in one move.

Translation Built for the Clock

Dubbing translation is timing work as much as language work. Each segment's translation is generated against its slot duration and the target language's natural speaking rate — compressing or expanding phrasing so the line can actually be spoken in the time available, with tighter tolerances on short lines than long ones. Expressive delivery tags (urgency, relief, hesitation) ride along in the text and steer the voice synthesis.

Translations are reviewed like any other copy: edit them directly, or re-translate any segment with extra instructions ("more formal", "keep the institution's official name"). The editor tracks provenance — if the source text changes after translation, the translation flags stale; if a translation or voice changes after audio was generated, the audio flags stale too. Nothing silently drifts out of sync.

Voices and Audio

Each segment's line is synthesized in its assigned voice, then tempo-fitted to its slot and locked to the exact duration — the editor warns when a take had to stretch or squeeze beyond comfort, which is usually a cue to shorten the translation.

The compile step assembles the full-length audio: every take placed at its true timecode, dub segments ducking the original speech underneath them (with a per-segment level), CC segments slicing the original voice back in, and the separated background bed underneath everything. It produces a master mix plus stems — one per voice, one per CC speaker, one for the background — so the mix can be inspected track by track, taken into an external audio suite, and reused by lip-sync. Compiles are versioned; with or without the background bed.

On-Screen Graphics

The detection pass inventories every piece of burned-in text — lower thirds, titles, subtitles, logos, tickers — with its on-screen time window, position, movement, and category. The inventory is reviewable on the player itself: drag and resize boxes, retime entries, recategorize, add what the model missed.

Each language version translates that inventory, and translated lower thirds render through an HTML template into transparent animated clips — previewed over a checkerboard before they are committed to a final cut. Templates are plain HTML/CSS with full typographic control; see Designing Lower-Third Templates.

Lip-Sync

Optionally, the dubbed cut can be lip-synced: an AI model re-animates the speakers' mouths to match the dubbed audio. The generation is driven multi-track — each voice's stem is a separate input, mapped to the picture segment by segment — so the model only touches frames where someone is actually being dubbed. CC and discarded ranges keep their original frames, untouched.

Lip-sync generations are long; they are resume-safe end to end. If anything interrupts the handoff after the render finished, the editor offers to retrieve the finished result rather than rendering again.

The Final Cut

The combine step produces the deliverable: pick a base — the compiled audio laid over the source picture, or a lip-synced video — and a rendered graphics version, or none for a clean cut. Lower-third clips are burned in only during their time windows.

Every stage versions its output — compiled audio, lip-synced video, rendered graphics, final cuts — and every version records its lineage: a final cut knows which graphics version it burned and which audio or lip-sync base it sits on. Bases and graphics mix and match freely without re-running the stages upstream.

Review and Export

The editor plays the work at every stage: original or dubbed audio against the picture (kept in sync while scrubbing), continuous segment-by-segment playback of the generated takes, per-stem listening on any compile, and in-place preview of every rendered video. The transcript and translations export to CSV, SRT, WebVTT, and a print-ready PDF dubbing script — discarded segments excluded, right-to-left scripts handled throughout.

For the underlying detail — every field, every mixing rule, every version schema — see the Dubbing Reference.