Dubbing Reference

This document is the complete working reference for Dubbing: every data model field, the segment-type semantics, the translation timing contract, the audio compiler's mixing rules, the lip-sync mapping, the graphics pipeline, and the version schemas.

It is the technical companion to Dubbing, which explains the concepts and workflows. Read that first if you are new to the module; come here when you need the exact behavior of a field, a rule, or a version. Template authoring for translated lower thirds has its own guide: Designing Lower-Third Templates.

Table of Contents

1. Overview
2. The Source Item
3. Segments & Types
4. Language Versions
5. Translation
6. Voice Generation
7. The Audio Compiler
8. Lip-Sync
9. On-Screen Graphics
10. Graphics Rendering
11. The Final Combine
12. Review & Playback
13. Exports
14. Storage & Key Map

1. Overview

Dubbing lives at /inference/dubbing: a project list, and a two-mode editor — source mode for the original-language item and dubbed mode for each language version. Both modes share the same layout: the video player and stage controls on one side, the segment/graphics workspace on the other, with a resizable split that follows the user across devices.

A dubbing project is a small tree of JSON documents in the project's key/value store:

A language version's id is its source id plus the language code ({sourceId}:{lang}, e.g. 449926:en), and that id appears in the editor URL. Target languages: Arabic (ar), English (en), French (fr), Farsi (fa) — the source language is excluded from the list, and a language that already exists cannot be added twice.

Every long-running stage runs as a background task with a deterministic identity per item and stage. Progress is observable by anyone viewing the item, survives page reloads, and re-attaches when the editor reopens. The full key map is in §14.

2. The Source Item

Created from the list page by picking (or uploading) a video from the project's media library. The new item starts as draft with the media title as its reference.

Statuses

The three setup passes

Run together ("Set up media") or individually; each is a background task and each can be re-run later (re-transcribing replaces the transcript; re-separating replaces the stems; re-detecting replaces the inventory):

1. Transcribe — speech-to-text with speaker diarization. Words are grouped per speaker and split into a new segment whenever that speaker pauses for more than 0.8 s; segments from different speakers may overlap in time — that is intentional, and the mixer is built for it (§7). Every segment arrives typed lipsync (§3).
2. Separate audio — splits the soundtrack into the speech and background stems. The background survives into every dubbed mix; the speech stem supplies dub underlays and CC slices. Both stems are playable and downloadable from the editor.
3. Detect on-screen text — the vision inventory of burned-in text (§9). Gated by a confirmation dialog with a cost estimate; limited to videos of 5 minutes or less.

3. Segments & Types

The transcript is an ordered list of timed, speaker-attributed segments:

Type semantics

The type decides the treatment in every downstream stage:

Setting a segment to cc or discard clears its voice. A dub or lipsync segment without a voice renders a red ring; saving still works, but compiling will not (§7).

Editing operations

All transcript editing happens before locking; afterwards the source is read-only.

• Timecodes — every time field accepts HH:MM:SS.mmm, MM:SS.mmm, or bare seconds. Arrow keys step ±1 frame (25 fps), Shift-arrows ±10 frames; the first arrow press pauses the player; a pull button grabs the current player time. Values clamp to the video duration.
• Split — place the cursor in the text and split; the dialog previews both halves with a proportional split time (editable, or "Use player time") and assigns the second half a new id.
• Merge — adjacent segments join: texts joined with a space, notes with ;, and the first segment's speaker, type, and voice win.
• Insert — the divider between rows creates an empty segment filling the time gap.
• Speaker scope — changing the type or voice of a segment whose speaker has more segments prompts for scope: this segment only, all by this speaker (only where unset), or all by this speaker (replace all).

Timeline view

The segment list toggles to a per-speaker timeline: one lane per speaker (ordered by first appearance), blocks colored by type, a time ruler, and a playhead synced to the video. Clicking a block seeks the player and activates the row. The timeline is the surface for spotting crosstalk — overlaps that the list view cannot show.

4. Language Versions

Created from the source item's Language Versions panel. Creating the first one locks the source (status: locked) after an explicit warning — translations are anchored to source segment ids and timing, so the foundation must stop moving. The new version starts as translating and machine translation begins immediately (§5).

A language version stores only the translation layer — everything time- and speaker-related stays on the source:

All four version arrays are ordered newest-first and displayed as v1…vN with a latest badge.

Dubbed segments

Staleness signals

Nothing regenerates silently; instead, three provenance checks flag what has drifted:

5. Translation

Translation runs automatically when a language version is created, as a background task that fans out one sub-task per segment. Segments typed discard and segments with empty source text are skipped. Failed segments render as error rows with individual Retry buttons; the rest of the batch is unaffected. When the batch lands, the version's status returns to draft.

The timing contract

Each segment is translated against its slot, not just its words. The translation model receives the source text, the slot duration, and the language pair, and works to a duration tolerance that scales with the slot — tight on short lines, looser on long ones (on the order of ±0.5 s for lines up to 5 s, widening to ±3 s beyond 30 s). It fits by phrasing: choosing shorter synonyms, dropping redundancies, restructuring — never by cutting meaning. Per-language natural speaking rates anchor the length estimate.

Translations may carry bracketed delivery tags — [urgent], [relieved], and similar — which ride through to voice generation and steer the read.

Re-translation

Any segment can be re-translated at any time from its row, with optional free-text instructions ("more formal", "keep the official title in French"). Instructions apply to that call only and are not stored. The result lands in the editor as an unsaved edit — review, then save.

6. Voice Generation

A take is generated per segment, in the segment's assigned voice. A voice is eligible when its voices-library entry has a chosen preview take — voices without one appear disabled in the pickers.

• Single — each row has a generate (and later, regenerate) control; the take is playable inline.
• Bulk — Generate all missing audio covers every dub/lipsync segment that has a voice, has text, and has no take yet.

Duration fitting

Every take is fitted to its slot in two passes: a tempo adjustment when the natural length is within 0.8×–1.3× of the target (beyond that range the take is trimmed or padded instead), then an exact-length lock so each take is sample-accurate to its slot. dub segments fit to the slot minus their 0.5 s lead-in — the breath of original speech heard before the voice-over enters. The natural pre-fit length is kept (rawAudioDurationSecs) and powers the mismatch warning of §4.

Each take snapshots the text and voice it was generated from, so later edits surface as stale audio rather than passing unnoticed.

7. The Audio Compiler

Compile assembles the full-length dubbed soundtrack from the takes, the original speech, and the background bed. It is versioned, and offered in two flavors: with background and speech only.

Gates

• Every non-discarded dub/lipsync segment must have a voice assigned — the compiler groups segments by voice to build per-voice stems, so a missing voice blocks the compile with the offending segment ids listed.
• There must be something to mix: at least one generated take, or at least one cc segment.

Lanes

Renderable segments are grouped by stem identity:

Within one stem, overlapping segments (possible after manual retiming) are split into parallel sub-lanes so each lane is internally sequential. Each lane is assembled as one continuous track from time zero — silence for the gaps, every piece locked to its exact slot — which preserves absolute timing and lets different speakers genuinely overlap in the mix.

The mix

• The authoritative output length is the longest of: the source video, the speech stem, the background stem, and the last segment's end. Every lane, stem, and the master are locked to it.
• All voice and CC lanes mix at full level.
• Each dub segment contributes an underlay: the matching slice of original speech, at the segment's underlayVolumeDb (default −12 dB), under the take.
• With background enabled, the background stem sits under everything at −3 dB.

Output

A CompiledAudioVersion is prepended to the version list:

The stems are the editorial hand-off: play or download any of them individually, take them into an external audio suite, and they are the exact inputs lip-sync consumes (§8). The compile toast reports telemetry — segment counts by type, lanes mixed, stems produced, and warnings such as a missing speech stem leaving dub underlays silent.

8. Lip-Sync

Lip-sync re-animates the speakers' mouths to match a compiled audio version — pick one (defaults to the latest) and generate. Two modes:

The multi-track contract

• Only voice stems are sent. The background stem (music degrades the model) and CC stems (those frames must not move) are never part of the payload.
• One mapping entry is built per source dub/lipsync segment, pointing the segment's time range at its voice's stem. cc and discard ranges are simply absent from the map — the model leaves those frames untouched, byte for byte.
• Segment boundaries snap to the video's frame grid (at its real frame rate) and keep at least one frame of separation; ranges swallowed by the snapping merge or drop.
• Requirements enforced up front: every mapped segment has a voice, every voice has a matching stem in the chosen compile. A compile predating per-voice stems is rejected with a "recompile audio" message (or use single-track).

Resume safety

Generations are long, and the generation id is persisted the moment it exists:

• An interrupted wait re-attaches to the same generation instead of starting a new one.
• If the render finished upstream but the result never made it back, the version panel surfaces an amber Retry download action that fetches the finished output without re-running the generation. The pending marker clears on success.

Output

A LipsyncedVideoVersion: { videoFileId, durationSeconds, generatedAt, sourceAudioFileId, model, mode }. The sourceAudioFileId ties the video back to the compiled audio version that drove it — shown as a "from Audio vN" chip, and used by the final combine's re-mux rule (§11).

9. On-Screen Graphics

Detection

Detection inventories the source video's burned-in text. It is confirmed through a dialog with a cost estimate, capped at 5-minute videos, and produces:

Each element:

Review on the player

The inventory is edited both as rows (text, times, category, language, numeric bbox) and directly on the player overlay: click to select, drag to move, eight resize handles, arrow-key nudges (Shift for coarse), Delete to remove, "Add graphic" at the playhead for anything the model missed. Category chips filter the overlay; an optional calibration grid helps with precise placement; elements with unknown timing render dashed and always-visible until timed. Like the transcript, graphics editing closes when the source locks.

Translation

Each language version translates the inventory: a bulk pass over all elements (failures fall back to the original text), plus per-element re-translation. Brand names, handles, hashtags, URLs, and numbers pass through untranslated; casing and line breaks are preserved. Each translation snapshots the source text it was made from (translatedFromValue), so source-side edits surface as Stale badges. The player overlay swaps in translated values, previewing placement before anything renders.

10. Graphics Rendering

Rendering (the first of the two graphics phases) turns translated lower thirds — elements categorized lower_third with a non-empty translation; other categories are inventory-only — into broadcast-ready clips:

1. Each translation splits into a kicker (first line) and headline (the rest).
2. Each graphic renders through an HTML template in a headless browser into a transparent animated clip — alpha-true, cropped to the graphic's band, designed at 1920×1080. The template owns placement and animation; see Designing Lower-Third Templates.
3. All clips composite over a checkerboard into a preview video, so the full set is reviewable at a glance.

Rendering is all-or-nothing: if any clip fails, everything produced in that run is discarded and no version is written. On success a RenderedGraphicsVersion is prepended:

Clips belong to their graphics version and are retained — every final combine that uses the version reuses the same clips, with no re-render.

11. The Final Combine

The combine (the second graphics phase) produces the deliverable MP4 from two choices:

Rules:

• Lower-third clips scale from the 1920×1080 design space to the base video's real dimensions and are burned in only during each clip's in_time–out_time window.
• Lip-sync bases are re-muxed: the combine drops the lip-sync output's re-encoded audio and lays the original compiled audio version (the one that drove the lip-sync) back over the picture — the soundtrack of the final cut is byte-identical to what was reviewed in the editor.
• Nothing upstream is re-run: bases and graphics versions mix and match freely.

Each run prepends a RenderedVideoVersion:

Version rows render the lineage as chips — "Graphics v2", "Audio v3" or "Lip-sync v1" — with a "(deleted)" fallback when an upstream version has since been removed.

12. Review & Playback

• Audio source toggle — play the picture against the original soundtrack or the dubbed compile (which mutes the video's own audio).
• Sync audio & video — locks the audio bar and the player together: play, pause, and seeks mirror both ways, with buffering handled on either side.
• Continuous playback — plays the generated takes segment by segment, auto-advancing in timeline order; discarded segments are skipped. Disabled (with the reason) while conflicting tasks run.
• Stems — every compiled version expands to its stems, each individually playable and downloadable.
• Rendered videos — lip-sync results, graphics previews, and final cuts — open in the floating picture-in-picture player and download with versioned filenames.
• Every item keeps a full revision history panel: who changed what, field by field.

13. Exports

The transcript and translations export from the editor header in four formats:

Rules: discard segments are excluded everywhere; in dubbed mode the export pairs source timing with the translated text (falling back to source text where untranslated); speaker labels resolve to their display names; files are named {reference}-{language}.

14. Storage & Key Map

Every dubbing object is a JSON value in the project's key/value store:

Task identities

Tasks are deterministic per item and stage — re-requesting a running stage attaches to the existing task instead of starting a second one, and anyone opening the item sees its progress:

Deletion & preferences

Archiving a dubbing item is a soft delete: it disappears from the default list and returns via the Archived filter. Per-user interface preferences — the editor split position, the active workspace tab, and the list/timeline segment view — are stored per user and follow the user across devices.