Most video teams still localize content through manual handoffs: exporting files, emailing vendors, waiting days for delivery, then stitching audio tracks back together by hand. When your library holds hundreds or thousands of assets and you need to reach audiences in dozens of languages, that workflow collapses under its own weight. Turnaround stretches into weeks, costs multiply, and quality becomes inconsistent.
API-led dubbing replaces that fragile chain with a programmable pipeline. By integrating AI dubbing directly into your video infrastructure, engineering teams can trigger localization jobs from a CMS event, track progress through webhooks, receive mixed deliverables automatically, and publish multi-language streams without human file-shuffling. This article walks through exactly how to build that pipeline, from authentication and request schemas to muxing, manifests, monitoring, and compliance.
Ollang's API supports idempotent job submission, webhook events, and enterprise controls to integrate with existing pipelines. If your team is evaluating how to embed AI dubbing into an existing media stack, request a walkthrough of Ollang's integration surface to see how the pipeline fits your architecture.
Core API Contract: Requests, Responses, and Authentication
A well-designed dubbing API behaves like any media-processing service: you submit a job, receive a handle, and poll or listen for completion. The difference lies in the domain-specific parameters that control voice selection, lip-sync behavior, and multi-format output.
Request Schema Essentials
A typical dubbing job submission includes these key fields:
- source_url (string, URL): Presigned URL or public endpoint for the source video or audio file.
- target_langs (array of strings): BCP 47 language codes (e.g., ["es-MX", "de-DE", "ja-JP"]).
- voice_id (string): Identifier for a pre-built synthetic voice from the provider's library.
- clone_ref (string, optional): Reference to a cloned voice model for speaker similarity to original talent.
- lip_sync (boolean): Whether to apply viseme-aligned timing adjustments to synthesized speech.
- export_formats (array of strings): Desired deliverables (e.g., ["mp4", "wav", "ttml"]).
- callback_url (string, URL): Webhook endpoint for status events.
- idempotency_key (string, UUID): Client-generated key to prevent duplicate job creation on retries.
The source_url should be a presigned URL with a time-limited signature, typically valid for 30 to 60 minutes, so the service can pull the asset without requiring persistent credentials to your storage. AWS S3 presigned URLs, GCS signed URLs, and Azure SAS tokens all work for this pattern.
Authentication and Security
API authentication should follow standard patterns: bearer tokens issued through an OAuth 2.0 client-credentials flow, or API keys scoped to specific projects. Rotate keys on a regular cadence and never embed them in client-side code.
For transport security, enforce TLS 1.2 or higher on every endpoint. If your dubbing provider supports mutual TLS for webhook delivery, enable it, this prevents spoofed callback events from injecting false status updates into your pipeline.
Idempotency and Safe Retries
Network failures between your client and the dubbing API are inevitable. An idempotency_key (a client-generated UUID attached to each job submission) ensures that retrying a failed request does not create a duplicate job. The API should return the same job ID and status for any request bearing a previously seen idempotency key within a defined window, typically 24 to 48 hours.
This is not optional for production pipelines. Without idempotency, a timeout on the initial POST followed by an automatic retry will produce two jobs, two invoices, and two sets of deliverables, a debugging headache that compounds at scale.
Job Orchestration and Webhook-Driven Workflows
Dubbing is inherently asynchronous. A single job may take minutes for short-form content or considerably longer for feature-length video with multiple target languages. Polling for status wastes compute and complicates error handling. Webhooks are the correct pattern.
Lifecycle Events
A robust dubbing API emits structured webhook events at each stage of the pipeline:
- job.created, The job has been accepted and queued.
- job.transcription.complete, Source audio has been transcribed; the transcript is available for review or override.
- job.adaptation.complete, The script has been adapted (translated and time-fitted) for each target language.
- job.synthesis.complete, Voice generation is finished for all target languages.
- job.mixing.complete, Synthesized dialogue has been mixed against the music-and-effects (M&E) stem.
- job.complete, All deliverables are ready for download.
- job.failed, The job has terminated with an error; the payload includes an error code and human-readable message.
Each event payload should include the job ID, a timestamp, the event type, and, for terminal states, presigned download URLs for each deliverable.
Designing Your Webhook Consumer
Your webhook endpoint needs to be idempotent on the receiving side as well. Dubbing providers may retry delivery if they do not receive a 2xx response within a timeout window. Deduplicate incoming events by storing the event ID and skipping processing if you have already handled it.
Respond to the webhook request quickly, within a few seconds, and offload any heavy processing (downloading files, updating your CMS, triggering downstream encodes) to a background worker. If your endpoint is slow to respond, the provider's retry logic may fire prematurely.
Retries and Exponential Backoff
For outbound API calls from your side (job submissions, status queries, deliverable downloads), implement exponential backoff with jitter. A common pattern:
- Wait 1 second after the first failure.
- Double the wait on each subsequent retry, up to a maximum of 60 seconds.
- Add random jitter (±25% of the wait interval) to prevent thundering-herd effects when multiple jobs fail simultaneously.
- Cap total retries at a reasonable number (five to seven attempts) before routing the job to a dead-letter queue for manual investigation.
Source Preparation: Stems, Separation, and Loudness
The quality of your dubbed output depends heavily on the quality of your input. Feeding a dubbing API a final mixed-down stereo file forces the system to perform source separation, isolating dialogue from music and effects, which introduces artifacts and degrades the final mix.
Providing Discrete Stems
Whenever possible, supply separate audio stems: a dialogue-only track and a music-and-effects (M&E) track. If your production workflow already delivers these (common in broadcast and streaming), pass them as distinct files or as separate tracks within a single MXF or multitrack WAV container.
When discrete stems are unavailable, the dubbing pipeline will apply AI-based source separation. Modern separation models (descendants of architectures like Meta's Demucs) have improved dramatically, but they still introduce subtle artifacts, particularly on content where dialogue overlaps with prominent sound effects or where music contains vocal elements. The cleaner your input, the cleaner your output.
Loudness and Level Standards
Dubbed dialogue must conform to the same loudness standards as your original content. For broadcast delivery, that typically means EBU R 128 (−23 LUFS integrated) in Europe or ATSC A/85 (−24 LKFS) in North America. Streaming platforms generally target −14 LUFS for music-heavy content and −16 to −24 LUFS for spoken-word and film.
Specify your target loudness in the API request or as a project-level default. The mixing stage should normalize the synthesized dialogue to match the M&E stem's reference level, then true-peak limit the final mix to −1 dBTP or −2 dBTP depending on your delivery spec.
Muxing Multi-Language Audio Tracks
Once the dubbing API returns individual language tracks as WAV or AAC files, you need to combine them into containers your players and CDNs can consume.
Container Formats
- MP4 (ISOBMFF), The standard for web and mobile delivery. Supports multiple audio tracks with language metadata via the track language field and the elng box. Tools like FFmpeg handle this natively: each audio stream is added with -map flags and tagged with BCP 47 language codes using -metadata:s:a:N language=es.
- MKV (Matroska), Common for archival and OTT workflows. Supports an arbitrary number of audio and subtitle tracks with rich metadata. MKVToolNix provides precise control over track ordering, default flags, and forced-subtitle markers.
- MXF (Material Exchange Format), Required for broadcast playout and regulatory delivery. Multi-language audio is carried as separate essence tracks within the MXF wrapper, typically conforming to the AS-11 or AS-02 shim specifications.
Attaching Caption Sidecars
Your dubbing pipeline should produce timed-text files alongside the audio. Common formats include:
- TTML (Timed Text Markup Language), Preferred for broadcast and SVOD delivery, with rich styling and region support.
- WebVTT, The web-native standard, supported by all modern browsers and HLS.
- SRT, Simple and widely supported, though it lacks styling metadata.
Attach these as sidecar files rather than burning them into the video. This preserves flexibility: viewers can toggle captions on or off, and you can update translations without re-encoding the video.