Batching at Scale: Async Translation API Jobs and Webhooks

When your localization pipeline needs to process hundreds of thousands of strings, product catalogs, legal disclosures, help-center articles, UI copy across dozens of locales, synchronous, one-request-at-a-time API calls collapse under their own weight. Rate limits throttle you, timeouts kill long-running jobs, and partial failures leave your database in an inconsistent state with no clear path to recovery. The real engineering challenge isn't calling a translation endpoint; it's building a reliable, high-throughput batch pipeline that can submit work asynchronously, listen for completions via webhooks, handle every failure mode gracefully, and do it all without translating the same sentence twice. This article is the blueprint for that system.

Ollang is the AI execution layer for enterprise localization, integrating async batching, translation memory, and quality controls across text, audio, video, software, websites, and legal documents. If you're evaluating how to wire async translation into your stack, schedule a walkthrough with Ollang's integration team to see how these patterns map to your architecture.

Why Synchronous Translation Calls Break at Scale

Synchronous translation API calls follow a simple pattern: send text, wait for the response, move on. That works for a settings page with twelve strings. It fails when you need to localize 200,000 product descriptions into 14 languages overnight.

The first wall you hit is rate limits. Most translation APIs enforce per-second or per-minute request caps. If you're limited to 100 requests per second and each request carries a single paragraph, translating a million paragraphs takes hours of uninterrupted, perfectly paced calls, assuming zero errors.

The second wall is timeout risk. Synchronous calls that carry large payloads or target multiple languages in one request can exceed gateway timeouts (commonly 30-60 seconds), causing the entire request to fail even if the translation engine was still working.

The third wall is resource waste. Your application thread or worker sits idle while waiting for a response. At scale, you're burning compute on waiting, not working.

Latency Ceilings and Timeout Risks

Even well-optimized translation engines need time to process large segments. A single synchronous request carrying a 50 KB JSON payload with nested HTML markup can take several seconds to parse, segment, translate, and reassemble. Multiply that by thousands of concurrent users or batch jobs, and you're stacking latency on top of latency. Gateway timeouts (HTTP 504) become routine rather than exceptional, and each one represents work that may or may not have completed on the server side, leaving you with an ambiguous state that's expensive to reconcile.

Thread-Pool Exhaustion in Monolithic Pipelines

In a monolithic pipeline where a single process iterates through a queue of translation requests synchronously, thread-pool exhaustion is inevitable. Each blocked thread waiting on an HTTP response is a thread unavailable for other work. In Java or .NET services with fixed thread pools, this leads to request queuing, cascading timeouts, and eventually service-level failures that affect far more than localization. The async pattern exists precisely to decouple submission from completion, freeing your application to do useful work while the translation engine processes in the background.

Designing an Async Batch Translation Architecture

An async batch architecture separates three concerns: job submission, job processing, and result delivery. Your system submits work, receives an acknowledgment with a job identifier, and then learns about completion through either polling or a webhook callback. This decoupling is what makes scale possible.

Job Submission Endpoints and Payload Design

A well-designed job submission endpoint accepts a structured payload containing the source content, source language, target language(s), and any processing directives (glossary IDs, formality settings, translation-memory references). The response is immediate: an HTTP 202 Accepted with a job ID and an estimated completion time.

Here's a representative submission payload:

{
"source_language": "en",
"target_languages": ["de", "fr", "ja", "pt-BR"],
"content": [
{
"key": "product.description.12847",
"text": "Lightweight running shoe with responsive cushioning.",
"context": "product catalog"
},
{
"key": "product.description.12848",
"text": "Waterproof hiking boot rated for sub-zero temperatures.",
"context": "product catalog"
}
],
"callback_url": "https://api.yourcompany.com/webhooks/translation",
"glossary_id": "gl_sportswear_2024",
"idempotency_key": "batch-20250115-catalog-run3"
}

The idempotency_key is critical. If your submission request times out and you retry, the API should recognize the duplicate and return the existing job ID rather than creating a second job. Without idempotency, retries create duplicate work and duplicate charges.

Polling vs. Webhook Callbacks

You have two options for learning when a job completes: polling and webhooks.

• Polling means your system periodically calls a status endpoint (e.g., GET /jobs/{job_id}) to check whether the job is done. It's simple to implement but wasteful, most polls return "still processing", and it introduces a latency floor equal to your polling interval.
• Webhooks mean the translation service calls your endpoint when the job completes (or fails). This is event-driven, efficient, and gives you near-instant notification. The tradeoff is that you must expose a publicly reachable HTTPS endpoint, validate incoming signatures, and handle the possibility that the webhook delivery itself fails.

The production-grade approach is to use webhooks as the primary notification channel with polling as a fallback. If you haven't received a webhook within a reasonable window (say, 2× the estimated completion time), your system polls to reconcile.

# Simplified webhook handler (Flask)
from flask import Flask, request, jsonify
import hmac, hashlib

app = Flask(__name__)
WEBHOOK_SECRET = b"your_shared_secret"

@app.route("/webhooks/translation", methods=["POST"])
def handle_translation_webhook():
signature = request.headers.get("X-Signature-256")
expected = hmac.new(WEBHOOK_SECRET, request.data, hashlib.sha256).hexdigest()
if not hmac.compare_digest(signature, expected):
return jsonify({"error": "invalid signature"}), 401

payload = request.json
job_id = payload["job_id"]
status = payload["status"]

if status == "completed":
store_translations(job_id, payload["results"])
elif status == "partial_success":
store_translations(job_id, payload["results"])
enqueue_retry(job_id, payload["failed_segments"])
elif status == "failed":
send_alert(job_id, payload["error"])

return jsonify({"received": True}), 200

Need help choosing and validating your webhook strategy? Discuss your webhook design with Ollang.

File and Document Format Support

Enterprise localization rarely involves plain strings alone. You need to translate DOCX contracts, HTML help articles, JSON resource bundles, XLIFF files, and sometimes PDF or subtitle formats. A capable async API accepts these as file uploads, handles internal segmentation (splitting a DOCX into translatable text runs while preserving formatting), and returns the translated file in the same format.

Key evaluation criteria across providers:

- Supported formats: DOCX, XLSX, HTML, JSON, XLIFF 1.2/2.0, SRT, PO, ARB, XML

- Max file size: varies widely; some cap at tens of MB, others allow 100 MB+

- Segmentation: does the API segment internally, or must you pre-segment?

- Markup preservation: are HTML tags, placeholders like {username}, and ICU patterns preserved?

- Multi-file jobs: can you submit a ZIP or batch of files in a single job?

When the API handles segmentation internally, you avoid the fragile work of parsing DOCX XML or splitting HTML without breaking tag nesting. When it doesn't, you need a pre-processing layer, and that layer becomes a source of bugs.

Ollang's platform handles internal segmentation and returns translated files in their original formats, reducing the need for brittle pre-processing in many enterprise workflows.

Chunking Strategies for Large Payloads

Even with async endpoints, you can't submit a 500 MB JSON file as a single request. You need to chunk intelligently.

Segment-Level vs. Document-Level Batching

• Document-level batching means each API job corresponds to one document (one DOCX file, one HTML page). This preserves context for the translation engine, which can use surrounding sentences to disambiguate meaning. It's the right default when documents are reasonably sized.
• Segment-level batching means you pre-segment content into individual strings or paragraphs and group them into batches of a fixed size (e.g., 500 segments per request). This gives you fine-grained control over payload size and makes partial-failure recovery simpler, you know exactly which segments failed. The tradeoff is loss of document-level context.

A practical hybrid approach: batch at the document level when documents are small, and switch to segment-level batching for large documents or string-table workloads. Use a threshold, say, 5,000 segments or 1 MB of text, to decide.

Respecting Maximum Payload Sizes and Segmentation Limits

Every API has limits, even if they're not always well-documented. Common constraints include:

- Maximum request body size (JSON): often 10-50 MB

- Maximum segments per request: sometimes capped at 1,000-5,000

- Maximum characters per segment: commonly 5,000-10,000

- Maximum target languages per request: some require one request per target language

Build your chunking logic to respect all of these simultaneously. A chunk that fits under the byte limit might still exceed the segment count limit. Validate both before submission.

def chunk_segments(segments, max_segments=500, max_bytes=5_000_000):
chunk = []
chunk_bytes = 0
for seg in segments:
seg_bytes = len(seg["text"].encode("utf-8"))
if (len(chunk) >= max_segments) or (chunk_bytes + seg_bytes > max_bytes):
yield chunk
chunk = []
chunk_bytes = 0
chunk.append(seg)
chunk_bytes += seg_bytes
if chunk:
yield chunk

Rate-Limit Compliance and Throughput Planning

Hitting rate limits isn't a sign that you're doing something wrong, it's a sign that you need a smarter submission strategy. Translation APIs typically enforce rate limits using a token-bucket algorithm, where you're allocated a certain number of tokens (requests or characters) per time window, and each request consumes tokens.

Token-Bucket Patterns and Adaptive Throttling

A token-bucket rate limiter refills at a steady rate. If you burst above the refill rate, you'll drain the bucket and start receiving HTTP 429 (Too Many Requests) responses. The response headers often tell you how long to wait:

HTTP/1.1 429 Too Many Requests
Retry-After: 2
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1705334400

Your client should read these headers and back off accordingly. A simple adaptive throttle works like this:

1. Start at your known rate limit (e.g., 50 requests/second).

2. On a 429, pause for the Retry-After duration.

3. After resuming, reduce your submission rate by 20%.

4. Gradually increase back toward the limit as successful responses accumulate.

This prevents the "thundering herd" problem where multiple workers all resume simultaneously after a backoff window and immediately trigger another round of 429s.

Forecasting Throughput and Cost per Million Characters

Before launching a batch pipeline in production, model your throughput and cost:

- Throughput: multiply your safe requests/second by average characters/request; then discount by 20-30% for retries, backoff, and variance.

- Cost: translation API pricing is typically per character or per word. At scale, even small per-character costs compound across many languages. Translation-memory deduplication (covered below) is the single most effective lever for reducing this cost.

Build a cost model before you scale. Include not just API fees but also compute for submission workers, webhook infrastructure, and storage for translated assets.

Idempotency, Retries, and Error Handling

Distributed systems fail. Networks drop packets, servers return 500 errors, and deployments cause brief outages. Your batch pipeline must handle all of this without duplicating work or losing data.

Implementing Idempotency Keys

An idempotency key is a client-generated unique identifier attached to each submission request. If the API receives two requests with the same idempotency key, it processes only the first and returns the same response for the second. This makes retries safe.

Generate idempotency keys deterministically from your input data, for example, a hash of the batch ID, chunk index, and content hash:

import hashlib

def make_idempotency_key(batch_id: str, chunk_index: int, content_hash: str) -> str:
raw = f"{batch_id}:{chunk_index}:{content_hash}"
return hashlib.sha256(raw.encode()).hexdigest()

This ensures that if you resubmit the same chunk from the same batch, you get the same idempotency key every time, even across process restarts.

Retrying 5xx Errors and Timeouts

For 5xx errors (server-side failures) and network timeouts, retry with exponential backoff and jitter:

import time, random, requests

def submit_with_retry(url, payload, headers, max_retries=5):
for attempt in range(max_retries):
try:
resp = requests.post(url, json=payload, headers=headers, timeout=30)
if resp.status_code == 202:
return resp.json()
if resp.status_code == 429:
wait = int(resp.headers.get("Retry-After", 5))
time.sleep(wait)
continue
if resp.status_code >= 500:
raise Exception(f"Server error: {resp.status_code}")
except (requests.Timeout, requests.ConnectionError, Exception) as e:
wait = min(2 ** attempt + random.uniform(0, 1), 60)
time.sleep(wait)
raise Exception("Max retries exceeded")

Never retry 4xx errors (except 429). A 400 Bad Request or 401 Unauthorized won't succeed on retry, these indicate a problem with your request or credentials.

Reconciling Partial Successes

Some APIs return partial results: 900 out of 1,000 segments translated successfully, with 100 failing due to unsupported characters, segment-length violations, or internal errors. Your pipeline must:

1. Accept and store the successful segments. Don't discard good work because of a few failures.

2. Log the failed segments with error details (segment key, error code, error message).

3. Route failed segments to a retry queue with a separate, more conservative retry policy.

4. Set a maximum retry count; after several attempts, move the segment to a dead-letter queue for human review.

Translation Memory and Deduplication to Cut Volume

Translation memory (TM) is a database of previously translated segments. Before sending a segment to the API, you check whether an identical or highly similar source string has already been translated. If it has, you reuse the existing translation and skip the API call entirely.

For enterprise content, where product descriptions, legal boilerplate, and UI strings repeat extensively, TM deduplication routinely eliminates a substantial share of billable volume. The savings compound across languages: a segment translated once from English can be reused for every target locale.

Implementing TM Lookup Before Submission

Your pipeline should include a dedup stage between content extraction and API submission:

- Normalize each source segment (trim whitespace, normalize Unicode, lowercase for matching purposes).

- Hash the normalized segment.

- Look up the hash in your TM store (a database table, Redis cache, or the TM service provided by your translation API).

- If a 100% match exists, use the cached translation. If a fuzzy match exists (typically 75%+ similarity), flag it for human review or use it with a confidence score.

- If no match exists, add the segment to the submission batch.

After the API returns translations, write the new source-target pairs back into TM for future reuse.

If you're looking to layer translation memory, glossary enforcement, and async batching into a single managed pipeline, explore Ollang’s end-to-end approach.

Glossary and Terminology Enforcement

Translation memory handles full-segment reuse. Glossaries handle term-level consistency, ensuring that "compliance dashboard" is always translated as "Compliance-Dashboard" in German, never "Konformitäts-Übersicht." Most translation APIs accept a glossary ID at submission time, and the engine applies those term constraints during translation. Maintain your glossary as a versioned artifact (CSV, TBX, or JSON) in source control, and reference the current version in every API call.

Placeholder and Markup Preservation

Translated content that breaks your application is worse than untranslated content. Placeholders like {user_name}, {{count}}, %d, and %s must survive translation intact. HTML tags must remain properly nested. ICU message format patterns must preserve their syntax.

Strategies for Safe Markup Handling

• Pre-processing: replace placeholders with XML-safe tokens (e.g., <x id="1"/>) before submission, then restore them after translation. This is the XLIFF approach and the most robust.
• API-native support: some translation APIs natively recognize and preserve common placeholder patterns. Verify this with your specific formats before trusting it.
• Post-processing validation: after receiving translations, run a validation pass that checks placeholder integrity, tag balance, and format-string argument counts. Reject and re-queue any segment that fails validation.

import re

def validate_placeholders(source: str, translated: str) -> bool:
pattern = r"\{[^}]+\}"
source_phs = sorted(re.findall(pattern, source))
translated_phs = sorted(re.findall(pattern, translated))
return source_phs == translated_phs

Production Failure Runbook

A batch translation pipeline in production will encounter failures. The question isn't whether, but how gracefully your system recovers.

Exponential Backoff and Circuit Breakers

Exponential backoff prevents your system from hammering a struggling API. Start with a 1-second delay, double it on each retry, add random jitter to prevent synchronized retries across workers, and cap the maximum delay (around a minute is reasonable).

A circuit breaker goes further: after a threshold of consecutive failures (e.g., 10 in a row), the circuit opens and your system stops making requests entirely for a cooldown period. This protects both your system and the API from cascading failures. After the cooldown, allow a single probe request. If it succeeds, close the circuit and resume normal operation.

Dead-Letter Queues and Poison Message Handling

A dead-letter queue (DLQ) is where messages go after exhausting their retry budget. In the context of batch translation:

- A segment that fails translation after several retries goes to the DLQ.

- A webhook payload that your handler can't parse goes to the DLQ.

- A job that returns an unrecognized status goes to the DLQ.

Every message in the DLQ should include the original payload, all error messages from each attempt, timestamps, and the job/batch context needed for a human operator to diagnose and re-process.

Poison messages are messages that crash your consumer on every attempt, perhaps due to malformed Unicode, an unexpectedly large payload, or a bug in your handler. Without DLQ handling, a poison message blocks the entire queue. Your consumer must catch all exceptions, increment a per-message failure counter, and route to the DLQ after the threshold.

Monitoring and Alerting

Instrument your pipeline with metrics that matter:

- Jobs submitted per minute: throughput baseline

- Jobs completed per minute: processing capacity

- Median and p99 job completion time: latency tracking

- 429 responses per minute: rate-limit pressure

- 5xx responses per minute: upstream API health

- DLQ depth: unresolved failures accumulating

- TM hit rate: dedup effectiveness and cost savings

- Webhook delivery success rate: callback reliability

Set alerts on DLQ depth (any sustained increase), 5xx rate (above a small baseline), and job completion time (exceeding roughly 2× your typical p99). These are early warning signals that something in your pipeline or the upstream API is degrading.

Frequently Asked Questions

How do I choose between polling and webhooks for batch translation jobs?

Use webhooks as your primary completion mechanism and polling as a fallback. Webhooks give you near-instant notification without wasting resources on empty polls. However, webhooks can fail (your endpoint is down, the provider's delivery system has an outage), so implement a reconciliation loop that polls for any job that hasn't received a webhook callback within a reasonable time window, typically 2× the expected completion time. This dual approach gives you both efficiency and reliability.

What's the best way to handle partial translation failures in a batch?

Accept and persist the segments that succeeded, then route the failed segments to a separate retry queue with their error details. Use a distinct retry policy for these segments, perhaps with longer backoff intervals or a different chunking strategy (smaller batches). After a fixed number of retries, move persistently failing segments to a dead-letter queue for human investigation. Never discard successful translations because a subset of the batch failed.

How much volume can translation memory deduplication actually save?

It depends heavily on your content type. Structured content like product catalogs, e-commerce listings, and UI strings tends to have high repetition, and TM dedup can eliminate a significant portion of redundant API calls. Legal and marketing content with more unique phrasing sees lower but still meaningful savings. The key is to measure your TM hit rate in production and track the resulting cost reduction over time. Even modest dedup rates translate to substantial savings when you're processing millions of characters across multiple languages.

How should I size my batches for optimal throughput?

Start with the API's documented limits (max segments per request, max payload size) and work backward. A good default is 200-500 segments per batch or roughly 1-2 MB of text, whichever limit you hit first. Smaller batches give you finer-grained failure recovery and more even load distribution. Larger batches reduce HTTP overhead and may improve translation quality through better context. Run benchmarks with your actual content to find the sweet spot, then monitor p99 completion times and adjust.

Start Building Your Batch Translation Pipeline

A reliable, high-throughput batch translation pipeline isn't a single API call, it's an architecture. Async job submission, webhook-driven completion, idempotent retries, translation-memory dedup, placeholder validation, and dead-letter queues all work together to turn a fragile script into a production system that handles millions of characters across dozens of languages without losing data or burning budget.

Get Started

See how these patterns map to your stack and where Ollang can remove complexity from day one. Book a Demo