Batching at Scale: Bulk Translation API Jobs, Rate Limits, Retries

When your pipeline needs to push tens of millions of characters through a translation API every day, the naive approach, one request per string, fire-and-wait, collapses under its own weight. Rate limits throttle you, transient failures corrupt batches, and costs spiral because you retranslate content you already paid for. Building a reliable bulk translation pipeline means understanding how major APIs handle batch operations, how to chunk and queue work intelligently, how to recover from failures without duplicating effort, and how to keep throughput predictable enough to attach an SLA. This guide walks through the architecture decisions, concurrency math, and error-handling patterns that separate a fragile script from a production-grade localization pipeline.

If your team is already wrestling with these problems at scale, see how Ollang’s translation API layer can handle batching and retries for you: Get a tailored walkthrough.

Synchronous Batch Endpoints vs Asynchronous Job APIs

The first architectural decision is whether to use synchronous or asynchronous translation calls. The choice affects everything downstream: how you handle failures, how you monitor progress, and how much throughput you can realistically sustain.

When to Use Synchronous Batch Calls

Synchronous batch endpoints accept an array of text segments in a single HTTP request and return all translations in the response body. Google Cloud Translation’s basic translateText endpoint, for example, accepts multiple text segments per call (with character and item limits documented in Google’s quotas), and Azure Translator’s /translate endpoint handles up to 50 text elements with a combined limit of 50,000 characters per request, as documented in Microsoft’s request limits.

Synchronous calls work well when:

- Individual payloads are small (UI strings, metadata, short product descriptions).

- You need results immediately for a user-facing flow.

- Total daily volume stays under a few million characters.

The tradeoff is straightforward: you get simplicity and low latency at the cost of throughput ceiling. Each call blocks until the response returns, so your concurrency is bounded by the number of parallel connections you can sustain without hitting rate limits.

When to Move to Asynchronous Job APIs

Once you cross into document-scale or corpus-scale work, translating hundreds of files, millions of segments, or entire product catalogs, asynchronous job APIs become essential. These endpoints accept a job definition, return a job ID immediately, and process the work in the background.

• Ollang’s translation API layer can orchestrate asynchronous job workflows across providers, manage job retries and idempotency, and surface webhook or polling notifications so you don’t need to wire provider-specific logic directly into your services.
• Google Cloud’s batchTranslateText accepts source files from Cloud Storage and writes translated output back to a bucket. Jobs can run for hours on large corpora.
• Azure Document Translator operates similarly, pulling source documents from Azure Blob Storage and writing results to a target container.
• Amazon Translate’s batch operations use S3 as the input/output layer and process files asynchronously through StartTextTranslationJob.

All three follow the same pattern: submit a job, receive a job ID, then either poll for status or receive a webhook/notification when the job completes. The API provider manages internal parallelism, retry logic, and resource allocation on their side.

Asynchronous APIs are the right choice when:

- Total volume exceeds what you can push through synchronous calls within your rate-limit window.

- You are translating files (XLIFF, HTML, DOCX) rather than raw strings.

- You can tolerate minutes-to-hours latency in exchange for much higher throughput.

If you’re deciding whether to keep investing in synchronous batching or move to job-based orchestration, see how Ollang routes work across providers and job types: Evaluate your options with our team.

Payload Sizing and Chunking Strategies

Getting payload sizing wrong is one of the most common causes of batch failures. Too large, and the API rejects the request or times out. Too small, and you waste quota on HTTP overhead and underutilize your rate-limit budget.

Calculating Optimal Chunk Sizes

Start with the hard limits documented by your provider:

- Google Cloud Translation (v3) synchronous: character and request-item limits apply; see Google’s quotas and limits.

- Azure Translator synchronous: up to 50 text elements and 50,000 characters per request (see Microsoft’s service limits).

- Amazon Translate synchronous: accepts a single text string per request, with a byte-size limit (UTF-8).

- Asynchronous/batch endpoints (Google/Azure/AWS): file-based I/O with provider-defined job and file-count limits.

For synchronous calls, target roughly 70-80% of the hard limit you measure in your own environment. This leaves headroom for encoding overhead and prevents edge-case rejections when character counts differ between your counting method and the API’s (UTF-8 bytes vs Unicode code points vs surrogate pairs).

A practical first-fit-decreasing chunking algorithm:

1. Sort segments by character length descending.

2. Bin-pack segments into batches using a greedy first-fit-decreasing strategy, respecting both the character limit and the item-count limit.

3. Tag each batch with a deterministic hash of its contents (see Idempotency below).

import math

def chunk_segments(
segments,
max_chars=24000, # tune to ~70-80% of your provider’s request limit
max_count=500 # tune to provider’s max items/request
):
# Sort by length descending (first-fit-decreasing)
ordered = sorted(segments, key=lambda s: len(s), reverse=True)
batches = []
current_batch = []
current_chars = 0

for seg in ordered:
seg_len = len(seg) # beware: len() is code units in Python; measure bytes/code points if needed
if (current_chars + seg_len > max_chars) or (len(current_batch) >= max_count):
if current_batch:
batches.append(current_batch)
current_batch = []
current_chars = 0
current_batch.append(seg)
current_chars += seg_len

if current_batch:
batches.append(current_batch)

return batches

Notes:

- If your provider counts bytes, precompute seg_len as len(seg.encode("utf-8")).

- Keep a safety margin for markup expansion when using HTML-aware modes.

Handling Markup and Placeholders Safely

Segments containing HTML, XML, or ICU MessageFormat placeholders require special care during chunking. Most translation APIs offer a parameter to indicate that the input contains markup, Google’s mimeType: "text/html" or Azure’s textType: "html", for instance, but these modes change how the API counts characters and can alter rate-limit consumption.

Key rules:

- Never split a segment mid-tag. Treat each segment as atomic.

- Validate that placeholder tokens (e.g., {username}, %1$s, <x id="1"/>) survive the round trip. Run a post-translation check that counts placeholders in the source and target.

- If your segments contain XLIFF inline elements, consider pre-extracting translatable text and reinserting translations after the API call. This reduces payload size and eliminates an entire class of markup-corruption errors.

Provider references:

- Google Cloud Translation quotas and troubleshooting: https://cloud.google.com/translate/quotas

- Azure Translator service limits: https://learn.microsoft.com/en-us/azure/ai-services/translator/service-limits

Idempotency Keys and Deduplication

Retrying a failed batch without idempotency guarantees means you might translate, and pay for, the same content twice. Worse, if the API processed part of the batch before the failure, you could end up with duplicate or inconsistent results.

Generating Deterministic Batch Hashes

Assign each batch an idempotency key derived from its content. A SHA-256 hash of the concatenated source segments, source language, and target language produces a stable identifier:

import hashlib

def batch_id(segments, source_lang, target_lang):
content = f"{source_lang}:{target_lang}:" + "\n".join(segments)
return hashlib.sha256(content.encode("utf-8")).hexdigest()

Store this hash alongside the batch status in your job tracking database. Before submitting a batch, check whether a completed result already exists for that hash. This pattern gives you:

- Cost savings: no double-billing for retranslated content.

- Idempotent retries: safe to resubmit after a timeout without side effects.

- Cache hits: if the same content appears in a future job, skip the API call entirely.

Prefilling with Translation Memory to Cut Cost

Before any batch hits the API, run each segment against your translation memory (TM). Exact matches (100% TM matches) can be resolved locally with zero API cost. Fuzzy matches above a configurable threshold (typically 85%+) can be sent to the API with the TM suggestion as context, reducing the work the engine needs to do and often improving output quality.

A well-maintained TM can eliminate a substantial share of API calls in mature localization programs, particularly for product updates where large portions of the UI remain unchanged between releases. The savings compound: fewer API calls means lower cost, fewer rate-limit collisions, and faster end-to-end job completion.

Rate Limits, Quotas, and Concurrency Controls

Every translation API enforces rate limits, and the specifics vary enough between providers that you need to design your pipeline to adapt rather than hardcode.

Understanding Quota Structures

Rate limits typically come in two flavors:

- Requests per second (RPS): a cap on how many HTTP calls you can make per unit time, regardless of payload size.

- Characters per period: a cap on total translation volume, often measured per minute or per day.

Some providers enforce both simultaneously. Azure Translator, for example, applies per-second request limits that vary by pricing tier, while also enforcing monthly character quotas. Google Cloud Translation applies per-project quotas measured in characters per period, configurable via the Cloud Console.

Track both dimensions in your pipeline’s rate limiter. A token-bucket algorithm works well: maintain one bucket for requests and another for characters, and only dispatch a batch when both buckets have sufficient capacity.

Throughput Math: Characters per Second per Worker

Model throughput with your own measurements rather than assumptions:

- Throughput per worker ≈ batch_size_chars / avg_call_latency_seconds

- Sustained throughput ≈ per_worker_throughput × number_of_workers × utilization_factor

Example (illustrative):

- Target: 10,000,000 characters in 4 hours (14,400 seconds) → ~694 chars/second sustained.

- Measured p50 call time: 1.5s for a 3,000-character batch → ~2,000 chars/second per worker.

- Utilization after backoff and jitter: ~35%.

- Effective per-worker rate: ~700 chars/second.

- Workers required: ~1-2, rounded up for headroom and failure recovery.

For asynchronous job APIs, the math is different: you submit the corpus and the provider manages parallelism internally. Your bottleneck shifts to upload bandwidth, job queue depth, and provider-side concurrency rather than per-request latency.

Retry Policies and Exponential Backoff

Transient failures are inevitable at scale. Network blips, provider-side throttling (HTTP 429), and intermittent 500-series errors will all occur. Your retry policy determines whether these are minor hiccups or pipeline-breaking events.

Designing a Backoff Strategy

Use exponential backoff with jitter. A simple implementation:

import time
import random

class TransientError(Exception):
pass

def retry_with_backoff(func, max_retries=5, base_delay=1.0):
for attempt in range(max_retries):
try:
return func()
except TransientError:
if attempt == max_retries - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)

Key design decisions:

- Max retries: 3-5 for synchronous calls. For async job submissions, you can afford more retries since the cost of resubmission is low.

- Base delay: start at 1 second (and respect provider guidance).

- Jitter: always add randomized jitter to prevent thundering-herd effects when multiple workers hit the same rate limit simultaneously.

- Retry-After headers: many APIs return Retry-After with 429 responses. Respect this value, it overrides your calculated backoff.

Retry-Safe Storage Patterns

Every batch submission should be logged to durable storage before the API call and updated after the response. A minimal schema:

This table is your single source of truth. On pipeline restart after a crash, query for batches in submitted or retrying status and resume from there. Combined with idempotency keys, this makes your pipeline crash-safe.

Job Status: Polling vs Webhook Handling

For asynchronous jobs, you need to know when they finish. The two approaches, polling and webhooks, have different operational profiles.

Polling Patterns

Polling is simple to implement. After submitting a job, periodically call the status endpoint:

curl -X GET \
"https://translation.googleapis.com/v3/projects/my-project/locations/us-central1/operations/op-id" \
-H "Authorization: Bearer $(gcloud auth print-access-token)"

A typical response includes a done boolean and, when complete, either a response or error field. Poll at increasing intervals, every 10 seconds for the first minute, then every 30 seconds, then every 2 minutes, to avoid wasting quota on status checks.

The downside: polling adds latency (you discover completion only at the next poll interval) and consumes API quota on status calls.

Webhook Callbacks

Webhook-based notification eliminates polling overhead. When available, configure a callback URL that the API hits when the job completes. Azure Document Translator supports this natively, and you can configure Pub/Sub or Eventarc notifications for Google Cloud operations.

Your webhook endpoint should:

1. Validate the request signature or origin to prevent spoofed callbacks.

2. Acknowledge receipt immediately (return HTTP 200) before processing.

3. Enqueue the result for downstream processing rather than handling it inline.

4. Be idempotent, the provider may retry the callback if your endpoint was temporarily unavailable.

For production pipelines, use webhooks as the primary notification mechanism and polling as a fallback. If no webhook arrives within a configurable timeout (e.g., 2× the expected job duration), fall back to polling to detect stuck or silently failed jobs.

Error Taxonomies and Triage

Not all errors deserve the same response. Categorizing errors into actionable groups lets your pipeline handle each type appropriately.

Classifying API Errors

Common Content Errors

Beyond HTTP-level errors, watch for content-level failures that the API may return as partial successes:

- Invalid markup: the API could not parse HTML/XML in the source. Fix the source segment and resubmit.

- Glossary misses: a term that should have been enforced by a glossary was translated differently. This usually indicates the glossary entry’s source term didn’t match the segment’s surface form (case sensitivity, morphological variation). Update the glossary or normalize input.

- Placeholder corruption: translated output is missing placeholders or has reordered them incorrectly. Flag for human review or apply automated placeholder restoration.

Build automated checks that run on every completed batch: placeholder parity validation, glossary term verification, and length-ratio sanity checks (a translation that is 3× longer than the source in a language pair where ~1.2× is typical likely has a problem).

If you want a platform that handles error classification, automatic retries, and quality checks as part of the translation pipeline, see how Ollang’s API integration layer manages this end-to-end: Talk to our team.

Designing a Bulk Pipeline with SLAs

Pulling all of these patterns together, a production-grade bulk translation pipeline looks like this:

1. Ingest: accept source content via file upload, CMS webhook, or repository sync.

2. Dedupe and TM prefill: hash each segment, check TM for exact matches, and remove already-translated content.

3. Chunk: bin-pack remaining segments into optimally sized batches with idempotency keys.

4. Dispatch: submit batches through a concurrency-controlled worker pool with token-bucket rate limiting.

5. Monitor: track job status via webhooks (primary) and polling (fallback).

6. Retry: automatically retry transient failures with exponential backoff; route non-retryable errors to a dead-letter queue for manual triage.

7. Validate: run post-translation quality checks (placeholders, glossary, length ratios).

8. Deliver: write validated translations to TM, push to downstream systems (CMS, repository, CDN).

For SLA definition, express guarantees in terms of:

- Throughput: maximum characters per hour the pipeline can sustain.

- Latency: time from job submission to delivery of validated translations.

- Error rate: percentage of segments requiring manual intervention.

- Recovery time: maximum time to resume after a pipeline failure.

Instrument each metric with monitoring and alerting. If your throughput drops below the SLA threshold, because a provider tightened rate limits, a new language pair has higher latency, or retries spike, your pipeline should automatically scale workers, reduce batch size, or route overflow to a secondary translation engine.

FAQ

How many characters can I send in a single translation API request?

It depends on the provider and endpoint type. Synchronous endpoints enforce per-request character and item limits; asynchronous/batch endpoints are often file-based and governed by job/file-count limits. Check the provider documentation for hard limits and target 70-80% of the documented maximum to leave headroom for encoding differences.

Should I use polling or webhooks to track async translation jobs?

Use webhooks as your primary notification mechanism whenever the provider supports them. They eliminate polling overhead and reduce latency between job completion and downstream processing. Always implement polling as a fallback: if no webhook arrives within a reasonable timeout window, poll the status endpoint to catch silently failed or stuck jobs. Ollang’s integration layer supports webhook-first notifications with an automated polling fallback to simplify this pattern.

How do I prevent paying twice for the same content during retries?

Generate a deterministic idempotency key for each batch, typically a SHA-256 hash of the source segments, source language, and target language. Store this key alongside the batch status in your job-tracking database. Before submitting any batch, check whether a completed result already exists for that key. This prevents duplicate API charges on retry and also enables cross-job caching when the same content appears in future translation runs. Combining this with translation memory prefill further reduces redundant API consumption.

What is the best retry strategy for translation API rate limits?

Exponential backoff with jitter is the standard approach. Start with a 1-second base delay, double it on each subsequent attempt, and add a random jitter component (for example, 0-1 second) to prevent multiple workers from retrying in lockstep. Cap retries at 3-5 attempts for synchronous calls. Always check for and respect the Retry-After header in 429 responses, it provides the provider’s recommended wait time and should override your calculated delay. If you consistently hit rate limits, reduce concurrency, lower per-request payload size, or request a quota increase from your provider.

Get Started with Scalable Bulk Translation

Building a resilient bulk translation pipeline requires careful orchestration of chunking, rate limiting, retry logic, and quality validation. Ollang’s translation API integration layer handles these complexities natively, managing batching, backoff, deduplication, and post-translation quality checks so your engineering team can focus on the product, not the plumbing.

Book a Demo