Running a translation API in production is straightforward until the invoice arrives or your p95 latency breaches its SLO. Many teams bolt on a machine translation provider, ship it, and move on, only to discover months later that they are overpaying by a wide margin, dropping requests during traffic spikes, and delivering inconsistent response times across language pairs. The root cause is rarely the provider itself; it is the absence of a disciplined tuning layer between your application and the API. This guide is a practical playbook for engineering teams that need to reduce translation API costs by twenty to forty percent while keeping latency predictable, covering batching, concurrency, caching, retry logic, multi-provider routing, and the monitoring infrastructure that holds it all together.
If you're evaluating how to build or optimize this layer, explore how Ollang handles production-grade API integration before investing engineering cycles in a custom build.
Request Batching Heuristics
The single biggest lever for throughput improvement is batching. Sending one string per HTTP request is the most expensive and slowest pattern possible: each call pays the full round-trip overhead, TLS handshake cost, and per-request billing minimum.
Optimal Batch Sizes by Provider and Language Pair
Most translation APIs accept arrays of strings in a single request, but their sweet spots vary. Google Cloud Translation, for example, documents a limit of 128 segments or 30,720 code points per request, while DeepL accepts up to 50 text parameters. Empirically, the best batch size depends on two factors: the provider's internal parallelism and the average segment length for a given language pair.
A useful heuristic:
| Factor | Guidance |
|---|---|
| Short segments (< 50 chars) | Batch aggressively, fill to provider max |
| Long segments (> 500 chars) | Smaller batches (10-20) to avoid timeouts |
| CJK languages | Character counts inflate quickly; monitor byte limits |
| Mixed source languages | Separate by source language; most APIs require a single source per call |
Start with a batch size that fills roughly 60-70 percent of the provider's payload limit, then adjust based on observed p95 latency. Over-stuffing batches trades latency for throughput, acceptable in async pipelines, problematic for interactive UIs.
Time-Based vs. Size-Based Flushing
Batching introduces a buffering delay: you hold segments in a queue until you have enough to fill a batch. Two strategies control when the queue flushes:
- Size-based flushing fires the request when the batch reaches a target segment count or byte size. Simple, predictable, but can leave the last few segments waiting indefinitely during low-traffic periods.
- Time-based flushing sets a maximum dwell time (e.g., 200 ms). When the timer expires, whatever is in the queue ships, even if the batch is small.
In practice, combine both: flush on whichever trigger fires first. A typical starting configuration is 20 segments or 150 ms, tuned per endpoint. Log the actual batch fill rate, if most batches flush on the timer rather than on size, your batch target is too high for your traffic volume.
Concurrency, Connection Pooling, and Timeouts
Tuning Max Connections and Thread Pools
Translation API calls are I/O-bound. You gain throughput not by adding CPU but by increasing the number of concurrent outbound connections. However, most providers enforce rate limits (requests per second or characters per minute), so your concurrency ceiling is ultimately the provider's, not yours.
Configure your HTTP client with a connection pool sized to your target concurrency. In a Node.js environment using undici or in Python using httpx, this looks like setting max_connections or max_keepalive_connections explicitly:
import httpx
client = httpx.AsyncClient(
limits=httpx.Limits(max_connections=50, max_keepalive_connections=20),
timeout=httpx.Timeout(connect=5.0, read=30.0, write=10.0)
)
Keep-alive connections eliminate repeated TLS handshakes. If your provider supports HTTP/2, a single connection can multiplex many requests, reducing the pool size you need.
Choosing Read and Connect Timeouts
Set connect timeouts tight (2-5 seconds) because DNS or TCP failures should surface fast. Read timeouts depend on your batch size and language pair: a 20-segment batch to a well-served language pair might return in under a second, while a large batch to a low-resource pair could take 10-15 seconds.
A good practice is to set a baseline read timeout at twice your observed p95 latency for a given batch configuration. If your p95 is 1.8 seconds, a 4-second read timeout catches genuine stalls without prematurely killing healthy requests.
Retries, Circuit Breakers, and Hedged Requests
Jittered Exponential Backoff
Naive retries with fixed delays create thundering-herd problems. When a provider returns a 429 (rate limit) or 503 (service unavailable), every client retrying at the same interval will spike traffic simultaneously on the next attempt.
Jittered exponential backoff solves this. The delay for attempt n is calculated as:
delay = min(base * 2^n, max_delay) * random(0.5, 1.0)
A reasonable starting point is a 500 ms base, a cap of 30 seconds, and a maximum of three retries. Log every retry with the status code and provider so you can distinguish transient blips from systemic issues.
When to Use Circuit Breakers vs. Hedged Requests
Circuit breakers and hedged requests address different failure modes:
- Circuit breakers protect you from a provider that is consistently failing. After a threshold of errors in a rolling window (e.g., five failures in 60 seconds), the circuit opens and all requests to that provider are short-circuited to a fallback for a cool-down period. This prevents wasted latency and billing on requests that will fail anyway.
- Hedged requests protect against tail latency. You send the request to one provider and, if no response arrives within a percentile-based deadline (e.g., p50 + buffer), you fire a duplicate to a second provider and take whichever responds first. This is powerful but doubles cost on hedged calls, so apply it selectively, only to latency-critical paths and only when you have a secondary provider configured.
Use circuit breakers universally. Use hedged requests only where your latency SLO is strict and the cost of a duplicate call is justified.
Caching: Translation Memory and CDN Layers
Caching is the most cost-effective optimization available. Repeated strings, UI labels, navigation elements, error messages, product attribute values, can constitute a significant share of total translation volume. Serving these from cache eliminates both cost and latency.
Cache-Key Design That Preserves Placeholders and Locale Variants
A cache key must uniquely identify the translation input. A minimal key includes:
- Source text (normalized)
- Source language code
- Target language code
- Glossary or model identifier (if applicable)
The subtlety is placeholder handling. Consider these two strings:
"Hello, {user_name}! You have {count} items."
"Hello, {user_name}! You have {count} items."
They are identical and should share a cache entry. But if your application substitutes values before translation, "Hello, Alice! You have 3 items." and "Hello, Bob! You have 7 items." become unique strings that never hit cache.
The fix: always translate with placeholders intact and substitute after retrieval. Normalize whitespace and casing in the source text before hashing. Use a deterministic hash (SHA-256 truncated to 128 bits is sufficient) as the cache key, with the full source text stored alongside for collision verification.
For locale variants (e.g., pt-BR vs. pt-PT), include the full BCP 47 tag in the key. Never collapse regional variants unless you have verified the translations are identical.
TM Lookups Before API Calls
A translation memory (TM) lookup layer sits in front of the API. For every incoming segment, check the TM first. Exact matches are served immediately; fuzzy matches (typically 85 percent or above) can be served with a confidence flag or routed to post-editing.
The architecture is straightforward: a key-value store (Ollang, Redis, DynamoDB, or a dedicated TM system) indexed by the cache key described above. On API response, write the result back into the TM. Over time, hit rates climb as your corpus grows, mature systems routinely see cache hit rates above 50 percent for product and UI content.
CDN Edge Caching for Repeated Segments
For web-facing translation (e.g., translating user-generated content snippets displayed to many visitors), a CDN cache layer can serve translations at edge latency. Set cache headers with a TTL appropriate to your content volatility, static UI strings can cache for hours or days; user-generated content may need shorter windows or purge-on-update semantics.
The combination of TM and CDN caching can reduce billable API volume dramatically, directly cutting costs while simultaneously improving p95 latency by serving most requests from memory or edge.