Throughput, Latency, and Cost: Sizing Your Translation API Stack

Most teams discover their translation API costs are unpredictable only after the first invoice arrives. A product ships in twelve languages, traffic spikes during a launch, and suddenly the bill is three times the estimate because nobody modeled minimum billable units, retry-induced duplicate charges, or the cost delta between synchronous and asynchronous calls. Sizing a translation API stack is not a procurement exercise, it is an architecture decision that touches throughput planning, latency budgets, caching strategy, and cost control simultaneously. This guide provides a quantitative framework so you can build a sizing sheet, compare providers on equal footing, and make infrastructure choices that hold up at scale. If you need help mapping this to your own content volumes, talk to the Ollang team about your stack.

Understanding Translation API Pricing Models

Translation API pricing looks straightforward until you read the fine print. The three dominant models, per-character, per-token, and per-word, each introduce measurement quirks that can inflate costs by double digits if you are not careful.

Per-Character vs Per-Token vs Per-Word Billing

Per-character billing counts every Unicode character in the source text, including whitespace and punctuation. Many MT vendors, such as Google Cloud Translation and Amazon Translate, use this model. Orchestration platforms like Ollang surface those counts in your billing model so you can compare unit metering accurately against token- and word-based providers. Per-token billing, common with LLM-based quality assurance endpoints, splits text into subword units using a tokenizer like BPE; a single English word may produce one to three tokens, while agglutinative languages like Finnish or Turkish routinely generate more tokens per word. Per-word billing, used by some legacy providers, counts whitespace-delimited tokens, a model that disadvantages CJK languages where words are not separated by spaces.

The practical consequence: the same 10,000-word English document can cost meaningfully different amounts across providers depending on which unit they meter. Always normalize your estimates to a common denominator (characters are the most portable) before comparing quotes.

Minimum Billable Units and Free-Tier Traps

Many providers enforce a minimum billable unit per request. If the minimum is 1,000 characters and you send a 12-character button label, you pay for 1,000 characters. For applications that translate many short UI strings individually, think tooltips, menu items, error messages, this rounding penalty can multiply effective cost by orders of magnitude.

Free tiers introduce a different trap. A provider may offer 500,000 characters per month at no charge, but the free tier often excludes advanced features like glossary enforcement, adaptive translation, or custom model endpoints. Teams prototype on the free tier, bake the provider into their CI pipeline, and then discover that the production feature set they actually need sits on a paid plan with a different rate card. Evaluate pricing against the feature set you will use in production, not the feature set available during a proof of concept.

Hidden Surcharges: Language Pairs, Glossaries, and QA Layers

Not all language pairs cost the same. Some providers apply surcharges for low-resource language pairs or for routing through a pivot language. Glossary enforcement, where you supply a terminology list that the engine must respect, may add a per-request premium or require a dedicated custom model that carries its own hourly or monthly fee.

LLM-based quality assurance layers, increasingly popular for post-editing and fluency scoring, are billed on token consumption at rates that can be five to fifteen times higher than raw MT character pricing. If your pipeline calls an MT endpoint and then passes the output through an LLM QA step, you are paying twice, once for translation, once for review, and the QA leg may dominate total cost.

Ollang centralizes glossary enforcement and QA stages in billing and reporting so these surcharges are visible when you build your sizing sheet and cost projections. When you are assessing providers and need a side-by-side, get an Ollang walkthrough tailored to your stack.

Building a Throughput Model

Throughput planning answers a deceptively simple question: can your translation API stack handle the volume you need, when you need it?

Queries Per Second, Concurrency, and Job Duration

Start with three numbers: peak queries per second (QPS), maximum concurrent connections your client will open, and average job duration (the wall-clock time from request submission to response receipt). These three variables are interdependent. If your average job duration is 200 milliseconds and you need to sustain 50 QPS, you need at least 10 concurrent connections just to keep up, and more in practice, because latency variance means some requests will take longer.

A basic model:

Build this table from your own traffic data. If you are translating user-generated content in real time, your QPS profile will be spiky and you need to model the peak, not the average.

Provider Rate Limits and Quota Structures

Every major translation API enforces rate limits, but the structure varies. Some providers cap QPS globally across your project; others cap per region or per API key. Quota may be expressed as characters per minute rather than requests per second, which means a single large-document request can consume the same quota as hundreds of short-string requests.

Key questions to answer during provider evaluation:

• Is the rate limit per API key, per project, or per billing account?
• Can you request a quota increase, and what is the lead time?
• Does the provider distinguish between synchronous and asynchronous quotas?
• Are batch endpoints subject to different (usually more generous) limits?

Document these answers in your sizing sheet. A provider that offers the lowest per-character price but caps you at 20 QPS may be unusable for a real-time chat translation feature.

Regional Deployment and Network Proximity

API latency is bounded from below by physics: a round trip from São Paulo to a translation endpoint in Northern Virginia adds on the order of 120-150 ms of network latency before the engine even begins processing. If you serve users globally, regional routing matters.

Some providers operate translation endpoints in multiple regions. Others run from a single region and rely on CDN edge nodes for caching but not for inference. When you model throughput, measure latency from each region where your application servers run, not from your laptop. Use the provider's own latency metrics if published, and validate with synthetic benchmarks from your actual deployment zones.

Latency Optimization Strategies

Low latency is not just a performance nicety, in synchronous translation workflows (live chat, in-app UI rendering, real-time subtitles), every millisecond of API latency is a millisecond of user-visible delay.

Batching vs Fragmentation: Finding the Sweet Spot

Batching multiple strings into a single API call reduces per-request overhead (TLS handshake, HTTP headers, connection setup) and often improves throughput. But over-batching introduces its own problem: if you pack 500 strings into one request, the entire batch blocks until the slowest string is translated, and a single malformed segment can fail the whole payload on some providers.

The sweet spot depends on your string profile. For UI localization with short, uniform strings, batches of 50-100 segments typically balance overhead reduction against tail-latency risk. For long-form content, sending one document per request and parallelizing across connections usually wins.

A practical batching heuristic:

- Short strings (< 100 chars): Batch aggressively, 50-128 per request.

- Medium strings (100-1,000 chars): Batch moderately, 10-20 per request.

- Long documents (> 1,000 chars): Send individually, parallelize at the client.

HTTP/2 Multiplexing and Connection Reuse

HTTP/2 multiplexing lets you send multiple API requests over a single TCP connection without head-of-line blocking at the HTTP layer. If your HTTP client library supports it, and most modern ones do, enabling HTTP/2 can cut effective latency for high-concurrency workloads by eliminating repeated TLS handshakes and TCP slow-start penalties.

Ensure your client reuses connections. Creating a new HTTPS connection for every translation request adds 50-150 ms of overhead per call. Connection pooling is table stakes.

import httpx

# Reuse a single HTTP/2-capable client across requests
client = httpx.Client(http2=True, timeout=10.0)

response = client.post(
"https://api.provider.example/v2/translate",
json={"q": ["Hello", "Goodbye"], "target": "de"},
headers={"Authorization": "Bearer YOUR_KEY"}
)

Edge Caching for Deterministic UI Strings

Many UI strings are deterministic: the source text never changes, and the translation should not change either. Button labels, navigation items, form field names, and static error messages are prime candidates for edge caching.

Place a caching layer, Redis, a CDN with cache keys derived from source text and target language, or an in-application LRU cache, between your application and the translation API. Cache hits bypass the API entirely, eliminating both latency and cost. Even a modest cache-hit ratio of 60-70% on UI strings can cut API call volume dramatically, because UI strings are requested far more frequently than long-form content.

Use a cache key structure that includes the source text hash, target locale, glossary version, and model version. If any of these change, the cache entry should be invalidated.

Retry Budgets, Timeout Envelopes, and Idempotency

Failures happen. The question is whether your retry logic makes things better or worse.

Designing a Retry Budget

A retry budget caps the total number of retries your system will attempt within a time window, preventing a cascade where retried requests overwhelm a recovering provider and extend the outage. A common pattern is to allow no more than 10-20% of total requests to be retries within any rolling 60-second window. Once the budget is exhausted, fail fast and surface the error to the caller.

Combine retry budgets with exponential backoff and jitter:

import random
import time

def backoff_delay(attempt, base=0.5, max_delay=8.0):
delay = min(base * (2 ** attempt), max_delay)
return delay + random.uniform(0, delay * 0.1)

Timeout Envelopes: Layered Deadlines

Set timeouts at three layers:

1. Connection timeout (1-3 seconds): How long to wait for a TCP/TLS handshake.

2. Request timeout (5-15 seconds): How long to wait for the full response after the request is sent.

3. End-to-end deadline (10-30 seconds): The maximum wall-clock time the calling service will wait, inclusive of retries.

The end-to-end deadline is the most important. If your first attempt times out at 10 seconds and you retry twice, your end-to-end deadline must accommodate at least 30 seconds of potential waiting, or you must reduce the per-attempt timeout to fit within a tighter envelope. Never let retries extend beyond the deadline the upstream caller is willing to tolerate.

Idempotency to Prevent Duplicate Charges

When a request times out, you do not know whether the provider processed it. If you retry and the original request did succeed, you may be billed twice for the same translation. Some providers support idempotency keys, a unique identifier you attach to each request so that the provider de-duplicates retries server-side.

If the provider does not support idempotency natively, implement it client-side: hash the source text, target language, and glossary version into a request fingerprint, and check your cache or a deduplication store before sending a retry. This is especially important for large-document translations where a single duplicate request can cost meaningfully.

{
"idempotency_key": "a3f8c9e1-4b2d-4e7a-9c3f-1d2e3f4a5b6c",
"source_lang": "en",
"target_lang": "ja",
"text": "Your subscription has been renewed."
}

If your pipeline involves both MT and LLM QA steps and you want to ensure idempotent, cost-controlled execution across the chain, explore how Ollang orchestrates multi-step translation workflows.

Estimating Monthly Cost Under Real-World Conditions

A sizing sheet is only useful if it reflects how your system actually behaves, not how it behaves in a vacuum.

Modeling Cache-Hit Ratios

Cache-hit ratio is the single largest lever on translation API cost. The table below illustrates how different cache-hit ratios affect monthly API call volume for a workload of 10 million source characters per month:

Achieving a 70% or higher cache-hit ratio is realistic for applications with a significant UI string component. Content-heavy applications with mostly unique text will see lower ratios, but even 30-40% caching of repeated headers, footers, and boilerplate can meaningfully reduce spend.

MT + LLM QA Cost Mixes

If your pipeline uses machine translation followed by an LLM-based quality review, model the two costs separately. MT character pricing and LLM token pricing operate on different scales, and the QA step may process both the source and the translated output (doubling the token count relative to the source alone).

A simplified monthly cost formula:

Monthly Cost = (chars_to_MT × MT_rate) + (tokens_to_QA × QA_rate)

Where tokens_to_QA includes both source and target tokens, plus any system prompt overhead. For a 10-million-character English-to-German workload with a 70% cache-hit ratio and LLM QA applied to 100% of MT output:

- MT cost: 3,000,000 chars × $20/M = $60

- QA cost: ~2,400,000 tokens (source + target, assuming ~0.8 token-per-char average for combined EN+DE) × an LLM token rate

The QA leg can easily equal or exceed the MT leg. Decide which content types warrant full QA and which can ship with MT-only output. Marketing copy and legal text may justify the QA spend; internal knowledge base articles may not.

Building Your Sizing Sheet

Your sizing sheet should capture these inputs:

- Monthly source volume (characters) by content type

- Target languages and any per-pair surcharges

- Expected cache-hit ratio by content type

- QA coverage (percentage of output routed through LLM review)

- Peak QPS and concurrency requirements

- Retry rate (historical or estimated) and idempotency coverage

- Glossary and custom model fees (monthly or per-request)

Populate it with real data from your analytics and staging environment. Run the model against at least two providers to compare total cost of ownership, not just unit price. If you want a sanity check and recommendations before you lock decisions, let Ollang review your numbers.

Frequently Asked Questions

How do I calculate the right number of concurrent API connections?

Multiply your peak QPS by your P95 or P99 response time (in seconds) to get the minimum concurrency. Then add headroom, typically 1.5× to 2×, to absorb latency variance. For example, 40 QPS with a P99 of 600 ms requires at least 24 connections, so provision 36-48. Monitor connection utilization in production and adjust.

What cache-hit ratio should I target for UI string translation?

For applications with a stable UI, 70-90% is achievable and should be the target. The key is caching at the right granularity: cache by source text hash, target locale, and glossary version. Invalidate on glossary or model changes. Even partial caching of high-frequency strings delivers disproportionate cost and latency savings.

How do I avoid being double-billed when retrying a timed-out request?

Use idempotency keys if the provider supports them. If not, implement client-side deduplication by fingerprinting each request (source text hash + target language + glossary version) and checking a local store before retrying. For large documents, this is critical, a single duplicate request can represent significant cost.

Should I use synchronous or asynchronous translation API calls?

Use synchronous calls for real-time, user-facing workflows where latency matters (live chat, in-app UI). Use asynchronous (batch) calls for background processing of large volumes, documentation, knowledge bases, bulk content migration. Async endpoints typically have more generous rate limits and lower per-unit costs, but introduce polling or webhook complexity. Many production systems use both patterns for different content types. Ollang can orchestrate and route content to the appropriate pattern as part of a unified pipeline.

Next Steps: Right-Size Your Translation Infrastructure

A well-built sizing sheet turns translation API selection from a gut decision into a data-driven architecture choice. The variables, pricing model, rate limits, regional latency, caching potential, retry overhead, and QA mix, interact in ways that only become visible when you model them together against your actual workload.

Book a Demo