Most localization teams discover quality problems the same way their users do: after release. A mishandled placeholder crashes a UI string. A glossary term drifts across product lines. A tone shift in a legal disclaimer introduces liability. These failures share a root cause, quality review is manual, episodic, and disconnected from the translation pipeline.
API-driven quality review changes the equation. By wiring quality estimation models, back-translation checks, LLM-based style audits, and webhook-driven pass/fail gates directly into your translation workflow, you can catch regressions automatically, log every decision for audit, and escalate to human reviewers only when the system flags genuine risk. This article walks through the architecture, scoring schemas, thresholds, and code patterns you need to build an auditable, API-first QA layer that runs before a single translated string reaches production. Ollang ties these checks into CI/CD, TMS, and review workflows to deliver auditable quality controls at enterprise scale.
If you're evaluating how to embed automated quality controls into your localization stack, explore how Ollang's platform handles this end-to-end.
Why Manual Translation QA Fails at Scale
Manual quality review worked when products shipped in three languages and release cycles lasted months. It does not work when a SaaS product pushes daily deploys across forty locales.
The core problems are structural:
- Throughput mismatch. Even an experienced reviewer can only evaluate on the order of low thousands of words per hour under MQM. A modern MT pipeline can produce that volume in seconds.
- Inconsistent coverage. Without statistical sampling, reviewers tend to check what's convenient, recent strings, high-visibility pages, leaving long-tail content unexamined.
- Late feedback loops. When review happens after translation is "done," rework is expensive and disruptive. Developers have moved on; context is lost.
- No audit trail. Spreadsheet-based review produces no structured data. You cannot trend quality over time, compare vendors, or prove compliance.
The result is a quality process that is simultaneously too slow and too unreliable. API-driven QA addresses each of these failures by making quality checks programmatic, continuous, and traceable.
Quality Estimation via API
COMET, BLEU, and Other Scoring Proxies
Quality estimation (QE) refers to predicting translation quality without a human reference translation. This is distinct from quality evaluation, which compares output against a gold-standard reference.
COMET (Crosslingual Optimized Metric for Evaluation of Translation) is a neural metric trained on human judgments. It accepts a source segment, a machine-translated segment, and optionally a reference, then returns a score that correlates strongly with human MQM ratings. Research from the WMT Metrics Shared Task has consistently shown COMET outperforming surface-level metrics in system-level correlation with human judgment.
BLEU (Bilingual Evaluation Understudy) remains widely used but measures n-gram overlap against a reference translation. It is fast and cheap to compute but poorly captures meaning preservation, fluency, or terminology adherence. It is best used as a regression detector, a sudden BLEU drop between builds signals something changed, rather than an absolute quality measure.
Other proxies worth considering:
| Metric | Requires Reference? | Best Use Case | Compute Cost |
|---|---|---|---|
| COMET-QE | No | Referenceless scoring at scale | Medium (GPU) |
| COMET | Yes (optional) | Segment-level quality ranking | Medium (GPU) |
| BLEU | Yes | Regression detection across builds | Low (CPU) |
| chrF | Yes | Character-level similarity, morphologically rich languages | Low (CPU) |
| BERTScore | Yes | Semantic similarity | Medium (GPU) |
Calling a QE Endpoint and Interpreting Scores
Most QE models can be self-hosted behind a REST API or accessed through managed inference services. A typical request sends a batch of source-translation pairs and receives segment-level scores:
POST /v1/quality-estimate
Content-Type: application/json
{
"model": "comet-qe",
"segments": [
{
"id": "str_1042",
"source": "Your subscription renews on {{date}}.",
"translation": "Ihr Abonnement verlängert sich am {{date}}.",
"source_lang": "en",
"target_lang": "de"
},
{
"id": "str_1043",
"source": "Cancel anytime from your account settings.",
"translation": "Kündigen Sie jederzeit in Ihren Kontoeinstellungen.",
"source_lang": "en",
"target_lang": "de"
}
]
}
The response returns per-segment scores:
{
"model": "comet-qe",
"results": [
{ "id": "str_1042", "score": 0.87, "confidence": 0.92 },
{ "id": "str_1043", "score": 0.94, "confidence": 0.96 }
]
}
COMET scores are typically normalized so that higher values indicate better quality. There is no universal "good" threshold, the right cutoff depends on your content type, risk tolerance, and the specific model version. However, a practical starting pattern is to establish a baseline by scoring a set of human-reviewed translations and using the distribution to set percentile-based thresholds.
Setting Thresholds and Scoring Schemas
A scoring schema maps raw QE scores to actionable decisions. Here is an example schema for a product with moderate risk tolerance:
| Score Range | Action | Escalation |
|---|---|---|
| ≥ 0.90 | Auto-approve | None |
| 0.75, 0.89 | Flag for sampling review | Add to review queue |
| 0.60, 0.74 | Require human review | Assign to linguist |
| < 0.60 | Block release, trigger rollback | Alert localization lead |
These thresholds should be calibrated per language pair. High-resource pairs like English→German will produce higher baseline scores than low-resource pairs like English→Khmer. Applying a single global threshold across all locales will either over-escalate high-resource languages or under-flag low-resource ones.
Store the schema as configuration, not code. When you need to tighten quality for a regulated content type (legal, medical), you adjust the threshold without redeploying your pipeline.
Back-Translation Checks
Dual-Pass Translation Architecture
Back-translation is a straightforward but powerful verification technique: translate the output back into the source language, then compare the back-translated text against the original source. Semantic divergence between the two signals a potential quality issue.
The architecture involves two sequential API calls:
- Forward pass: Source (English) → Target (German) via your primary MT engine.
- Reverse pass: Target (German) → Source (English) via the same or a different MT engine.
Using a different engine for the reverse pass reduces the risk of systematic blind spots. If your primary engine consistently mistranslates a term and its reverse pass has the same bias, back-translation will not catch the error. A second engine provides an independent signal.
import requests
def back_translate(source_text, source_lang, target_lang, forward_api, reverse_api):
# Forward pass
fwd_response = requests.post(forward_api, json={
"text": source_text,
"source": source_lang,
"target": target_lang
})
translated = fwd_response.json()["translation"]
# Reverse pass (different engine)
rev_response = requests.post(reverse_api, json={
"text": translated,
"source": target_lang,
"target": source_lang
})
back_translated = rev_response.json()["translation"]
return {
"source": source_text,
"translation": translated,
"back_translation": back_translated
}
Computing Diffs and Semantic Similarity
Raw string comparison between source and back-translation is noisy, paraphrases will differ lexically even when meaning is preserved. You need semantic comparison.
A practical approach combines two signals:
- Token-level diff to identify specific insertions, deletions, and substitutions. Libraries like Python's difflib or dedicated diff APIs produce structured change lists.
- Embedding cosine similarity to measure overall semantic alignment. Encode both the source and back-translation with a multilingual sentence encoder (such as sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) and compute cosine similarity.
{
"id": "str_1042",
"source": "Your subscription renews on {{date}}.",
"back_translation": "Your subscription will be renewed on {{date}}.",
"token_diff": [
{ "type": "replace", "source_span": "renews", "bt_span": "will be renewed" }
],
"cosine_similarity": 0.96,
"verdict": "pass"
}
A cosine similarity above 0.93 with no critical token diffs (missing placeholders, negation changes, number alterations) generally indicates acceptable fidelity. Below 0.85, the segment warrants human review regardless of the QE score.
Back-translation is computationally expensive, it doubles your MT API calls. Use it selectively: on segments that fall in the QE "flag for review" band, on high-risk content types, or as a periodic batch audit rather than on every segment.
LLM-Based Style, Glossary, and Placeholder Checks
Prompt Engineering for Automated Review
Large language models add a layer of quality review that rule-based systems and QE metrics cannot provide: contextual judgment about style, tone, terminology consistency, and formatting integrity.
The key is structured prompting. Rather than asking an LLM "Is this translation good?", you provide explicit review criteria and request structured output:
POST /v1/chat/completions
{
"model": "gpt-4o",
"response_format": { "type": "json_object" },
"messages": [
{
"role": "system",
"content": "You are a translation quality reviewer. Evaluate the translation against the provided glossary, style guide rules, and source text. Return a JSON object with issue annotations."
},
{
"role": "user",
"content": "Source (en): Cancel your subscription anytime from Account Settings.\nTranslation (de): Kündigen Sie Ihr Abonnement jederzeit über die Kontoeinstellungen.\nGlossary: {'Account Settings': 'Kontoeinstellungen', 'subscription': 'Abonnement'}\nStyle rules: Use formal address (Sie). Maximum segment length: 80 characters. Preserve all placeholders.\n\nReview this translation and return issues as JSON."
}
]
}
The response provides actionable annotations:
{
"segment_id": "str_1044",
"issues": [],
"glossary_compliance": true,
"style_compliance": true,
"placeholder_integrity": true,
"overall_verdict": "pass",
"confidence": 0.95
}
When issues exist, the LLM should categorize them using a taxonomy compatible with your downstream tooling, ideally aligned with MQM issue types such as terminology, accuracy, fluency, style, and locale convention.
Detecting Glossary Drift and Placeholder Corruption
Two of the most damaging translation defects are glossary violations and placeholder corruption, and both are well-suited to automated detection.
Glossary checks compare translated terms against a term base. An LLM can do this contextually, recognizing that "Konto" is an acceptable translation of "account" in some contexts but "Kontoeinstellungen" is the required compound for "Account Settings." A simpler, faster alternative is a deterministic glossary lookup: extract glossary source terms from the segment, find their required translations, and verify they appear in the target. Use the LLM for ambiguous cases only.
Placeholder integrity is more mechanical. Placeholders like {{date}}, %s, {0}, or <x id="1"/> must appear in the translation exactly as they appear in the source. A regex-based check is faster and more reliable than an LLM for this:
import re
def check_placeholders(source, translation):
pattern = r'\{\{.*?\}\}|%[sd]|\{[0-9]+\}|<x[^>]*/>'
source_ph = sorted(re.findall(pattern, source))
target_ph = sorted(re.findall(pattern, translation))
return {
"match": source_ph == target_ph,
"missing": [p for p in source_ph if p not in target_ph],
"extra": [p for p in target_ph if p not in source_ph]
}
Layer these checks: run deterministic placeholder and glossary verification first (fast, cheap, definitive), then route ambiguous or style-related issues to the LLM. This keeps costs down and reduces false positives.