Translation quality breaks silently. A misplaced placeholder crashes a mobile app. A truncated string hides a checkout button. A culturally inappropriate term slips into a regulated market. When you localize at scale, thousands of segments across dozens of locales per sprint, manual QA cannot keep pace. The question is not whether to automate quality assurance, but how to architect a QA pipeline that catches defects programmatically, scores severity reliably, and hands off to human reviewers only when it matters.
This article walks through building that pipeline end to end: automated checks via API, LLM-based quality estimation, scoring thresholds, structured human review handoff, and regression detection, all orchestrated through translation API endpoints, webhooks, and status transitions. The goal is a system where every translated segment has a verifiable quality record and a clear audit trail.
If your localization pipeline already struggles with inconsistent quality across languages, explore how Ollang can streamline your QA workflows. Ollang centralizes QA automation, scoring, and reviewer orchestration behind a single API to reduce handoff latency and surface a complete audit trail.
Automated QA Checks via API
Automated checks are the first line of defense. They run instantly, cost nothing per execution after setup, and catch the mechanical defects that account for a large share of localization bugs. The key is structuring these checks as composable API calls that run against every segment before it enters your product.
Length, Placeholder, and Tag Consistency Validators
Three validators should run on every translated segment without exception.
- Length validation compares the character or word count of the target against the source, flagged against locale-specific thresholds. German translations commonly expand 30% or more relative to English, while Chinese often contracts. A simple API check sends the source and target strings and returns a ratio:
{
"source": "Your order has been confirmed.",
"target": "Ihre Bestellung wurde bestätigt.",
"source_locale": "en",
"target_locale": "de",
"char_ratio": 1.07,
"threshold_max": 1.40,
"status": "pass"
}
Any segment exceeding the threshold gets flagged for UI truncation risk.
- Placeholder validation ensures that variables like {username}, %d, or {{count}} survive translation intact. The check extracts all placeholder tokens from source and target using regex patterns and compares the sets. A mismatch, a missing {price} or a duplicated %s, is a critical defect that will cause runtime errors.
- Tag consistency applies to formats like XLIFF, HTML, and XML where inline markup (<b>, <a href="...">, <x id="1"/>) must be preserved. The validator parses both source and target, compares tag sequences, and flags missing, added, or reordered tags. Mismatched tags can break rendering or, worse, expose raw markup to end users.
These three checks should be implemented as a single composite endpoint or a batch call that returns per-segment results with severity levels (critical, major, minor).
Profanity, PII, and Format Validators
Beyond structural integrity, content-level validators catch problems that automated translation engines introduce or fail to filter.
- Profanity and offensive content filters run the target text against locale-specific dictionaries and pattern matchers. This is especially important for user-generated content workflows and for markets with strict advertising standards. The API call should accept the target text and locale, and return flagged terms with positions:
{
"target": "...",
"target_locale": "ja",
"flags": [
{ "term": "...", "position": [12, 18], "severity": "major", "category": "profanity" }
]
}
- PII detection scans for patterns like email addresses, phone numbers, national ID formats, and credit card numbers that should have been masked before translation or that the translation engine inadvertently introduced. Regex-based detectors combined with locale-aware format libraries (e.g., recognizing German IBAN formats or Japanese phone number patterns) provide reliable coverage.
- Format validators verify that the output conforms to the expected file format, valid JSON, properly escaped strings, well-formed XML, correct ICU MessageFormat syntax. A translation that breaks JSON parsing will break your build. These validators should return structured errors with line numbers and expected vs. actual syntax.
All of these checks run in milliseconds and should be mandatory gates before any segment advances in your pipeline.
LLM-Based Quality Estimation and Scoring
Automated validators catch mechanical defects, but they cannot assess whether a translation is fluent, accurate, or contextually appropriate. This is where quality estimation models, increasingly LLM-powered, fill the gap.
Side-by-Side Comparison and Score Thresholds
Quality estimation (QE) assigns a score to a translated segment without requiring a human reference translation. Traditional QE models like COMET produce scores on a continuous scale by comparing source, target, and optionally a reference. LLM-based QE takes this further by evaluating fluency, adequacy, and terminology adherence in a single pass.
A typical QE API request sends the source, target, and context (surrounding segments, glossary terms, domain):
{
"source": "Tap 'Continue' to finalize your payment.",
"target": "Appuyez sur « Continuer » pour finaliser votre paiement.",
"source_locale": "en",
"target_locale": "fr",
"context": {
"domain": "fintech",
"glossary_id": "gl_payments_fr"
}
}
The response includes dimensional scores and an overall quality estimate:
{
"overall_score": 0.91,
"fluency": 0.94,
"accuracy": 0.89,
"terminology": 0.92,
"threshold": 0.85,
"action": "auto_approve"
}
The threshold field drives the decision logic. Segments scoring above the threshold are auto-approved. Segments below it are routed to human review. You can set different thresholds per content type, marketing copy might demand 0.90, while internal documentation accepts 0.80.
Side-by-side comparison endpoints let you evaluate multiple translation candidates (e.g., outputs from different engines or engine versions) and select the best one programmatically. The API returns ranked results with per-candidate scores, enabling A/B testing of translation providers at the segment level.
When you are ready to tie scores directly to routing logic, Route by quality score with Ollang’s scoring API.
MQM and COMET Metrics in Practice
The Multidimensional Quality Metrics (MQM) framework provides the industry-standard taxonomy for translation errors. MQM categorizes issues by type (accuracy, fluency, terminology, style, locale convention) and severity (critical, major, minor, neutral). An MQM-scored segment example:
- Mistranslation, Major (penalty -5)
- Grammar, Minor (penalty -1)
- Terminology, Major (penalty -5)
- Total penalty: -11
MQM scores are typically expressed as error penalties per thousand words. A score below a defined penalty threshold (e.g., fewer than 5 penalty points per 1,000 words) indicates acceptable quality.
COMET (Crosslingual Optimized Metric for Evaluation of Translation) provides a learned metric that correlates more closely with human judgments than traditional metrics like BLEU. COMET models are available as open-source checkpoints and can be deployed behind an internal API endpoint or consumed through provider APIs that support them.
In practice, combine both: use COMET for fast, reference-free scoring during automated QA, and apply MQM for structured human evaluation when segments are escalated. This dual approach gives you speed at the automated layer and diagnostic depth at the human layer.