API-Driven Localization QA: Automations, Scoring, and Handoff

Translation quality breaks silently. A misplaced placeholder crashes a mobile app. A truncated string hides a checkout button. A culturally inappropriate term slips into a regulated market. When you localize at scale, thousands of segments across dozens of locales per sprint, manual QA cannot keep pace. The question is not whether to automate quality assurance, but how to architect a QA pipeline that catches defects programmatically, scores severity reliably, and hands off to human reviewers only when it matters.

This article walks through building that pipeline end to end: automated checks via API, LLM-based quality estimation, scoring thresholds, structured human review handoff, and regression detection, all orchestrated through translation API endpoints, webhooks, and status transitions. The goal is a system where every translated segment has a verifiable quality record and a clear audit trail.

If your localization pipeline already struggles with inconsistent quality across languages, explore how Ollang can streamline your QA workflows. Ollang centralizes QA automation, scoring, and reviewer orchestration behind a single API to reduce handoff latency and surface a complete audit trail.

Automated QA Checks via API

Automated checks are the first line of defense. They run instantly, cost nothing per execution after setup, and catch the mechanical defects that account for a large share of localization bugs. The key is structuring these checks as composable API calls that run against every segment before it enters your product.

Length, Placeholder, and Tag Consistency Validators

Three validators should run on every translated segment without exception.

• Length validation compares the character or word count of the target against the source, flagged against locale-specific thresholds. German translations commonly expand 30% or more relative to English, while Chinese often contracts. A simple API check sends the source and target strings and returns a ratio:

{
"source": "Your order has been confirmed.",
"target": "Ihre Bestellung wurde bestätigt.",
"source_locale": "en",
"target_locale": "de",
"char_ratio": 1.07,
"threshold_max": 1.40,
"status": "pass"
}

Any segment exceeding the threshold gets flagged for UI truncation risk.

• Placeholder validation ensures that variables like {username}, %d, or {{count}} survive translation intact. The check extracts all placeholder tokens from source and target using regex patterns and compares the sets. A mismatch, a missing {price} or a duplicated %s, is a critical defect that will cause runtime errors.
• Tag consistency applies to formats like XLIFF, HTML, and XML where inline markup (<b>, <a href="...">, <x id="1"/>) must be preserved. The validator parses both source and target, compares tag sequences, and flags missing, added, or reordered tags. Mismatched tags can break rendering or, worse, expose raw markup to end users.

These three checks should be implemented as a single composite endpoint or a batch call that returns per-segment results with severity levels (critical, major, minor).

Profanity, PII, and Format Validators

Beyond structural integrity, content-level validators catch problems that automated translation engines introduce or fail to filter.

• Profanity and offensive content filters run the target text against locale-specific dictionaries and pattern matchers. This is especially important for user-generated content workflows and for markets with strict advertising standards. The API call should accept the target text and locale, and return flagged terms with positions:

{
"target": "...",
"target_locale": "ja",
"flags": [
{ "term": "...", "position": [12, 18], "severity": "major", "category": "profanity" }
]
}

• PII detection scans for patterns like email addresses, phone numbers, national ID formats, and credit card numbers that should have been masked before translation or that the translation engine inadvertently introduced. Regex-based detectors combined with locale-aware format libraries (e.g., recognizing German IBAN formats or Japanese phone number patterns) provide reliable coverage.
• Format validators verify that the output conforms to the expected file format, valid JSON, properly escaped strings, well-formed XML, correct ICU MessageFormat syntax. A translation that breaks JSON parsing will break your build. These validators should return structured errors with line numbers and expected vs. actual syntax.

All of these checks run in milliseconds and should be mandatory gates before any segment advances in your pipeline.

LLM-Based Quality Estimation and Scoring

Automated validators catch mechanical defects, but they cannot assess whether a translation is fluent, accurate, or contextually appropriate. This is where quality estimation models, increasingly LLM-powered, fill the gap.

Side-by-Side Comparison and Score Thresholds

Quality estimation (QE) assigns a score to a translated segment without requiring a human reference translation. Traditional QE models like COMET produce scores on a continuous scale by comparing source, target, and optionally a reference. LLM-based QE takes this further by evaluating fluency, adequacy, and terminology adherence in a single pass.

A typical QE API request sends the source, target, and context (surrounding segments, glossary terms, domain):

{
"source": "Tap 'Continue' to finalize your payment.",
"target": "Appuyez sur « Continuer » pour finaliser votre paiement.",
"source_locale": "en",
"target_locale": "fr",
"context": {
"domain": "fintech",
"glossary_id": "gl_payments_fr"
}
}

The response includes dimensional scores and an overall quality estimate:

{
"overall_score": 0.91,
"fluency": 0.94,
"accuracy": 0.89,
"terminology": 0.92,
"threshold": 0.85,
"action": "auto_approve"
}

The threshold field drives the decision logic. Segments scoring above the threshold are auto-approved. Segments below it are routed to human review. You can set different thresholds per content type, marketing copy might demand 0.90, while internal documentation accepts 0.80.

Side-by-side comparison endpoints let you evaluate multiple translation candidates (e.g., outputs from different engines or engine versions) and select the best one programmatically. The API returns ranked results with per-candidate scores, enabling A/B testing of translation providers at the segment level.

When you are ready to tie scores directly to routing logic, Route by quality score with Ollang’s scoring API.

MQM and COMET Metrics in Practice

The Multidimensional Quality Metrics (MQM) framework provides the industry-standard taxonomy for translation errors. MQM categorizes issues by type (accuracy, fluency, terminology, style, locale convention) and severity (critical, major, minor, neutral). An MQM-scored segment example:

• Mistranslation, Major (penalty -5)
• Grammar, Minor (penalty -1)
• Terminology, Major (penalty -5)
• Total penalty: -11

MQM scores are typically expressed as error penalties per thousand words. A score below a defined penalty threshold (e.g., fewer than 5 penalty points per 1,000 words) indicates acceptable quality.

COMET (Crosslingual Optimized Metric for Evaluation of Translation) provides a learned metric that correlates more closely with human judgments than traditional metrics like BLEU. COMET models are available as open-source checkpoints and can be deployed behind an internal API endpoint or consumed through provider APIs that support them.

In practice, combine both: use COMET for fast, reference-free scoring during automated QA, and apply MQM for structured human evaluation when segments are escalated. This dual approach gives you speed at the automated layer and diagnostic depth at the human layer.

Triggering Human Review: Webhooks, Assignments, and Status Transitions

Automated QA identifies problems. The handoff to human reviewers is where most pipelines break down, tasks get lost in email threads, assignments lack context, and there is no structured way to track resolution. API-driven handoff solves this.

Webhook-Driven Escalation Patterns

When a segment fails automated QA or scores below the quality threshold, a webhook fires to your review orchestration system. The webhook payload should contain everything a reviewer needs to act immediately:

{
"event": "segment.qa_failed",
"timestamp": "2025-01-15T09:32:00Z",
"segment_id": "seg_4821",
"project_id": "proj_payments_q1",
"source_locale": "en",
"target_locale": "de",
"source_text": "Your refund will be processed within 3-5 business days.",
"target_text": "Ihre Rückerstattung wird innerhalb von 3-5 Werktagen bearbeitet.",
"qa_results": {
"overall_score": 0.72,
"failed_checks": ["terminology", "accuracy"],
"mqm_errors": [
{ "type": "mistranslation", "severity": "major", "span": [45, 56] }
]
},
"glossary_violations": ["Rückerstattung → Erstattung (preferred)"],
"assigned_reviewer": null
}

Your orchestration layer receives this webhook and applies routing rules: assign to a reviewer with the right language pair and domain expertise, set a due date based on your SLA, and transition the segment's status from translated to review_pending.

Webhook retry logic matters here. If your review system is temporarily unavailable, the webhook should retry with exponential backoff and eventually dead-letter the event for manual triage. Most TMS and translation API platforms support configurable retry policies.

If you are formalizing this handoff today, see how Ollang’s platform handles QA orchestration end to end.

Request/Response Shapes for Review Tasks and Comment Threads

The review assignment itself is created via an API call. A well-structured review task includes the segment, its QA results, relevant glossary entries, translation memory matches, and any prior reviewer comments:

POST /v1/review-tasks
{
"segment_id": "seg_4821",
"project_id": "proj_payments_q1",
"reviewer_id": "rev_maria_de",
"priority": "high",
"due_at": "2025-01-16T17:00:00Z",
"context": {
"qa_score": 0.72,
"failed_checks": ["terminology", "accuracy"],
"tm_match": { "score": 0.88, "source": "tm_payments_v3" },
"glossary_terms": [
{ "source": "refund", "approved_target": "Erstattung" }
]
}
}

The response confirms the task creation and returns a task ID for tracking:

{
"task_id": "task_9920",
"status": "assigned",
"reviewer_id": "rev_maria_de",
"created_at": "2025-01-15T09:33:00Z",
"sla_deadline": "2025-01-16T17:00:00Z"
}

Comment threads attach to review tasks, enabling structured conversation between reviewers, project managers, and automated systems. A comment might flag a domain-specific nuance or request clarification from the original requester:

POST /v1/review-tasks/task_9920/comments
{
"author": "rev_maria_de",
"body": "Source says 'refund', glossary specifies 'Erstattung' but 'Rückerstattung' is standard in Austrian German. Requesting PM decision.",
"tagged_users": ["pm_alex"]
}

This creates an auditable thread directly linked to the segment, the QA failure, and the resolution.

Retranslate Flows and Status Transitions

When a reviewer rejects a translation, the system needs a clean retranslation path. The reviewer updates the task status to rejected with a reason code, which triggers a retranslation:

PATCH /v1/review-tasks/task_9920
{
"status": "rejected",
"reason": "terminology_violation",
"action": "retranslate",
"instructions": "Use 'Erstattung' per glossary. Maintain formal register."
}

The retranslation request feeds the reviewer’s instructions back into the translation engine as additional context, improving the next attempt. The segment transitions through a defined state machine:

• translated, QA passes → approved
• translated, QA fails → review_pending
• review_pending, Reviewer assigned → in_review
• in_review, Reviewer approves → approved
• in_review, Reviewer rejects → retranslate_pending
• retranslate_pending, New translation received → translated (re-enters QA)

Every transition is logged with a timestamp, actor, and reason, creating a complete audit trail for compliance and SLA tracking.

Sampling Strategies and Regression Detection

Running every check on every segment is ideal for automated validators, but human review at 100% coverage is neither practical nor cost-effective. Sampling strategies let you allocate human effort where it has the highest impact.

Statistical Sampling for Human QA

The most common approach is stratified random sampling: divide your segments by risk category (content type, locale, translation engine, newness) and sample at different rates.

• High-risk content (legal, medical, financial, UI strings affecting transactions): 20-30% human review rate, or 100% for critical paths.
• Medium-risk content (marketing, help articles): 5-10% review rate.
• Low-risk content (internal docs, metadata): 1-2% review rate, or automated QA only.

Within each stratum, select segments randomly to avoid bias. Your sampling API endpoint should accept the project, locale, content type, and desired sample size, and return the selected segment IDs for review assignment.

Adaptive sampling adjusts rates based on observed quality. If a particular locale or engine consistently scores above threshold, reduce the sample rate. If quality drops, increase it automatically. This feedback loop keeps human effort proportional to actual risk.

Detecting Quality Regressions Across Releases

Regression detection compares quality metrics across time, typically across translation engine updates, glossary changes, or new content batches. The pattern is straightforward:

1. Store per-segment QA scores with version metadata (engine version, glossary version, date).
2. After each new batch, compute aggregate metrics (mean COMET score, MQM error density) and compare against the previous baseline.
3. If the delta exceeds a configured threshold (e.g., mean COMET drops by more than 0.03), trigger an alert.

An API endpoint for regression checks might accept two batch IDs and return a comparison:

{
"baseline_batch": "batch_2025q1_v2",
"current_batch": "batch_2025q1_v3",
"locale": "ja",
"baseline_comet_mean": 0.88,
"current_comet_mean": 0.84,
"delta": -0.04,
"regression_detected": true,
"affected_segments": 142,
"recommended_action": "increase_sample_rate"
}

This lets you catch engine downgrades, glossary regressions, or domain drift before they reach production.

Stitching It All Together: SLAs and Auditability

The value of an API-driven QA pipeline is not just catching errors, it is proving that you caught them, when you caught them, and how they were resolved. This matters for regulated industries, enterprise clients who require quality certificates, and internal teams that need to hold vendors accountable.

Defining SLAs in Your Pipeline

Every stage of the QA pipeline should have a measurable SLA:

• Automated QA completion: within seconds of translation delivery (enforced by synchronous API calls or near-real-time async processing).
• Human review assignment: within a defined window after escalation (e.g., 2 hours for high-priority, 24 hours for standard).
• Review completion: based on priority and volume (e.g., 4-hour turnaround for critical segments, 48 hours for standard batches).
• Retranslation cycle: time from rejection to new translation delivery.

Track SLA adherence via API-queryable dashboards. An endpoint that returns SLA metrics per project, locale, and time period enables both real-time monitoring and historical reporting.

Audit Trails and Compliance

Every API call in the pipeline, QA check results, score computations, reviewer assignments, status transitions, comments, approvals, should be stored as immutable audit events. The audit log for a single segment tells the complete story:

1. Segment received from translation engine at T1.
2. Automated QA ran at T1 + 2s: placeholder check passed, terminology check failed.
3. Quality estimation score: 0.74 (below 0.85 threshold).
4. Webhook fired at T1 + 3s, review task created.
5. Reviewer assigned at T1 + 45m.
6. Reviewer commented at T1 + 2h: requested glossary clarification.
7. PM responded at T1 + 3h.
8. Reviewer approved revised translation at T1 + 4h.
9. Segment status transitioned to approved.

This level of traceability is not optional for enterprises operating in regulated markets, it is a baseline requirement.

FAQ

What automated QA checks should run on every translated segment?

At minimum, run length validation, placeholder validation, and tag consistency checks, plus profanity/PII filtering and format validation for user-facing content; these execute in milliseconds. Ollang’s APIs support implementing these checks as mandatory gates so segments never progress without passing them.

How do MQM and COMET metrics differ, and when should I use each?

MQM is a human-annotation framework that provides diagnostic error types and severities, while COMET is a learned metric that gives a fast, reference-free quality score. Use COMET for automated scoring at scale and MQM for structured human evaluation when you need detailed error attribution; both approaches are supported in practical API-driven workflows.

What is a good sampling rate for human review in localization QA?

Stratify by risk: high-risk content may need 20-30% or 100% review, medium-risk 5-10%, and low-risk 1-2% or automated-only, then apply adaptive sampling based on observed quality. Implement sampling via an API so you can automatically adjust rates when regressions or quality anomalies appear.

How do webhooks improve the handoff from automated QA to human review?

Webhooks deliver the full QA context immediately to your orchestration layer so you can auto-assign reviewers, set SLAs, and create audit-linked review tasks without polling. This reduces routing latency from hours to seconds and ensures every failure is tracked end to end.

Start Automating Your Localization QA

Building an API-driven QA pipeline transforms localization quality from a manual bottleneck into a scalable, auditable system. The components are clear: automated validators for mechanical defects, LLM-based scoring for fluency and accuracy, threshold-driven escalation via webhooks, structured human review with comment threads and retranslation flows, and regression detection to catch quality drift before it reaches users.

Ollang provides the execution layer to orchestrate these workflows across text, software, and document localization, with the API integration, quality scoring, and review management built in.

Book a Demo