API-Driven Translation Quality: LQA, Back-Trans, and Scoring

Most localization teams discover quality problems after content ships. A reviewer flags an error in production, a support ticket arrives in a language no one on the team reads, or a regulatory audit surfaces inconsistent terminology across product versions. By then the cost of remediation has multiplied. The solution is to treat translation quality assurance as an automated, API-driven pipeline, one that runs alongside your existing machine translation and TMS workflows, not after them. This article maps the end-to-end architecture: from machine pre-translation and LQA task creation, through automated scoring with models like COMET and BLEURT, to CI-integrated acceptance gates that decide whether a segment ships or routes to a human reviewer. The goal is a measurable, audit-ready QA system you can stand up against any MT provider.

If your localization pipeline lacks programmatic quality controls, explore how Ollang’s API layer can close that gap.

Why Translation Quality Needs API Automation

Manual quality assurance does not scale. When you localize into dozens of languages across continuous release cycles, spreadsheet-based reviews create bottlenecks, introduce subjectivity, and leave no structured audit trail. API automation addresses each of these problems directly.

First, it makes quality measurement consistent. Every segment passes through the same scoring models, the same terminology checks, and the same threshold logic, regardless of which reviewer happens to be available. Second, it makes quality measurable over time. Structured scores stored per segment and per release let you track drift, compare MT engine performance, and justify investment in human post-editing where it actually matters. Third, it connects quality enforcement to your deployment pipeline. A failing quality gate in CI can block a release just as reliably as a failing unit test.

The economics are straightforward, too. Quality estimation API calls, whether to reference-based models or referenceless estimators, cost a fraction of full human review. The goal is not to eliminate human reviewers but to focus their time on the segments that genuinely need attention.

End-to-End QA Pipeline Architecture

A robust API-driven QA pipeline has five stages: machine pre-translation, LQA task creation, automated scoring, acceptance gating, and drift monitoring. Each stage communicates via API calls and webhooks, producing structured data that feeds the next.

Machine Pre-Translation and Segment Extraction

The pipeline begins when source content enters your TMS or content repository. An API call to your MT provider translates each segment, returning the target text along with metadata such as confidence scores, model version, and character counts.

A typical request to an MT endpoint looks like this:

{
"source_lang": "en",
"target_lang": "de",
"segments": [
{"id": "seg_001", "text": "Your subscription renews automatically."},
{"id": "seg_002", "text": "Cancel anytime from Settings."}
],
"glossary_id": "gloss_product_v3",
"model": "latest"
}

The response includes translated segments and any engine-reported quality signals. Store both source and target alongside the segment ID, you will need the pair for every downstream QA step.

At this stage, also extract and preserve any inline markup, placeholders, or ICU MessageFormat tokens. Placeholder corruption is one of the most common MT failure modes, and catching it early avoids expensive downstream debugging.

LQA Task Creation via TMS API

Once segments are pre-translated, create Linguistic Quality Assurance tasks programmatically. Most TMS platforms expose REST endpoints for task or job creation. The API call should specify the project, language pair, workflow step (e.g., lqa_review), assigned reviewer pool, priority, and due date.

{
"project_id": "proj_4821",
"task_type": "lqa_review",
"language_pair": "en-de",
"segments": ["seg_001", "seg_002"],
"priority": "high",
"due_date": "2025-07-20T12:00:00Z",
"mqm_schema": "mqm_v2",
"instructions": "Flag accuracy, fluency, and terminology issues per MQM."
}

The key detail here is the mqm_schema field. By specifying the MQM (Multidimensional Quality Metrics) schema at task creation time, you ensure reviewers annotate errors using a consistent taxonomy. This structured annotation feeds directly into your scoring and reporting layer.

Webhook-Driven Review Workflow

Rather than polling the TMS for task completion, register a webhook endpoint that receives events when reviewers submit annotations. A well-designed webhook payload includes the segment ID, the reviewer's MQM annotations, severity ratings, and a timestamp.

{
"event": "lqa_review_complete",
"task_id": "task_9930",
"segment_id": "seg_001",
"annotations": [
{
"category": "accuracy/mistranslation",
"severity": "major",
"comment": "Subscription translated as membership"
}
],
"reviewer_id": "rev_42",
"completed_at": "2025-07-18T14:32:00Z"
}

Your webhook handler should validate the payload, persist the annotations to your QA database, and trigger the next pipeline stage, automated scoring, for the affected segments.

Automated Quality Scoring via API

Human LQA annotations are essential but slow. Automated quality estimation models let you score every segment at machine speed, reserving human review for the segments that fall below threshold.

Quality Estimation with COMET and BLEURT

COMET and BLEURT are learned evaluation metrics that correlate more strongly with human judgments than traditional string-overlap metrics like BLEU. Both are available as self-hosted models or through API wrappers.

A quality estimation request typically takes a source segment, a machine translation, and optionally a reference translation:

{
"source": "Your subscription renews automatically.",
"translation": "Ihr Abonnement verlängert sich automatisch.",
"reference": "Ihr Abonnement wird automatisch verlängert.",
"model": "comet-22"
}

The response returns a score, COMET scores typically range from 0 to 1, with values above 0.85 generally indicating acceptable quality for most content types. BLEURT returns a similar continuous score.

For referenceless estimation (when no human reference exists), models like COMET-QE evaluate the source-translation pair alone. This is the more practical mode for continuous pipelines, since maintaining reference translations for every segment defeats the purpose of automation.

Store scores per segment alongside the model version and timestamp. You will need this data for threshold gating and drift analysis.

Back-Translation for Accuracy Verification

Back-translation, translating the target text back into the source language and comparing the result to the original, remains a useful heuristic, especially for high-stakes content such as legal or medical text.

The API flow is simple: send the target segment to your MT provider with reversed language direction, then compute a similarity score between the back-translation and the original source. Cosine similarity on sentence embeddings (from models like LaBSE or multilingual E5) outperforms string-level comparison for this purpose.

back_translation = mt_api.translate(source="de", target="en", text=translated_segment)
similarity = embedding_model.cosine_similarity(original_source, back_translation)

A similarity score below your threshold (commonly 0.80 for general content, 0.90 for regulated content) flags the segment for human review.

Back-translation is not infallible, symmetric errors can survive the round trip, but combined with quality estimation scores, it provides a valuable second signal.

LLM-Based Judges and Hallucination Detection

Large language models can serve as automated quality judges, evaluating translations against criteria you define in a prompt. This approach is particularly effective for detecting hallucinations, content in the translation that has no basis in the source, and for checking fluency in context.

A typical LLM judge prompt might instruct the model to evaluate a source-translation pair across accuracy, fluency, and completeness, returning structured JSON with per-dimension scores and explanations. The advantage over learned metrics is flexibility: you can add domain-specific criteria (tone, formality register, brand voice) without retraining a model.

The risk is cost and latency. LLM judge calls are substantially more expensive than COMET or BLEURT inference. Use them selectively, on segments that fall into an ambiguous score range, on high-visibility content, or as a tiebreaker when quality estimation and back-translation disagree.

For hallucination detection specifically, compare named entities, numbers, and dates between source and translation. An LLM judge can also be prompted to identify added information, claims or instructions present in the translation but absent from the source. In regulated domains, hallucinated content is not just a quality issue; it is a compliance risk.

Terminology Adherence and Bias Detection

Glossary Validation via API

Terminology consistency is non-negotiable in technical, legal, and medical localization. Your QA pipeline should validate every translated segment against the active glossary for that project and language pair.

The validation logic is straightforward: for each glossary term that appears in the source segment, check whether the approved target term appears in the translation. Flag mismatches as terminology errors with an MQM category of terminology/inconsistent-use.

{
"segment_id": "seg_001",
"source_term": "subscription",
"expected_target": "Abonnement",
"actual_target": "Mitgliedschaft",
"status": "mismatch"
}

Enhance robustness by handling case, morphology, and locale variants (e.g., compounds in German, pluralization in Romance languages). If your MT provider supports glossary enforcement at translation time (most major providers do), terminology errors in the QA stage indicate either a glossary gap or an enforcement failure, both worth investigating.

Bias and Sensitivity Checks

Automated bias detection in translation is an emerging but increasingly important capability. Gender bias is the most common issue: many languages require gendered forms, and MT engines frequently default to masculine forms regardless of context.

API-based bias checks can flag segments where gendered language was introduced without justification from the source, where culturally sensitive terms were translated literally rather than adapted, or where inclusive language guidelines were not followed.

These checks can be implemented as rule-based validators, as classifier models, or as LLM judge prompts with specific bias-detection instructions. The output should be structured annotations that feed into the same MQM-compatible scoring schema as other quality signals.

Storing Quality Data: MQM Schema Design

Every quality signal, human annotations, automated scores, terminology mismatches, bias flags, needs a consistent storage schema. The MQM framework provides the taxonomy; you need a data model that captures it at the segment level.

A practical per-segment quality record looks like this:

{
"segment_id": "seg_001",
"project_id": "proj_4821",
"language_pair": "en-de",
"source_text": "Your subscription renews automatically.",
"target_text": "Ihr Abonnement verlängert sich automatisch.",
"mt_engine": "engine_x_v4",
"scores": {
"comet_qe": 0.87,
"bleurt": 0.82,
"back_translation_similarity": 0.91,
"llm_judge_accuracy": 4,
"llm_judge_fluency": 5
},
"mqm_annotations": [
{
"category": "accuracy/mistranslation",
"severity": "minor",
"source": "human",
"reviewer_id": "rev_42",
"timestamp": "2025-07-18T14:32:00Z"
}
],
"terminology_checks": [
{
"source_term": "subscription",
"expected": "Abonnement",
"found": "Abonnement",
"status": "pass"
}
],
"decision": "auto_approved",
"release_version": "v2.14.0"
}

Key design decisions:

• Separate scores from annotations. Automated scores are continuous values; human annotations are categorical with severity. Keep them in distinct structures.
• Track provenance. Record which model version, which reviewer, and which glossary version produced each data point. Without provenance, audit trails are meaningless.
• Version by release. Tie every quality record to a release version so you can analyze drift over time.

Store this data in a queryable format, a relational database or a document store with indexing on segment ID, language pair, and release version. You will query it for threshold gating, drift dashboards, and audit reports.

Acceptance Gates in CI/CD

Threshold Logic: Auto-Approve vs. Human Review

The acceptance gate is where quality scores become deployment decisions. Define thresholds that route segments into one of three buckets:

These thresholds are starting points. Calibrate them against your own human judgment data: run a sample of segments through both automated scoring and human review, then adjust thresholds until the auto-approve bucket has an acceptable false-positive rate for your content type and risk tolerance.

In your CI pipeline, the gate is a script that queries the QA database for all segments in the current release, applies threshold logic, and exits with a non-zero code if any segment falls into the auto-reject bucket or if the percentage of segments requiring human review exceeds a project-level cap.

# Simplified CI gate check
qa_result=$(curl -s "$QA_API/release/v2.14.0/gate-status")
if [ "$(echo $qa_result | jq -r '.gate')" != "pass" ]; then
echo "Translation QA gate failed. See report: $(echo $qa_result | jq -r '.report_url')"
exit 1
fi

If you're building CI-integrated quality gates and want to see how this behaves end-to-end, request a live walkthrough.

Drift Monitoring Across Releases

Quality scores for a given language pair should remain stable or improve across releases. A sudden drop in average COMET-QE scores for Japanese, or a spike in terminology mismatches for French legal content, signals a problem, a degraded MT model, a glossary update that introduced conflicts, or a shift in source content complexity.

Build a drift monitor that compares per-language, per-content-type quality distributions between the current release and a rolling baseline (e.g., the trailing five releases). Alert when the mean score drops by more than a configurable delta or when the proportion of segments requiring human review increases significantly.

This monitoring does not require exotic tooling. A scheduled job that queries your QA database, computes summary statistics, and posts alerts to Slack or your incident management system is sufficient. The value is in the data you have already collected, drift monitoring is a reporting layer on top of your per-segment quality records.

Budgeting for QA API Calls

Automated quality estimation is cheap relative to human review, but costs accumulate at scale. Budget for QA API calls by understanding the per-call cost of each model and the volume of segments per release.

The most cost-effective strategy is a cascade: run cheap checks first (placeholders, terminology, COMET-QE), and only escalate to expensive checks (LLM judges, human review) for segments that fail or score ambiguously. This tiered approach can reduce human review volume substantially while maintaining or improving overall quality coverage.

Track QA API spend per language pair and per release. If a particular language consistently requires more LLM judge calls or human review, that is a signal to invest in better glossaries, fine-tuned MT, or additional training data for that locale.

Frequently Asked Questions

What is the difference between COMET and BLEURT for translation quality scoring?

Both are learned metrics that predict human judgments but differ in architecture and training data: COMET supports both reference-based and referenceless QE modes, while BLEURT fine-tunes a pre-trained language model on human ratings. Ollang can ingest scores from either metric and use them in a composite scoring pipeline.

How do I handle segments where automated scores and human reviewers disagree?

Treat disagreements as calibration signals: log them, inspect the error types, and adjust thresholds or model selection accordingly. Ollang stores provenance and review outcomes to make that feedback loop actionable.

Can I use this pipeline with multiple MT engines simultaneously?

Yes, route segments to multiple engines, score each output, and select the best. Ollang records per-engine outputs and scores so you can track engine performance and implement best-of-N selection.

What MQM categories should I track at minimum?

Start with accuracy, fluency, terminology, and locale conventions, and add style categories for UI or marketing content. Use the same MQM set across automated checks and human annotations; Ollang's schema supports this alignment.

Build Your Audit-Ready QA Pipeline

An API-driven translation quality pipeline is not a luxury, it is the infrastructure that makes localization measurable, defensible, and scalable. By combining quality estimation models, terminology validators, LLM judges, and structured MQM storage with CI-integrated acceptance gates, you move from reactive firefighting to proactive quality control. Every segment gets scored, every decision gets logged, and every release gets a quality certificate grounded in data rather than intuition.

Ollang provides the API execution layer to connect these components, MT providers, TMS platforms, quality models, and your deployment pipeline, into a single, auditable workflow.

Book a Demo