AI Translation Quality Deep Dive: MQM, LLM-TQA, Risk and Metrics

Most enterprise localization teams know their AI translation output is "pretty good", but few can quantify how good, where it fails, or what to do when it breaks. Without a structured quality framework, translation defects surface unpredictably: a mistranslated drug interaction in a patient leaflet, a tone-deaf marketing tagline, or a legal clause that reverses liability. The gap between "pretty good" and production-ready is where measurable quality assurance lives. This article walks through the frameworks, metrics, and workflows needed to build an auditable AI translation QA program, from the MQM error taxonomy and automatic metrics like COMET and BLEURT, to LLM-assisted quality assessment, risk-based sampling, and root-cause feedback loops that actually improve output over time.

Understanding MQM and DQF: The Foundation of Translation Quality

What MQM and DQF measure

The Multidimensional Quality Metrics (MQM) framework, developed through the QT21 project, provides a hierarchical error typology for evaluating translation quality. Rather than relying on subjective impressions, MQM categorizes every defect by type (accuracy, fluency, terminology, style, locale convention, verity, design) and assigns a severity level: critical, major, minor, or neutral.

DQF (Dynamic Quality Framework), originally developed by TAUS, takes a complementary approach. Where MQM focuses on granular error annotation, DQF provides standardized productivity and quality benchmarks that allow organizations to compare performance across vendors, engines, and language pairs. The two frameworks have been harmonized into what is often referred to as the MQM-DQF integrated model, giving teams a single taxonomy that supports both fine-grained error analysis and high-level quality scoring.

The practical value is straightforward: MQM/DQF replaces "this translation feels off" with "this segment contains a major accuracy error (mistranslation) and a minor fluency error (grammar)." That precision is what makes quality auditable.

Setting criticality tiers by content type

Not all content carries equal risk. A mistranslation in a regulatory submission has different consequences than one in a knowledge base article. Criticality tiers map content types to quality thresholds, ensuring that review effort and acceptance criteria are proportional to business impact.

These thresholds are starting points. Organizations should calibrate them against their own regulatory environment, brand sensitivity, and historical defect data.

Calibrating evaluators for inter-annotator agreement

An MQM taxonomy is only as reliable as the people, or models, applying it. If two reviewers look at the same segment and one marks a major accuracy error while the other marks a minor fluency issue, your quality data is noise.

Calibration requires three things:

- Shared annotation guidelines that define each error category with examples specific to your domain and language pairs. Generic definitions invite inconsistency.

- Calibration sessions where evaluators annotate the same sample set independently, then compare results. Disagreements are discussed and resolved, and guidelines are updated accordingly.

- Inter-annotator agreement (IAA) measurement using metrics like Cohen's kappa or Krippendorff's alpha. An alpha of 0.67 or higher is generally considered acceptable for translation quality annotation, though higher-stakes content should target 0.80 or above.

For teams using LLM-assisted evaluation (covered below), calibration also means validating that the model's annotations align with human expert judgments on a representative sample before trusting it at scale.

Automatic Metrics: Where They Work and Where They Don't

BLEU, COMET, BLEURT, and BERTScore compared

Automatic metrics offer speed and scale that human evaluation cannot match, but each metric captures different aspects of quality, and each has blind spots.

• BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between a machine translation and a reference translation. It remains widely used as a development-time diagnostic, but its correlation with human quality judgments is weak for individual segments. BLEU cannot detect meaning-preserving paraphrases and penalizes valid translations that diverge lexically from the reference.
• BERTScore improves on surface-level matching by computing cosine similarity between contextual embeddings of tokens in the candidate and reference translations. It handles synonyms and paraphrases better than BLEU, but still depends on a reference translation and can miss factual errors that are fluently expressed.
• BLEURT is a learned metric that fine-tunes BERT on human quality ratings. It has shown substantially higher correlation with human judgments than BLEU, particularly for high-quality MT output where differences are subtle.
• COMET (Crosslingual Optimized Metric for Evaluation of Translation), developed by Unbabel, has become one of the leading reference-based metrics in recent WMT Metrics shared tasks. COMET models are trained on direct assessment scores and can incorporate source text, reference, and candidate translation. Its segment-level correlation with human judgments consistently outperforms BLEU and BERTScore.

Reference-free quality estimation and when to trust it

Reference-free quality estimation (QE) scores translations without needing a human reference, a significant advantage when reference translations don't exist, which is most of the time in production localization.

Modern QE models, including COMET's reference-free variants (CometKiwi) and approaches built on multilingual language models, predict quality scores using only the source text and the MT output. The WMT Quality Estimation shared task has driven steady improvements, and top QE systems now approach the segment-level correlation of reference-based metrics.

When to trust QE:

- High confidence: Flagging catastrophic errors, detecting untranslated segments, identifying hallucinations where output diverges entirely from source meaning.

- Moderate confidence: Ranking alternative translations, triaging segments for human review, monitoring quality trends across batches.

- Low confidence: Making fine-grained distinctions between "good" and "very good" translations, evaluating stylistic nuance, or assessing domain-specific terminology accuracy without domain-tuned models.

QE works best as a filter and triage mechanism, not as a replacement for human judgment on high-stakes content.

LLM-Assisted Translation Quality Assessment (LLM-TQA)

Rubric-guided error annotation with LLMs

Large language models can perform structured translation quality annotation when given explicit rubrics, a pattern that has gained traction as models like GPT-4 and Claude demonstrate strong multilingual reasoning capabilities.

The approach works as follows: the LLM receives the source segment, the translation, and a detailed rubric that maps directly to your MQM taxonomy. The prompt instructs the model to identify errors, classify each by type and severity, and provide a brief justification. This is fundamentally different from asking an LLM "is this translation good?", the rubric constrains the evaluation to specific, auditable criteria.

Effective rubric-guided annotation prompts include:

- The MQM error categories relevant to the content tier

- Severity definitions with concrete examples

- Domain-specific terminology lists or glossary excerpts

- Instructions to output structured annotations (e.g., JSON with error spans, types, severities)

Studies have shown that LLM-based evaluation can achieve moderate to high agreement with expert human annotators, particularly for accuracy errors. Fluency and style judgments remain less reliable, especially for languages with limited representation in training data.

Hallucination detection, bias/toxicity filters, and glossary conformance

Beyond general quality annotation, LLM-TQA enables targeted checks that are difficult to scale with human reviewers alone.

• Hallucination detection identifies cases where the translation introduces information not present in the source or omits critical source content. This is particularly dangerous in AI translation because neural models can generate fluent, plausible text that is factually disconnected from the input. LLM-based checks can cross-reference source and target to flag semantic divergence, while QE models can catch extreme cases through low confidence scores.
• Bias and toxicity filtering screens translations for language that is offensive, discriminatory, or culturally inappropriate in the target locale. This goes beyond simple keyword blocklists, LLMs can evaluate contextual appropriateness, catching cases where a word is acceptable in one context but harmful in another.
• Glossary conformance checking verifies that approved terminology is used consistently. The LLM receives the relevant glossary entries alongside the source and translation, then flags any deviations, whether substitutions, inconsistencies, or unapproved variants. This is especially valuable for regulated industries where terminology precision is non-negotiable.

These checks can run in parallel as part of an automated QA pipeline, with results aggregated into a quality dashboard. For teams evaluating how to integrate these capabilities into production workflows, consider booking a demo to see Ollang’s LLM-TQA in action.

Risk-Based Sampling, Acceptance Thresholds, and Rollback

Designing a risk-based sampling strategy

Reviewing every translated segment is neither practical nor necessary. Risk-based sampling concentrates review effort where it matters most, using content criticality, language pair difficulty, and historical defect rates to determine sample sizes.

A practical sampling framework:

- Tier 1 content: 100% review. No sampling, every segment is evaluated.

- Tier 2 content: Stratified sampling at 15-25% of segments, weighted toward segments flagged by automatic QE or LLM-TQA as potentially problematic.

- Tier 3 content: Random sampling at 5-10%, supplemented by automatic QA checks on the full volume.

- Tier 4 content: Automated checks only, with human review triggered only when defect density exceeds a defined threshold.

Language pairs with historically higher error rates or lower MT engine performance should receive larger sample sizes regardless of content tier. Similarly, new terminology domains, recently updated engines, or first-time language pairs warrant increased sampling until baseline quality is established.

Ollang's platform supports programmatic configuration of stratified sampling rules and the ingestion of QE/LLM flags to prioritize review queues automatically.

Defining acceptance thresholds and rollback criteria

Acceptance thresholds translate MQM defect counts into pass/fail decisions. A batch passes when its defect density (errors per 1,000 words, weighted by severity) falls below the threshold defined for its content tier.

A common severity weighting scheme:

- Critical error: 10 points

- Major error: 5 points

- Minor error: 1 point

A Tier 2 batch might have an acceptance threshold of 15 weighted error points per 1,000 words. A batch scoring 18 fails and triggers corrective action.

Rollback criteria define when a failed batch requires more than rework:

- Rework: Defect density exceeds the threshold but is within 150% of the limit. Failed segments are corrected and re-evaluated.

- Rollback: Defect density exceeds 150% of the threshold, or any critical errors are found. The batch is rejected, root cause is investigated, and the translation is regenerated after the underlying issue is resolved.

- Engine suspension: Repeated rollbacks for the same language pair or content type trigger suspension of the MT engine or prompt configuration pending investigation.

Root-Cause Analysis and Feedback Loops

Common root causes: segmentation, context loss, and terminology drift

When quality degrades, the cause is rarely "the AI is bad." Specific, diagnosable patterns account for the majority of recurring defects.

• Segmentation errors occur when the source text is split at inappropriate boundaries, mid-sentence, mid-clause, or in ways that separate a term from its modifier. MT engines translate each segment independently, so poor segmentation strips away the grammatical and semantic context needed for accurate translation. This is especially problematic with sentence-splitting rules that don't account for abbreviations, numbered lists, or embedded quotations.
• Context loss extends beyond segmentation. When an MT engine processes segments without access to surrounding sentences, document-level coherence suffers. Pronoun resolution fails, register shifts mid-document, and discourse connectors become inconsistent. This is a structural limitation of segment-level translation that requires document-level or paragraph-level processing to address.
• Terminology drift happens when approved terms are gradually replaced by synonyms or variants, either because the MT engine's underlying model shifts during updates, or because translation memories and glossaries fall out of sync with current usage. Drift is insidious because each individual deviation may seem minor, but cumulative inconsistency erodes brand voice and can create confusion in technical or regulated content.

Building feedback loops that improve prompts, models, and TMs

Quality data is wasted if it doesn't flow back into the systems that produce translations. Effective feedback loops close the gap between error detection and error prevention.

• Prompt refinement: When LLM-TQA or human review identifies recurring error patterns, those patterns should inform prompt engineering. If a specific language pair consistently produces overly literal translations of idiomatic expressions, the prompt can be updated with explicit instructions and examples for handling idioms. Track which prompt versions correlate with which quality scores.
• Model fine-tuning and selection: Aggregate MQM data by error type and language pair to identify whether quality issues are best addressed by switching engines, fine-tuning existing models on corrected output, or adjusting decoding parameters. Not every problem requires retraining, sometimes a different model is simply better for a given domain.
• Translation memory updates: Corrected segments should feed back into TMs systematically, not ad hoc. Establish a workflow where post-edited segments are reviewed for TM suitability, tagged with domain and quality metadata, and ingested into the TM with appropriate match penalties for AI-generated versus human-verified entries.
• Glossary governance: Terminology drift is prevented by regular glossary audits triggered by conformance check data. When glossary violations spike for specific terms, investigate whether the glossary needs updating (the term has legitimately evolved) or enforcement needs tightening (the engine is ignoring valid terms).

Platforms like Ollang make it practical to automate these loops at scale by routing MQM annotations and post-edits into prompt versions, model selection workflows, and TMs so corrections reduce repeat defects rather than just documenting them.

Starter MQM Taxonomy and Quality Dashboards

A practical MQM taxonomy to begin with

The full MQM taxonomy contains dozens of error categories. Starting with the complete hierarchy is overwhelming and counterproductive. The following starter taxonomy covers the error types that account for the vast majority of defects in AI-translated content:

Accuracy

- Mistranslation: The target conveys a different meaning than the source.

- Omission: Source content is missing from the translation.

- Addition: The translation includes information not present in the source.

- Untranslated: Source text is left in the source language.

Fluency

- Grammar: Morphological, syntactic, or agreement errors.

- Spelling/Punctuation: Orthographic errors or incorrect punctuation.

- Register: Inappropriate formality level for the content type and audience.

- Awkward phrasing: Grammatically correct but unnatural expression.

Terminology

- Wrong term: An incorrect term is used where an approved equivalent exists.

- Inconsistent term: The same source term is translated differently within the same document.

Locale Convention

- Number format: Incorrect decimal separators, digit grouping, or currency symbols.

- Date/Time format: Dates or times not conforming to target locale conventions.

- Measurement: Units not converted or incorrectly converted for the target market.

Style

- Guideline violation: Translation does not follow documented style guide rules (sentence length, tone, voice).

Each error is annotated with a severity: critical (renders content unusable or dangerous), major (changes meaning or significantly impairs usability), minor (noticeable but does not impair understanding), or neutral (preferential, not a true error).

Dashboards for defect density and severity

Quality dashboards transform raw annotation data into actionable intelligence. At minimum, a production localization QA dashboard should track:

- Defect density over time (weighted error points per 1,000 words), broken down by language pair, content tier, and MT engine. Trend lines reveal whether quality is improving, stable, or degrading.

- Error type distribution showing which MQM categories account for the highest share of defects. If terminology errors dominate, the glossary process needs attention. If accuracy errors spike, the engine or prompt may need revision.

- Severity breakdown distinguishing critical and major errors from minor ones. A high volume of minor errors with zero criticals tells a very different story than a moderate volume with occasional criticals.

- Pass/fail rates by batch tracking what percentage of batches meet acceptance thresholds on the first submission. Low first-pass yield indicates systemic issues upstream.

- Evaluator agreement scores monitoring IAA over time to ensure annotation consistency doesn't drift.

Ollang's localization platform supports this kind of structured quality monitoring across text, software, and document localization workflows, connecting QA data directly to the translation pipeline so that feedback loops operate continuously rather than in retrospective audits.

Frequently Asked Questions

How does MQM differ from simple quality ratings like "good" or "bad"?

MQM replaces subjective impressions with structured, countable error annotations: reviewers identify specific errors and classify each by type and severity. This produces reproducible defect density scores that directly inform what to fix.

Can COMET or BLEURT replace human translation review?

They can't fully replace human review but can dramatically reduce the volume of content requiring human attention by triaging likely-problematic segments. For Tier 1 content human review remains essential, while automatic metrics plus LLM-TQA can cover most Tier 3 and Tier 4 content.

What is the biggest risk of relying solely on LLM-based quality assessment?

LLMs can be confidently wrong and tend to perform unevenly across languages, so validate LLM-TQA outputs against human experts on representative samples and revalidate periodically as models change.

How often should we recalibrate our quality thresholds?

Recalibrate thresholds quarterly or after significant changes (new MT engines, added language pairs, or major shifts in content). Use dashboard signals like first-pass yield and defect trends to decide whether to adjust thresholds or investigate underlying quality issues.

Build Your AI Translation QA Program with Ollang

Designing a measurable, auditable quality assurance program for AI translation requires more than good intentions, it requires the right taxonomy, the right metrics, and a platform that connects quality data to production workflows. The frameworks in this article give you the structure. Executing them at enterprise scale across dozens of language pairs, content types, and MT engines requires tooling purpose-built for the task.

Ollang provides the AI execution layer for enterprise localization, integrating MQM-based quality assessment, automatic metrics, LLM-assisted TQA, and feedback loops into a single platform that covers text, video, audio, software, website, and legal document localization. If you're ready to move from ad hoc quality checks to a systematic QA program, book a personalized Ollang demo to see how it works in practice.