Modern QA for Localization: MQM + LLM Quality Estimation

Localization quality is too often reduced to a gut feeling, a reviewer reads a translation, declares it "good enough," and the content ships. This approach collapses the moment you scale to dozens of languages, thousands of segments, and mixed content types spanning UI strings, legal documents, marketing copy, and dubbed video. Measurable quality requires a structured error typology, automated metrics that correlate with human judgment, and LLM-based quality estimation that can run continuously without bottlenecking your pipeline. This article lays out a hybrid QA framework that combines the Multidimensional Quality Metrics (MQM) standard with automated scoring and AI-driven estimation, giving you a deployable spec for gating releases, calibrating reviewers, and tracking quality drift across every content tier.

If your localization pipeline currently lacks this kind of structured QA layer, explore how Ollang can help you build one.

MQM Error Typology Explained

The Multidimensional Quality Metrics framework, developed through the QT21 research project and now maintained as part of the ASTM WK46396 standard effort, provides a hierarchical taxonomy of translation errors. Unlike older binary "correct/incorrect" models, MQM lets you categorize what went wrong and how badly, producing scores that are comparable across languages, reviewers, and time periods.

Core MQM Categories and Subcategories

MQM organizes errors into seven top-level dimensions, each with granular subcategories:

In practice, most teams start with Accuracy, Fluency, and Terminology as their primary dimensions and expand to others as their QA maturity grows. The key benefit of this taxonomy is that it makes quality decomposable: you can identify whether a language pair has a terminology problem versus a fluency problem, and route improvement efforts accordingly.

Severity Levels and Weighted Scoring

Each error instance receives a severity rating. The standard three-tier model is:

• Critical, Errors that could cause harm, legal liability, or complete misunderstanding. Examples include safety instructions translated with the opposite meaning, or offensive content introduced through mistranslation. Weight: typically 10 or 25 penalty points.
• Major, Errors that change meaning or significantly reduce usability but do not cause harm. A mistranslated feature name in a UI or a wrong number in a financial document. Weight: typically 5 or 10 points.
• Minor, Errors that are noticeable but do not impair comprehension. Stylistic awkwardness, a minor punctuation inconsistency, a slightly unnatural collocation. Weight: typically 1 point.

The composite MQM score for a sample is calculated as:

MQM Score = 1 − (Σ weighted error penalties / word count)

A perfect score is 1.0. Industry benchmarks from the research by Multidimensional Quality Metrics suggest that professional human translation typically scores between 0.95 and 0.98, while raw machine translation output without post-editing often falls below 0.90.

Sampling Plans and Inter-Annotator Agreement

Reviewing every segment is economically impractical at scale. A sound sampling plan balances statistical confidence with cost:

• Sample size: For large volumes, a random sample of 250-400 words per language pair per batch provides reasonable confidence. For high-risk content (medical, legal), increase to 800-1,200 words.
• Stratification: Sample proportionally across content types, MT engines, and translators to avoid blind spots.
• Frequency: Continuous sampling on rolling batches, not just milestone reviews.

Inter-annotator agreement (IAA) is the measure of consistency between reviewers. Without it, your MQM scores are unreliable. Track IAA using Cohen's kappa or Krippendorff's alpha, targeting κ ≥ 0.7 for production use. Scores below 0.6 indicate that your reviewers need recalibration, they are not applying the error typology consistently.

Automated Metrics: COMET, BLEURT, and chrF

Human review is the gold standard for quality measurement, but it is slow, expensive, and inherently limited in coverage. Automated metrics fill the gap by scoring every segment in near real-time.

Reference-Based Metrics

chrF (character n-gram F-score) is a lightweight metric that compares character-level overlap between the MT output and a reference translation. It is language-agnostic, easy to compute, and correlates reasonably well with human judgment for closely related language pairs. Its weakness is that it penalizes valid paraphrases.

BLEURT, developed by Google Research, is a learned metric that fine-tunes a pre-trained language model on human quality ratings. It captures semantic similarity beyond surface-level overlap, making it more robust to paraphrase than chrF. However, it requires a reference translation.

COMET (Crosslingual Optimized Metric for Evaluation of Translation), implemented in platforms such as Ollang and developed by Unbabel, has emerged as the metric most strongly correlated with human MQM judgments. COMET models are trained on direct assessment and MQM-annotated data, and they take source, MT output, and reference as inputs. The reference-free variant, COMET-QE, uses only source and MT output, making it suitable for scenarios where no reference exists.

When to Use Which Metric

For production localization, COMET or COMET-QE should be the primary metric. Use chrF as a fast sanity check and BLEURT for periodic deep evaluations. Ollang integrates COMET and COMET-QE into CI pipelines so teams can run continuous production scoring and gate releases automatically.

LLM-Based Quality Estimation

Large language models have introduced a new paradigm for translation quality assessment: prompt-based quality estimation that can explain its judgments in natural language.

How LLM QE Works

An LLM-based quality estimation system takes the source segment and the translation (and optionally a reference) and produces both a quality score and a structured error annotation. Unlike COMET, which outputs a single scalar, an LLM can identify specific error spans, classify them using MQM categories, and provide rationale, essentially performing automated MQM annotation.

Research from Google's GEMBA framework demonstrated that GPT-4-level models achieve segment-level quality estimation that correlates with human MQM scores at levels approaching trained human annotators. This does not mean LLMs replace human review, but it means they can dramatically expand coverage: you can run LLM QE on 100% of segments and reserve human review for flagged content. Ollang runs LLM QE at scale and routes flagged segments automatically to human reviewers for efficient escalation.

LLM QE vs. Trained Metrics

• Strengths of LLM QE: Explainability (natural-language rationale), zero-shot adaptability to new domains, ability to check terminology against provided glossaries, and detection of culturally inappropriate content.
• Weaknesses of LLM QE: Higher latency and cost per segment, potential for hallucinated error annotations, sensitivity to prompt design, and inconsistency across runs without temperature control.
• Strengths of trained metrics (COMET/BLEURT): Fast inference, deterministic output, well-studied correlation with human judgment, low cost at scale.

The optimal approach is layered: run COMET-QE on all segments for fast scoring, then route segments below a confidence threshold to LLM QE for detailed annotation, and escalate the worst cases to human review.

Terminology Adherence Checks

Terminology consistency is a frequent pain point in enterprise localization. An LLM QE system can be augmented with a term base and instructed to verify that every glossary term in the source is rendered with the approved target-language equivalent. This goes beyond simple string matching, the LLM understands morphological variants, can handle terms that appear in inflected forms, and can distinguish between a term used in its domain-specific sense versus its general meaning.

Placeholder and ICU Validation

Functional correctness checks are non-negotiable for software localization. These include:

• Placeholder integrity: Variables like {username}, %d, or {{count}} must pass through translation unchanged. A missing or altered placeholder causes runtime errors.
• ICU MessageFormat validation: Plural rules, select statements, and nested structures in ICU MessageFormat strings must be syntactically valid in the target. A mismatched brace or a missing plural category (e.g., omitting the other case) breaks the application.
• HTML/XML tag preservation: Tags must remain balanced, properly nested, and in the correct position relative to the translated text.

These checks are best handled by deterministic rule-based validators, not LLMs. LLMs can miss a single mismatched brace. Run placeholder and ICU validation as a hard gate before any other quality assessment.

Detecting Hallucinations and Bias

LLM-generated or LLM-post-edited translations can introduce hallucinated content, fluent text that has no basis in the source. Common hallucination patterns include:

• Fabricated details added to the translation that do not appear in the source.
• Repetitive loops where the same phrase is generated multiple times.
• Complete topic drift where the translation addresses a different subject.

Detection strategies include cross-referencing the translation against the source using entailment models, flagging segments where the translation is significantly longer than expected based on the source length, and using COMET-QE scores as a proxy (hallucinated content typically scores low on source-translation alignment).

Bias detection focuses on identifying cases where the translation introduces gendered language where the source is neutral, uses culturally insensitive phrasing, or reflects stereotypes. This is particularly important for marketing and user-facing content. LLM QE prompts can be specifically designed to flag these patterns.

Quality Assurance for Video, Audio, and Subtitles

Multimedia localization introduces quality dimensions that do not exist in text-only workflows. A perfectly translated subtitle that violates reading speed limits or falls out of sync with the audio is a failed localization.

Subtitle Reading Speed and Line Breaks

Subtitle readability is governed by characters per second (CPS) and lines per subtitle. The widely adopted guidelines from the Netflix Timed Text Style Guide recommend:

• Maximum reading speed: 17 characters per second for adult content, 13 CPS for children's content.
• Maximum lines per subtitle: Two.
• Maximum characters per line: 42 for most Latin-script languages; adjusted for CJK and other scripts.
• Line break logic: Break at natural syntactic boundaries (between clauses, after conjunctions). Never split a proper noun, a determiner from its noun, or a preposition from its object.

Automated QA tools should flag any subtitle event exceeding CPS limits and any line break that splits a syntactic unit.

Timecode Integrity

Subtitle timecodes must align with the spoken audio. Key checks include:

• Minimum display duration: Subtitles should remain on screen for at least 833 milliseconds (20 frames at 24fps).
• Gap between subtitles: A minimum gap of 2 frames prevents visual flicker.
• Shot change alignment: Subtitles should not span a hard cut unless unavoidable. When they do, the reading speed must be recalculated for each shot.
• Sync tolerance: Subtitle in-cues should fall within 250ms of the corresponding speech onset.

Loudness and Audio Quality

For dubbed audio and voice-over, loudness normalization is critical. The EBU R128 standard specifies a target loudness of −23 LUFS (±1 LU) for broadcast content. Streaming platforms often target −14 LUFS. QA checks should verify:

• Integrated loudness falls within the target range.
• True peak levels do not exceed −1 dBTP.
• The dubbed audio maintains consistent loudness relative to the original mix, especially for music and effects tracks that are retained.
• Lip-sync offset for dubbed content stays within the perceptual threshold (generally under 80ms for most viewers).

If your localization pipeline spans text, video, and audio content types, see how Ollang unifies QA across all of them.

Building Gold Sets and Calibrating Reviewers

A QA framework is only as reliable as the people and data that underpin it. Gold sets and reviewer calibration are the mechanisms that ensure consistency.

Creating Gold Sets

A gold set is a curated collection of source-translation pairs with pre-annotated errors that serve as the ground truth for calibration and testing. To build effective gold sets:

1. Select representative segments across content types, error categories, and severity levels. Include segments with no errors, segments with a single clear error, and segments with multiple overlapping issues.
2. Annotate with multiple reviewers, at least three independent annotators per segment, and resolve disagreements through adjudication. The adjudicated annotation becomes the gold standard.
3. Balance error distribution so that the gold set is not dominated by any single error type. Include examples of accuracy, fluency, terminology, and locale convention errors.
4. Version and refresh gold sets periodically. As your content evolves and MT engines improve, gold sets that are more than 12 months old may no longer reflect current error patterns.

Gold sets serve double duty: they calibrate human reviewers and validate automated metrics. If your COMET model does not flag errors that are present in the gold set, you know the model needs retraining or threshold adjustment. Ollang version-controls gold sets and automates their periodic insertion into production batches for continuous calibration.

Reviewer Calibration Workflow

Calibration ensures that all reviewers apply the MQM typology consistently. A practical calibration workflow:

1. Initial training: New reviewers complete a guided session covering MQM categories, severity definitions, and scoring methodology, with worked examples.
2. Blind gold-set test: Reviewers annotate a subset of the gold set without seeing the gold annotations. Their results are compared against the gold standard.
3. Feedback and discussion: Discrepancies are reviewed in a calibration meeting. Common disagreements, such as whether a stylistic choice is a minor fluency error or not an error at all, are resolved and documented as calibration notes.
4. Ongoing monitoring: Periodically insert gold-set segments into production review batches. Track each reviewer's agreement with the gold standard over time. Reviewers whose kappa drops below 0.65 are flagged for recalibration.

CI Gating: Thresholds, Regression Alerts, and Release Gates

Quality measurement only delivers value if it is wired into your release process. Continuous integration (CI) gating ensures that translations meeting quality standards ship, and those that do not are caught before they reach users.

Defining Quality Thresholds

Set thresholds based on content tier and risk level:

These thresholds are starting points. Calibrate them against your own gold sets and adjust based on observed correlation between metric scores and downstream user feedback.

Regression Alerts

Quality drift is insidious, a model update, a glossary change, or a new translator can degrade quality gradually. Regression detection catches this:

• Baseline establishment: After a successful release, record the mean and standard deviation of COMET-QE scores per language pair and content type.
• Statistical monitoring: On each new batch, compare the score distribution against the baseline using a two-sample t-test or Mann-Whitney U test. Flag any language pair where the mean drops by more than one standard deviation.
• Trend tracking: Plot rolling averages over weeks and months. A sustained downward trend, even if individual batches pass the threshold, warrants investigation.

Release Gate Workflow

A practical CI gate for localization:

1. Placeholder/ICU validation, hard gate. Any failure blocks the release.
2. COMET-QE scoring, soft gate. Segments below the threshold are routed to LLM QE.
3. LLM QE annotation, segments with critical or major errors flagged are routed to human review.
4. Human review, reviewer confirms or overrides automated annotations. Confirmed critical errors block the release.
5. Aggregate scoring, if the batch-level MQM score (from human + automated annotations) meets the tier threshold, the release proceeds. Otherwise, the batch is returned for correction.

If you want a quick, expert check on your thresholds and routing logic, get a CI gate review from the Ollang team.

Cost-Quality Tradeoffs by Content Tier

Not all content deserves the same QA investment. The tiered model above reflects a fundamental principle: the cost of a quality failure should determine the cost of quality assurance.

For Tier 1 content, the cost of an error, regulatory fines, patient harm, legal liability, dwarfs the cost of exhaustive human review. Full MQM annotation with dual review and adjudication is justified.

For Tier 2 content, the cost of an error is reputational and functional. A hybrid approach, automated scoring with sampled human review, provides strong coverage at a fraction of the Tier 1 cost.

For Tier 3 and Tier 4 content, automated QA alone is often sufficient. The key is to ensure that automated systems are well-calibrated (validated against gold sets) so that their scores are trustworthy proxies for human judgment.

The cost savings from this tiered approach are substantial. Organizations that apply Tier 1 QA processes uniformly across all content types typically spend multiple times more on quality assurance than those with a differentiated model, without meaningfully better outcomes for lower-tier content.

Tracking Quality Drift in Production

Quality is not a one-time measurement. Production systems degrade, and without ongoing monitoring, you will not know until users complain.

Signals to Monitor

• COMET-QE score distributions per language pair, per content type, per MT engine, tracked weekly.
• Error category breakdowns from LLM QE and human review. A spike in terminology errors may indicate a glossary that has not been updated. A spike in accuracy errors after an MT model update signals a regression.
• User-reported issues correlated with QA scores. If segments that passed QA are generating support tickets, your thresholds are too lenient.
• Reviewer agreement trends, declining IAA signals calibration drift among your review team.

Dashboards and Scorecards

Build a QA scorecard that surfaces:

• Language-pair health: A green/yellow/red status for each language pair based on the most recent batch scores relative to thresholds.
• Error hotspots: The top three error categories by volume and severity for each language pair.
• Trend lines: Rolling 30-day and 90-day COMET-QE averages with regression alerts highlighted.
• Reviewer performance: Per-reviewer agreement with gold standards and annotation throughput.

This scorecard becomes the artifact that localization managers, engineering leads, and product owners use to make informed decisions about where to invest in quality improvement.

FAQ

What is the difference between MQM and COMET?

MQM is an error typology, a framework for categorizing and scoring translation errors through human annotation. COMET is a trained neural metric that predicts quality scores from source-translation pairs. They are complementary: MQM provides the ground truth that COMET models are trained on, and COMET provides scalable estimation that approximates MQM scoring without requiring human annotators for every segment.

Can LLM-based quality estimation replace human reviewers?

Not entirely, but it can dramatically reduce the volume of content that requires human attention. LLM QE is most effective as a triage layer: it scores and annotates all segments, and only those with potential critical or major errors are escalated to human review. For Tier 3 and Tier 4 content, LLM QE may be sufficient as the sole quality gate. For Tier 1 content, human review remains essential. Platforms like Ollang use LLM QE as an automated triage layer to reduce human review load while keeping high-risk content under human control.

How do I set the right quality thresholds for my content?

Start with industry benchmarks (e.g., MQM ≥ 0.95 for professional-quality translation), then calibrate against your own gold sets and user feedback. Run a correlation analysis: do segments that score above your threshold actually perform well with users? If users report issues with content that passed QA, lower the threshold. If content that fails QA is consistently acceptable, raise it. Thresholds should be reviewed quarterly.

How does video/audio QA differ from text QA?

Video and audio localization add temporal and perceptual dimensions. Beyond linguistic accuracy and fluency, you must validate subtitle reading speeds, timecode synchronization, line break logic, dubbed audio loudness, and lip-sync alignment. These checks require specialized tools and standards (Netflix Timed Text guidelines, EBU R128) that do not apply to text-only workflows.

Start Building Your QA Framework

A measurable, tiered, and automated QA framework is not a luxury, it is the infrastructure that separates localization programs that scale from those that break under their own weight. Whether you are starting with MQM annotation for a single language pair or deploying LLM-based quality estimation across dozens of markets, the principles are the same: define your error typology, calibrate your reviewers and metrics, gate your releases, and monitor for drift.

Ollang provides the execution layer to operationalize this entire framework, from automated metric scoring and LLM QE to human review orchestration and CI gating across text, video, audio, and software content types.

Book a Demo