Translation Quality in AI: MQM, LLM Evaluation and Human QA

Measuring translation quality is one of the hardest problems in localization, and one of the most consequential. A mistranslated legal clause can void a contract. A poorly timed subtitle can confuse millions of viewers. A UI string stripped of context can break an entire user flow. Yet many teams still rely on gut feel, sporadic spot checks, or a single BLEU score to judge whether their AI-generated translations are good enough. This article breaks down the frameworks, metrics, and operational workflows you need to build a rigorous, scalable quality program, from MQM error taxonomies and automated evaluation metrics to LLM-as-a-judge approaches and staged human review. Whether you're localizing software, video, legal documents, or marketing content, the goal is the same: measurable quality with defensible thresholds.

If you're looking to operationalize translation quality across content types and languages, explore how Ollang's quality review capabilities can help.

What Is Translation Quality and Why Does It Matter?

Translation quality is not a single number, it is a multidimensional judgment about whether a translation serves its purpose for its intended audience. A "perfect" translation of a marketing tagline might be creative and culturally resonant but take liberties with literal meaning. A "perfect" translation of a pharmaceutical label must be terminologically exact, even if it reads stiffly. Quality, in other words, is always relative to function.

This matters because poor quality has concrete costs. Mistranslated medical device instructions have triggered FDA recalls. Inaccurate financial disclosures can violate regulatory requirements in dozens of jurisdictions simultaneously. Even in lower-stakes contexts like e-commerce product descriptions, translation errors erode trust and conversion rates.

For AI-generated translations, whether from neural machine translation engines or large language models, the quality question is especially urgent. These systems produce fluent output that can mask serious accuracy problems. A sentence can read beautifully in the target language while omitting a critical negation or substituting the wrong technical term. Fluency without fidelity is arguably more dangerous than an obviously broken translation, because it passes casual review undetected.

Defining Quality Beyond Fluency

Fluency, whether a translation reads naturally in the target language, is necessary but far from sufficient. A complete quality framework must also evaluate:

• Accuracy: Does the translation faithfully convey the source meaning without additions, omissions, or distortions?
• Terminology: Are domain-specific terms translated consistently and correctly?
• Style and register: Does the translation match the tone, formality, and voice guidelines for the brand or content type?
• Locale conventions: Are dates, currencies, units, and cultural references adapted appropriately?
• Functional adequacy: Does the translation work in its target context, fitting a UI element's character limit, syncing with audio timing, or meeting legal precision requirements?

Each of these dimensions can be measured, and each matters differently depending on the content type and its risk profile.

The Cost of Getting Quality Wrong

The financial impact of translation errors extends well beyond rework costs. In regulated industries, a single mistranslation can trigger compliance violations, product recalls, or litigation. GALA (the Globalization and Localization Association) has documented cases where localization failures led to multi-million-dollar consequences in medical, legal, and automotive sectors.

Even outside regulated domains, quality failures compound. Inconsistent terminology across a software product creates support tickets. Poorly localized marketing underperforms in-market, wasting campaign spend. And once a translation memory is polluted with errors, those mistakes propagate across future projects unless actively caught and corrected.

The takeaway is straightforward: investing in quality measurement and enforcement is not a luxury, it is risk management.

MQM and Error Taxonomies Explained

What Is the Multidimensional Quality Metrics Framework?

The Multidimensional Quality Metrics (MQM) framework, developed through the QT21 project and maintained by DFKI and its partners, is the most widely adopted standard for structured translation quality evaluation. Rather than assigning a single holistic score, MQM categorizes errors by type and severity, producing a granular quality profile that tells you not just how much is wrong but what kind of things are wrong.

The MQM taxonomy is hierarchical. At the top level, errors fall into broad categories:

Each category branches into more specific error types. An accuracy error, for instance, might be classified as a mistranslation (wrong meaning), an omission (missing content), or an addition (content not in the source). Each error also receives a severity rating, typically Critical, Major, or Minor, which determines its weight in the final quality score.

Platforms like Ollang can host MQM annotation workflows and track inter-annotator agreement so your scores reflect true quality differences rather than reviewer variation.

How MQM Scoring Works in Practice

MQM scoring converts error annotations into a quantitative metric. The standard approach assigns penalty points by severity: a common weighting is 25 points for a Critical error, 5 for a Major, and 1 for a Minor, though organizations can calibrate these weights to their own risk tolerance. The penalty points are summed and normalized by word count to produce a score per thousand words.

A typical quality threshold might look like this:

• Publishable quality: fewer than 5 penalty points per 1,000 words
• Acceptable with minor edits: 5-15 penalty points per 1,000 words
• Requires significant rework: more than 15 penalty points per 1,000 words

The power of MQM is that it makes quality actionable. If your MQM dashboard shows a spike in terminology errors for German legal content, you know exactly where to intervene, update the glossary, retrain the model, or add a terminology QA gate. If accuracy errors cluster in a specific language pair, you can investigate whether the MT engine underperforms for that pair or whether source content is ambiguous.

Building and Maintaining Error Annotation Guidelines

MQM is only as good as the consistency of its application. Two reviewers annotating the same translation should reach substantially similar conclusions. Achieving this requires:

• Detailed annotation guidelines that define each error type with examples drawn from your actual content
• Calibration sessions where reviewers annotate the same sample and discuss disagreements
• Inter-annotator agreement measurement (commonly using Cohen's kappa or Krippendorff's alpha) to quantify consistency
• Regular guideline updates as new error patterns emerge or content types change

Organizations that skip this calibration step often find that their MQM scores are noisy and unreliable, reflecting reviewer variation more than actual quality differences between translations.

Automated Metrics: COMET, BLEURT, and Beyond

Reference-Based Metrics: What BLEU Gets Wrong

For decades, BLEU (Bilingual Evaluation Understudy) was the default automated metric for machine translation quality. BLEU works by counting n-gram overlaps between a machine translation and one or more human reference translations. It's fast, cheap, and reproducible, and it correlates poorly with human quality judgments, especially at the segment level.

The fundamental problem is that BLEU treats translation as a word-matching exercise. A translation that uses a valid synonym scores lower than one that happens to match the reference's exact phrasing. BLEU also cannot detect meaning-altering errors like negation flips or entity substitutions if the surrounding n-grams still overlap. Research from the WMT (Workshop on Machine Translation) shared tasks has repeatedly shown that BLEU rankings of MT systems diverge significantly from human rankings.

COMET and BLEURT: Neural Evaluation Metrics

COMET (Crosslingual Optimized Metric for Evaluation of Translation) and BLEURT represent a newer generation of learned metrics that use neural language models to predict human quality judgments. Instead of counting surface-level matches, these metrics encode the source, the translation, and optionally a reference into dense vector representations and score them based on patterns learned from thousands of human quality annotations.

COMET, developed by Unbabel, has consistently achieved the highest correlation with human judgments in WMT metrics shared tasks. It comes in reference-based variants (comparing against a human translation) and reference-free variants (evaluating the translation against only the source), making it usable even when reference translations are unavailable.

BLEURT, developed by Google Research, similarly fine-tunes a pre-trained language model on human ratings and captures semantic similarity beyond lexical overlap.

Key advantages of these neural metrics:

• They detect meaning-level errors that surface metrics miss
• They handle paraphrases and valid translation variants gracefully
• Reference-free variants enable evaluation at scale without expensive reference creation
• They can be fine-tuned on domain-specific human judgments to improve accuracy for specialized content

Ollang integrates neural metrics such as COMET and BLEURT so teams can automate scoring in production and use configurable thresholds by content type.

When Automated Metrics Are Enough, and When They're Not

Automated metrics are excellent for relative comparisons: ranking MT engines, tracking quality trends over time, detecting regressions after model updates, and running A/B evaluations across system configurations. They are less reliable for absolute quality certification, determining whether a specific translation is "good enough" to publish.

A practical framework:

The most effective quality programs use automated metrics as a first filter and route uncertain or high-risk content to human review, a pattern that scales far better than reviewing everything manually.

LLM-as-a-Judge: Scalable Evaluation with Large Language Models

How LLM-Based Evaluation Works

A rapidly maturing approach uses large language models themselves as translation quality evaluators. In this paradigm, an LLM receives the source text, the translation, and a scoring rubric, then produces a quality judgment, either a numerical score, an error annotation, or both.

This approach gained significant traction following research showing that GPT-4-class models can produce quality assessments that correlate with expert human judgments at levels comparable to or exceeding traditional automated metrics. The GEMBA (GPT Estimation Metric Based Assessment) framework demonstrated that prompting LLMs with MQM-style rubrics yields segment-level scores with strong correlation to human MQM annotations.

LLM-as-a-judge evaluation can operate in several modes:

• Direct assessment: The LLM rates translation quality on a numerical scale given a rubric
• Error annotation: The LLM identifies and categorizes specific errors following an MQM-like taxonomy
• Pairwise ranking: The LLM compares two translations of the same source and selects the better one
• Explanation generation: The LLM provides natural-language rationale for its quality judgment, useful for reviewer training and disagreement resolution

Strengths, Limitations, and Bias Risks

LLM-based evaluation offers compelling advantages: it scales like an automated metric but captures nuance closer to human judgment. It can evaluate dimensions like style, register, and cultural appropriateness that traditional metrics largely ignore. And it can be adapted to new domains or quality criteria simply by updating the prompt, without retraining a model.

However, significant limitations remain:

• Fluency bias: LLMs tend to overweight fluency relative to accuracy, sometimes rating a fluent but unfaithful translation higher than an accurate but awkward one
• Position bias: In pairwise comparisons, LLMs may favor the first or second option depending on prompt structure
• Self-preference: An LLM may rate its own translations more favorably than those from other systems
• Inconsistency: The same LLM can produce different scores for the same input across runs, especially at higher temperature settings
• Language coverage: Evaluation quality degrades for lower-resource languages where the LLM has less training data

Mitigating these biases requires careful prompt engineering, temperature control (use zero or near-zero temperature for evaluation), position randomization in pairwise setups, and regular calibration against human MQM annotations.

Combining LLM Evaluation with Human Review

The most robust quality programs treat LLM evaluation as a middle tier between automated metrics and full human review. A practical three-tier architecture:

1. Automated metrics (COMET/BLEURT): Score all translations. Flag segments below threshold.
2. LLM-as-a-judge: Evaluate flagged segments and a random sample of passing segments. Annotate errors and assign severity.
3. Expert human review: Review LLM-flagged critical errors, high-risk content types, and a calibration sample to validate the LLM judge's accuracy.

This tiered approach can reduce the volume of content requiring human review by a large margin while maintaining or improving detection of critical errors, exactly the kind of efficiency gain that makes quality programs viable at enterprise scale. A localization platform that captures LLM annotations alongside MQM data simplifies calibration and auditability.

Setting Quality Targets by Content Type and Risk

Risk-Based Quality Tiers

Not all content warrants the same quality investment. A risk-based framework assigns quality tiers based on two factors: the consequence of a translation error and the visibility of the content.

This tiering has direct operational consequences. It determines how much you spend per word, how fast content moves through the pipeline, and which quality gates apply. Getting the tiers wrong, over-investing in low-risk content or under-investing in high-risk content, wastes budget or creates liability.

Building Gold Sets for Evaluation

A gold set (or benchmark set) is a curated collection of source segments paired with expert-validated reference translations and, ideally, MQM annotations. Gold sets serve as the ground truth against which you evaluate MT engines, LLM translation quality, and your own evaluation pipeline.

Effective gold sets share several characteristics:

• Representative: They cover the content types, domains, terminology, and linguistic challenges your production content actually contains
• Balanced: They include easy, moderate, and difficult segments, not just cherry-picked hard cases
• Annotated: Each segment has MQM error annotations from calibrated reviewers, not just a reference translation
• Versioned: They are updated as your content evolves, new terminology enters your domain, or new language pairs are added
• Sized appropriately: Research from WMT suggests a minimum of 500-1,000 segments per language pair for stable system-level rankings

Gold sets are an investment, but they pay dividends every time you evaluate a new engine, update a model, or validate a process change. Without them, you're making quality decisions on anecdotal evidence.

A/B Evaluation Across MT Engines and LLMs

With a gold set in hand, you can run controlled comparisons across translation systems. The methodology is straightforward but requires discipline:

1. Translate the gold set with each system under evaluation, using identical preprocessing and settings
2. Score all outputs with automated metrics (COMET, BLEURT) and LLM-as-a-judge evaluation
3. Run human MQM annotation on a statistically significant sample, at minimum, enough segments to detect meaningful quality differences with confidence
4. Analyze by dimension: Don't just compare aggregate scores. Break results down by error type (accuracy vs. fluency vs. terminology), content category, and language pair
5. Test for statistical significance: Use paired bootstrap resampling or similar methods to ensure observed differences aren't noise

This process lets you make evidence-based engine selection decisions and identify which system performs best for which content type, a common finding is that no single engine dominates across all dimensions and language pairs.

If you're running multi-engine evaluations and need a platform that supports quality benchmarking across content types, see how Ollang streamlines translation quality review.

Implementing Quality Gates in Production

Preflight Checks: Glossary and Terminology Validation

Quality gates that catch errors before translation even begins are the most cost-effective interventions. Preflight checks include:

• Glossary validation: Verify that source content uses approved terminology consistently, flagging ambiguous terms that could cause downstream translation errors
• Source quality analysis: Check for grammatical errors, ambiguous constructions, or untranslatable idioms in the source that will degrade MT output
• Terminology extraction: Identify domain-specific terms in new content and verify they exist in the translation memory and glossary before sending to translation

These checks are especially valuable for content types where terminology precision is non-negotiable, legal documents, medical device labeling, and software UI strings where a single term might map to a specific function or regulatory concept.

Platforms that maintain centralized translation memory and terminology databases make preflight checks operationally feasible. Ollang's translation memory and terminology management capabilities enable teams to enforce glossary consistency across documents, revisions, and content types, catching terminology drift before it reaches reviewers.

Constrained Decoding and Terminology Enforcement

Constrained decoding is a technique that forces the translation engine to use specific term translations in its output. Rather than hoping the MT engine or LLM selects the correct term from its training data, constrained decoding injects approved glossary entries as hard constraints during generation.

This is particularly important for:

• Regulated content where specific terms have legal definitions (e.g., "adverse event" in pharmacovigilance must not be rendered as "side effect")
• Brand terminology where product names, feature names, and marketing terms must remain consistent
• Software localization where UI strings must use the same term for the same function across the entire product

Not all translation engines support constrained decoding natively, so terminology QA as a post-processing step, scanning translations against the glossary and flagging deviations, serves as a complementary safeguard.

Staged Human Review Workflows

For content above the utility tier, human review remains essential, but it should be structured and targeted, not a blanket "review everything" mandate. A staged approach:

1. Automated triage: Automated metrics and LLM evaluation score all segments. Segments above the quality threshold and below the risk tier pass without human review.
2. Focused review: Segments flagged by automated checks, low confidence scores, terminology mismatches, or high-risk content categories, are routed to qualified reviewers.
3. Full review: Critical-tier content receives complete human review by domain-expert linguists, with MQM annotation.
4. Spot-check audit: A random sample of auto-approved content is reviewed periodically to validate that the automated gates are performing as expected.

This workflow requires a platform that can orchestrate routing rules, manage reviewer assignments, track MQM annotations, and aggregate quality data into actionable dashboards. Trying to manage this in spreadsheets or email breaks down quickly beyond a handful of language pairs.

Quality Considerations for Specific Content Types

Subtitles and Dubbing: Timing, Lip-Sync, and Emotion

Video localization introduces quality dimensions that text-only workflows never encounter. For subtitles, the translation must not only be accurate and natural but also:

• Fit reading speed constraints: Viewers need enough time to read each subtitle. The industry standard is roughly 17-20 characters per second for adult viewers, though this varies by language and platform
• Align with shot changes: Subtitles should not span cuts, as this disrupts reading flow
• Condense gracefully: When the source speech is too fast for subtitle reading speeds, the translator must compress meaning without losing critical information

For dubbing, the constraints multiply. The translated audio must match the original speaker's lip movements (lip-sync), preserve emotional tone and intensity, and fit the original timing. AI-generated dubbing has made significant strides, but quality evaluation must assess all these dimensions, a translation that is textually perfect but emotionally flat or visually mismatched fails the quality bar.

Evaluating these multimodal quality dimensions requires tooling that handles video, audio, and text together rather than treating them as separate streams. Ollang's multi-agent, multimodal localization capabilities allow teams to manage subtitle, dubbing, and audio localization within a single platform, enabling quality review that accounts for timing, synchronization, and textual accuracy simultaneously.

Legal Text: Fidelity and Precision

Legal translation operates under the strictest quality regime. Key requirements include:

• Semantic fidelity: The translation must preserve the exact legal meaning of the source, including distinctions between "shall" and "may," "warranty" and "guarantee," or jurisdiction-specific legal concepts that have no direct equivalent
• Terminological consistency: Legal terms must be translated identically throughout a document and across related documents (e.g., a contract and its amendments)
• Completeness: No clause, condition, or definition may be omitted, even partially
• Format preservation: Numbered clauses, cross-references, and defined terms must maintain their structure and referencing integrity

For legal content, automated metrics are useful for flagging potential issues but are never sufficient for quality certification. MQM annotation by legal-domain linguists, with a Critical error threshold of zero, is the standard. Document localization platforms that preserve complex formatting, tables, nested numbering, cross-references, reduce the risk of structural errors that manual reformatting introduces.

UI Strings: Context Is Everything

Software UI strings present a unique quality challenge: they are typically short, context-dependent, and must function within strict character limits and layout constraints. A button label, a tooltip, and an error message each demand different translation approaches, but they often arrive at the translator as a flat list of strings stripped of visual context.

Common quality failures in UI localization include:

• Ambiguity errors: "Save" could be a verb (save the document) or a noun (a save file), without context, the translator guesses
• Truncation: German, Finnish, and other languages that produce longer strings than English break layouts when character limits aren't communicated
• Inconsistency: The same action translated differently in different parts of the UI confuses users
• Placeholder handling: Strings with variables like {count} items selected require translators to understand the variable's role and handle pluralization rules correctly

Effective UI localization quality programs provide translators with screenshots or live context, enforce glossary consistency for UI-specific terminology, and test translations in the actual interface. API-based integration between the localization platform and the development pipeline enables continuous localization, translating new strings as they're committed to the codebase, rather than in batch at the end of a release cycle. Ollang's translation API integration supports this kind of programmatic localization workflow, keeping translations synchronized with development without manual handoffs.

Choosing a Platform That Operationalizes Quality

Essential Features for Quality at Scale

A localization platform that supports a real quality program needs more than a translation engine. It needs the infrastructure to measure, enforce, and improve quality systematically. The essential capabilities include:

• Automated quality scoring: Built-in or integrated COMET/BLEURT scoring, with configurable thresholds by content type and language pair
• MQM annotation tools: Structured error annotation interfaces for human reviewers, with severity weighting and inter-annotator agreement tracking
• Translation memory and terminology management: Centralized glossaries and TM that enforce consistency and enable preflight terminology validation
• Reviewer workflow orchestration: Routing rules that assign content to reviewers based on language, domain expertise, and risk tier
• Quality dashboards and analytics: Real-time visibility into quality metrics by language, content type, engine, and time period, enabling trend detection and root cause analysis
• Multi-format support: The ability to handle text, documents, video, audio, and software strings within a single quality framework rather than fragmenting quality data across tools
• API integration: Programmatic access to quality scores and review workflows so quality gates can be embedded in CI/CD pipelines, CMS publishing workflows, and content management systems

Comparing Platforms for Translation Quality Management

Ollang is positioned for teams that need quality management across multiple content modalities, not just text, but video, audio, documents, and software, within a single platform. Rather than stitching together separate tools for subtitle QA, document review, and software string validation, Ollang's multi-agent architecture coordinates quality workflows across all these content types, with shared translation memory and terminology ensuring consistency regardless of format. For organizations whose localization spans diverse content types, this eliminates the fragmentation that makes quality programs hard to sustain.

If you are deciding how to centralize quality across text, documents, and media without bolting together point tools, evaluate Ollang for unified, measurable quality control.

Building a Quality Program: Dashboards and Thresholds

Launching a translation quality program doesn't require perfection on day one. It requires a clear starting point, measurable targets, and a feedback loop. Here's a practical roadmap:

Month 1, Baseline

- Define your content risk tiers and assign quality thresholds (MQM penalty points per 1K words) for each

- Build or commission gold sets for your top three language pairs and highest-risk content types

- Run baseline evaluations of your current translation output against the gold sets using COMET and human MQM annotation

Month 2, Instrument

- Deploy automated quality scoring in your production pipeline

- Set up dashboards that track quality by language pair, content type, and translation engine

- Establish alert thresholds that trigger review when quality drops below acceptable levels

Month 3, Optimize

- Analyze error patterns from MQM data. Identify the most common and most costly error types

- Update glossaries, translation memories, and prompts to address systematic issues

- Implement terminology QA gates and constrained decoding for high-risk content

- Begin A/B testing alternative engines or configurations for underperforming language pairs

Ongoing

- Run quarterly gold set evaluations to track improvement

- Calibrate human reviewers and LLM judges against each other regularly

- Expand gold sets to cover new content types and language pairs as your program matures

- Review and adjust quality thresholds as your organization's risk tolerance and content mix evolve

The organizations that sustain quality programs are the ones that make quality data visible and actionable, not buried in spreadsheets but surfaced in dashboards that product managers, localization leads, and engineering teams can all access.

Frequently Asked Questions

What is MQM and how does it differ from BLEU?

MQM (Multidimensional Quality Metrics) is a framework for evaluating translation quality through structured error annotation, categorizing errors by type (accuracy, fluency, terminology, etc.) and severity (critical, major, minor), then computing a weighted quality score. BLEU is an automated metric that counts n-gram overlaps between a machine translation and a reference translation. The key difference is granularity and reliability: MQM tells you what is wrong and how serious it is, while BLEU gives a single correlation-based score that often fails to capture meaning-level errors. MQM requires human annotators (or LLM-based approximation), while BLEU is fully automated but much less informative.

Can LLMs reliably evaluate their own translations?

LLMs can serve as useful translation quality evaluators, but they should not be the sole judge, especially of their own output. Research has identified a self-preference bias where LLMs tend to rate their own translations more favorably. LLM-based evaluation works best as a middle tier: more nuanced than automated metrics, more scalable than human review, and most reliable when calibrated against human MQM annotations and used with zero-temperature settings and carefully designed rubrics. For critical content, expert human review remains essential.

How many segments do I need in a gold set for reliable evaluation?

For stable system-level comparisons, ranking one MT engine against another, WMT research suggests a minimum of 500-1,000 source segments per language pair. For segment-level evaluation (judging individual translations), you need fewer segments but more annotations per segment to establish inter-annotator agreement. Your gold set should be representative of your actual production content in terms of domain, difficulty, and content type. A gold set that only contains simple sentences will not reveal how systems handle complex, domain-specific content.

How do I decide which content types need human review versus automated evaluation?

Use a risk-based framework. Content where a translation error could cause legal liability, regulatory non-compliance, patient harm, or significant financial loss should always receive expert human review. Content that is customer-facing but lower-risk (marketing, help articles, UI strings) can be evaluated with a combination of automated metrics and LLM-based evaluation, with human review reserved for flagged segments. Internal or utility content, gisting, internal communications, typically needs only automated quality checks. The key is to define these tiers explicitly, assign measurable thresholds to each, and enforce them consistently through your localization platform.

Start Building Your Quality Program

Translation quality in AI localization is not a mystery, it's an engineering discipline. With the right frameworks (MQM for error taxonomy, COMET for automated scoring, LLM-as-a-judge for scalable evaluation), the right data (gold sets, calibrated reviewers, quality dashboards), and the right platform (one that orchestrates quality gates across content types and modalities), you can move from ad hoc spot checks to a defensible, measurable quality program.

Ollang brings these capabilities together, multi-format document localization, video and audio quality review, translation memory and terminology enforcement, API integration for automated pipelines, and structured reviewer workflows, so quality management scales with your content, not against it.

Book a Demo