Designing MQM-Driven QA Loops for AI Translation at Scale

When enterprise teams deploy AI translation across dozens of languages and millions of words per month, quality failures don't surface gradually, they arrive as regulatory notices, customer complaints, or brand-damaging screenshots on social media. The root cause is almost always the same: quality assurance was bolted on after the pipeline was built, not designed into it. The Multidimensional Quality Metrics (MQM) framework, developed through the QT21 research project and documented at themqm.info, gives localization teams a structured, auditable way to define what "good enough" actually means for every content type, language pair, and risk level. This article walks through the practical architecture of MQM-driven QA loops purpose-built for AI translation: how to select and weight error categories, calibrate automated and human evaluation, build escalation paths, and close the feedback loop so quality improves with every iteration.

Understanding the MQM Framework for Machine Translation

What MQM measures and why it matters for AI output

MQM is not a single score, it is a taxonomy of translation error types organized into a hierarchy that teams customize for their specific needs. The top-level categories in the MQM typology include Accuracy, Fluency, Terminology, Style, Locale Convention, Verity, and Design. Each category branches into finer subcategories; Accuracy alone includes Mistranslation, Omission, Addition, and Untranslated Text, among others.

What makes MQM especially relevant for AI translation is its ability to distinguish between error types that machine translation engines produce disproportionately. Neural MT and large language models, for instance, tend to hallucinate fluent-sounding text that is factually wrong, a failure that scores well on Fluency but catastrophically on Accuracy. Without a framework that separates these dimensions, aggregate quality scores mask the most dangerous errors.

MQM also provides a severity scale, typically Minor, Major, and Critical, that maps directly to business impact. A minor fluency issue in a marketing blog post is a different problem entirely from a critical accuracy error in a pharmaceutical label. This severity weighting is the mechanism that connects linguistic evaluation to enterprise risk management.

Mapping MQM error categories to content risk tiers

Not every piece of content warrants the same scrutiny. A pragmatic QA architecture starts by classifying content into risk tiers and then mapping MQM error categories to each tier with appropriate weights.

This tiering determines not just what you measure, but how much you sample and who reviews it. Critical-tier content may require 100% human review; low-tier content may rely entirely on automated quality estimation with periodic spot checks.

Selecting and Weighting Error Categories

Customizing the MQM typology for your domain

The full MQM typology contains over 100 error types. Using all of them is impractical and counterproductive, reviewers slow down, inter-annotator agreement drops, and the data becomes noisy. The better approach is to select a working subset of 15-25 error types that reflect your actual quality failures.

Start with a diagnostic phase: take a representative sample of AI-translated content across your top language pairs and have experienced reviewers annotate errors using a broad MQM subset. After two or three annotation rounds, patterns emerge. You may find that your MT engine rarely produces Addition errors but frequently generates Terminology inconsistencies. Prune the categories that appear in fewer than 2% of annotations and keep those that drive real quality impact.

Domain-specific subcategories often need to be added. Legal localization teams may need a subcategory under Accuracy for "Misinterpreted Legal Term of Art." Life sciences teams may need granular Terminology subcategories distinguishing between INN (International Nonproprietary Name) errors and dosage-form errors. The framework is designed to be extended, use that flexibility.

Assigning severity weights that reflect business impact

Severity weighting is where MQM stops being a linguistic exercise and becomes a business tool. The standard three-level severity scale (Minor = 1 point, Major = 5 points, Critical = 10 points) is a reasonable starting point, but teams should calibrate weights to their actual cost of failure.

Consider: a Terminology error in a regulated medical device interface may trigger a recall or a warning letter from a regulatory body. That same category of error in a social media post may generate a correction tweet. The penalty points should reflect this asymmetry.

A practical calibration method:

1. Identify the three to five most expensive quality failures your organization has experienced.
2. Trace each failure back to its MQM error type and severity.
3. Adjust weights so that the scoring model would have flagged those failures above your rejection threshold.
4. Validate the adjusted weights against a held-out sample of previously reviewed content.

Document your weighting rationale. When auditors or regulators ask why a translation was approved, you need to show that your scoring model was calibrated against real risk, not arbitrary defaults.

Automatic Quality Estimation vs. Reference-Based Metrics

How COMET, MQM-aligned QE, and reference-based metrics compare

Automated quality evaluation falls into two broad camps: reference-based metrics that compare MT output against a human reference translation, and reference-free quality estimation (QE) that predicts quality from the source and MT output alone.

• Reference-based metrics like BLEU, chrF, and TER have been workhorses of MT evaluation for decades, but they have well-documented limitations. BLEU, in particular, correlates poorly with human judgments at the segment level and penalizes valid paraphrases. More recent learned metrics like COMET have substantially improved correlation with human quality assessments. COMET models are trained on human evaluation data, including MQM annotations, and produce scores that track much more closely with how professional reviewers rate translations.
• Reference-free QE models are essential for production pipelines because reference translations rarely exist for new content. Models like CometKiwi (the QE variant of COMET) predict segment-level quality without a reference, making them suitable for real-time filtering. Some QE models are now trained directly on MQM-annotated data, meaning their predictions align with the same error typology your human reviewers use.

The practical recommendation is to use reference-free QE as the first automated gate in your pipeline, reference-based COMET for periodic benchmarking when reference sets exist, and to retire BLEU as a primary quality indicator for anything other than coarse regression checks.

Calibrating thresholds per language pair and content type

A single quality threshold across all languages is a recipe for either over-reviewing easy pairs or under-reviewing hard ones. German→English MT typically produces higher baseline quality than English→Japanese, and both differ from English→Arabic. Thresholds must be calibrated per language pair.

The calibration process works as follows:

• Collect a calibration set of at least 500 segments per language pair, annotated by human reviewers using your MQM scorecard.
• Run your QE model on the same segments and plot QE scores against human MQM scores.
• Identify the QE score threshold that achieves your target precision and recall for flagging segments above your MQM penalty limit.
• Set separate thresholds for each content risk tier, critical content gets a tighter threshold (higher QE score required to pass), low-risk content gets a more permissive one.

Recalibrate quarterly or whenever you change MT engines, fine-tune models, or update terminology resources. Threshold drift is one of the most common causes of QA loop degradation over time.

Human-in-the-Loop Review Protocols

Sampling strategies by content risk

Full human review of every translated segment is neither affordable nor necessary at scale. Sampling strategy should be risk-proportional:

• Critical-tier content: 100% human review, typically by two independent reviewers with adjudication for disagreements.
• High-tier content: Review all segments flagged by QE below threshold, plus a random sample of 10-20% of segments that passed automated checks.
• Medium-tier content: Review only QE-flagged segments, plus a smaller random sample (5-10%) for calibration purposes.
• Low-tier content: Periodic spot checks (1-3%) to monitor for systemic drift; no routine segment-level review.

The random samples that pass automated checks are essential, they are your mechanism for detecting false negatives in the QE model. Without them, you have no way to know if your automated gate is letting bad content through.

Escalation paths and decision authority

Every QA loop needs a clear escalation path that answers three questions: Who decides if a flagged segment is truly an error? Who decides if a pattern of errors requires halting publication? Who authorizes changes to the MT model, glossary, or prompt?

A typical escalation structure:

1. Reviewer level: Individual reviewers annotate errors using the MQM scorecard. They can approve, correct, or escalate individual segments.
2. Senior linguist / QA lead: Reviews escalated segments, adjudicates disagreements between reviewers, and identifies patterns across segments. Has authority to flag a batch for full re-review.
3. Localization program manager: Receives pattern-level escalations. Has authority to pause publication for a language pair, request MT model retraining, or invoke glossary updates.
4. Compliance / legal stakeholder: Engaged automatically when critical-severity errors are found in regulated content. Has authority to block go-live.

Document these escalation paths in a RACI matrix and review them during onboarding for every new reviewer or stakeholder. Ambiguity in decision authority is the fastest way to turn a QA loop into a bottleneck.

Running A/B tests for model and glossary changes

When you update an MT model, swap engines, modify a glossary, or change a prompt template, you need evidence that the change improved quality, not just an assumption. A/B testing within the QA loop provides that evidence.

The protocol:

1. Select a representative test set of source segments that covers your major content types and known difficult constructions.
2. Translate the test set with both the current (control) configuration and the proposed (variant) configuration.
3. Have reviewers annotate both outputs using the same MQM scorecard, blinded to which output came from which configuration.
4. Compare aggregate MQM scores, error type distributions, and severity distributions.
5. Apply statistical significance testing, a paired bootstrap resampling test is appropriate for MQM scores, before approving the change.

This discipline prevents the common failure mode where a glossary update fixes the five terminology errors that prompted it but introduces fluency regressions elsewhere. Teams using Ollang, which orchestrates the full localization pipeline including MT, can integrate A/B testing directly into their workflow without building custom tooling. You can book an end-to-end QA demo for enterprise pipelines at https://ollang.com/book-a-demo to see how this works in practice.

Building Quality Dashboards and Feedback Loops

Key metrics for quality dashboards

A quality dashboard is only useful if it surfaces actionable information, not just aggregate scores. Design your dashboard around these core views:

• MQM score trend by language pair: Track weekly or monthly average MQM penalty scores per 1,000 words. The trend matters more than the absolute number.
• Error type distribution: Show which MQM categories account for the most penalty points. This tells you where to focus MT improvement efforts.
• QE threshold performance: Track the false positive and false negative rates of your automated quality gate against human review results.
• Reviewer throughput and agreement: Monitor segments reviewed per hour and inter-annotator agreement (more on this below).
• Escalation volume and resolution time: Track how many segments get escalated, to whom, and how quickly they are resolved.
• Content type and risk tier breakdown: Allow filtering all metrics by content type and risk tier so stakeholders see the quality picture relevant to their domain.

Avoid vanity metrics like "percentage of segments with no errors", in high-volume AI translation, the interesting question is not whether errors exist but whether the error profile is within tolerance and improving over time. An integrated platform like Ollang can capture these metrics automatically and link them to review and model settings.

Reviewer training and inter-annotator agreement

The reliability of your entire QA loop depends on reviewers applying the MQM scorecard consistently. Inter-annotator agreement (IAA) is the metric that tells you whether they are.

Measure IAA by having two or more reviewers independently annotate the same set of segments at regular intervals, monthly is a reasonable cadence for active programs. Calculate agreement using Cohen's kappa or Krippendorff's alpha, both of which account for chance agreement. For MQM annotation, a kappa above 0.7 is generally considered acceptable; below 0.6 indicates a training or guideline problem.

When agreement drops:

• Review the MQM guidelines with the team, focusing on the error categories where disagreement is highest.
• Conduct calibration sessions where reviewers discuss and adjudicate specific disagreements.
• Simplify the scorecard if certain fine-grained distinctions consistently cause confusion, a reliable coarser annotation is more valuable than an unreliable granular one.

Invest in onboarding training that includes annotated examples for every error type and severity level in your scorecard. Provide a living annotation guide with borderline cases and their resolutions.

Closing the loop: using QA data to improve MT

The most valuable output of an MQM-driven QA loop is not a quality score, it is a structured dataset of errors that tells you exactly how to improve your MT system. Closing the feedback loop means routing QA findings back into the translation pipeline.

Concrete feedback paths:

• Terminology errors → Update glossaries and term bases; if using an LLM-based MT, update the terminology injection in your prompt.
• Consistent accuracy errors in a domain → Fine-tune the MT model on corrected segment pairs from that domain.
• Style and fluency patterns → Adjust style guides provided to the MT engine or update few-shot examples in prompt-based systems.
• Locale convention errors → Update locale-specific rules in your post-processing pipeline (date formats, number formats, address conventions).

Track the impact of each feedback action by comparing MQM scores before and after the change, using the A/B testing protocol described earlier. This creates a virtuous cycle: QA findings drive improvements, improvements reduce error rates, reduced error rates lower review costs, and lower costs allow you to invest in reviewing more content.

Ollang's platform is built around this principle of continuous improvement: it connects quality review data directly to MT customization, glossary management, and prompt optimization in a single workflow.

Acceptance Criteria for Go-Live and Audit Requirements

Defining go-live quality gates

Before any AI-translated content goes live, it must pass through defined quality gates. These gates should be documented, automated where possible, and non-negotiable for regulated content.

A robust go-live checklist:

• Automated QE gate: All segments score above the calibrated threshold for their language pair and content tier.
• Human review gate: Required human review is complete, all critical and major errors are resolved, and the aggregate MQM score per 1,000 words is below the acceptance threshold.
• Terminology compliance: Automated term-check confirms that all mandatory terminology is used correctly.
• Escalation resolution: No open escalations remain for the content batch.
• Stakeholder sign-off: For critical-tier content, the designated compliance or legal stakeholder has reviewed and approved.

Set numeric acceptance thresholds for each tier. For example, critical-tier content might require fewer than 5 MQM penalty points per 1,000 words with zero critical errors; medium-tier content might allow up to 15 penalty points per 1,000 words with no more than one major error per 500 words. These numbers should be calibrated against your organization's actual quality standards and adjusted as your MT pipeline matures.

Audit trail and compliance documentation

For regulated industries, pharmaceuticals, medical devices, financial services, legal, an auditable QA process is not optional. Your QA loop must produce documentation that demonstrates:

• Which MQM scorecard was used, including version and date of last update.
• Which segments were reviewed, by whom, and when.
• What errors were found, at what severity, and how they were resolved.
• What automated quality checks were run and what their results were.
• Who approved the content for publication and on what basis.
• What MT model version, glossary version, and configuration were used to produce the translation.

Store this documentation in a system of record that supports versioning and tamper-evident logging. When a regulator asks "How did you verify the quality of this translated patient information leaflet?", you need to produce a complete chain of evidence, not a verbal assurance that someone reviewed it.

This level of traceability is one of the reasons enterprise localization teams are moving toward integrated platforms rather than stitching together point tools. An end-to-end system captures the full provenance of every translated segment automatically.

Frequently Asked Questions

How many segments should we sample for human review?

The answer depends on your content risk tier and the maturity of your automated quality estimation. For critical content, review 100% of segments. For high-risk content, review all QE-flagged segments plus a 10-20% random sample. For medium-risk content, focus on QE-flagged segments with a 5-10% random check. The random sample is non-negotiable at every tier, it is your only mechanism for catching systematic false negatives in the automated gate.

Can MQM work with LLM-based translation, not just traditional NMT?

Yes. MQM is model-agnostic, it evaluates the output, not the system that produced it. In fact, LLM-based translation introduces error patterns (such as overtly creative paraphrasing or hallucinated content presented fluently) that make structured MQM evaluation even more important than it was for traditional NMT. The same scorecard, severity weights, and QA protocols apply regardless of whether the translation was produced by a dedicated NMT engine, a general-purpose LLM, or a hybrid pipeline.

How often should we recalibrate QE thresholds?

Recalibrate at least quarterly, and immediately after any significant change to your MT pipeline, a new model version, a major glossary update, a change in prompt strategy, or expansion to a new language pair. Threshold drift is gradual and invisible until it causes a quality incident. Build recalibration into your operational calendar rather than waiting for a failure to trigger it. Tools like Ollang can schedule and partially automate recalibration runs to keep thresholds aligned with human MQM data.

What inter-annotator agreement level is acceptable for MQM review?

A Cohen's kappa of 0.7 or above is generally considered acceptable for production MQM annotation. If agreement falls below 0.6, treat it as a process issue: review your annotation guidelines, run calibration sessions, and consider simplifying your scorecard. Unreliable annotations produce unreliable quality data, which undermines every downstream decision from MT improvement to go-live approval.

Start Building Your QA Loop Today

A well-designed MQM-driven QA loop transforms AI translation from a black box into a measurable, improvable, auditable process. The framework is proven, the automated tools are mature, and the methodology described here is implementable by any team willing to invest in structured quality management.

The hardest part is starting, defining your error categories, calibrating your first thresholds, training your first reviewers. Everything after that is iteration. If your organization is ready to move from ad hoc quality checks to a defensible, data-driven QA architecture for AI translation at scale, book a demo to see Ollang’s integrated approach to MT, review, and reporting: https://ollang.com/book-a-demo.