Text Localization QA: MQM, Automation, and Release Quality Gates

Most localization defects that reach production are preventable. They slip through not because teams lack skilled linguists, but because quality assurance is treated as a final hurdle rather than an engineered process. A missing placeholder crashes a mobile app. An untranslated legal clause exposes liability. A truncated button label erodes user trust in a new market. These failures share a common root: the absence of structured, repeatable QA with clearly defined error taxonomies, automated checks, and explicit release criteria.

This guide walks through the Multidimensional Quality Metrics (MQM) framework, automated validation layers, human review workflows, and the release quality gates that separate localization programs shipping with confidence from those shipping with crossed fingers. By the end, you will have a concrete blueprint for a defensible QA model.

What Is MQM and Why Does It Matter for Localization QA?

The Multidimensional Quality Metrics framework, referenced in standardization efforts such as the ASTM International work item WK54884, provides a hierarchical error typology purpose-built for translation quality evaluation. Rather than relying on vague judgments like "this translation feels off," MQM gives evaluators a shared vocabulary and a scoring mechanism that produces comparable, actionable results across languages, vendors, and content types.

MQM Error Typology Explained

MQM organizes translation errors into a tree of categories and subcategories. The top-level dimensions include:

Each error is assigned a severity level, typically critical, major, or minor, which feeds directly into scoring. A critical accuracy error (e.g., a dosage instruction mistranslated in a pharmaceutical label) carries far more weight than a minor fluency issue (e.g., a slightly awkward conjunction).

How MQM Scoring Works in Practice

MQM scoring converts annotated errors into a single quality score per language or per content segment. The most common formula applies penalty points by severity:

• Critical: 10 points per error (or higher, depending on the domain)
• Major: 5 points per error
• Minor: 1 point per error

The raw penalty total is then normalized against the word count of the evaluated sample to produce a score per thousand words. Organizations set pass/fail thresholds based on content type, marketing copy might tolerate a higher score than medical device instructions.

This scoring approach does two things traditional "percentage accuracy" methods cannot. First, it differentiates between a translation riddled with trivial typos and one containing a single catastrophic mistranslation. Second, it produces data that can be trended over time, compared across vendors, and used for contractual SLAs.

Setting Severity Weights and Pass/Fail Thresholds

The right thresholds depend on risk. A common starting framework:

These numbers are not universal rules, they are calibration points. The critical step is documenting the thresholds, getting stakeholder agreement, and applying them consistently. Revisit them quarterly as your error data accumulates and patterns emerge.

How to Build an LQA Sampling Plan

Evaluating every translated word is neither practical nor cost-effective. Linguistic Quality Assurance (LQA) relies on statistical sampling to draw reliable conclusions about overall quality from a manageable review volume.

Statistical Sampling: How Many Segments to Review

The goal of sampling is confidence: you need enough data points to trust that the sample reflects the full delivery. A common baseline is reviewing 1,500-2,000 words per language per delivery, which aligns with the sample sizes historically recommended by localization industry bodies for achieving a reasonable confidence interval.

For high-risk content, increase the sample. For mature language pairs with a strong track record, you can reduce it. The key variables are:

• Total word count of the delivery
• Number of unique translators contributing
• Content risk level (regulatory vs. informational)
• Historical quality trends for the language pair

Stratified sampling, selecting segments proportionally from different content types, translators, or modules, gives a more representative picture than random sampling alone.

Stratified vs. Random Selection

Random sampling is simple but can miss systematic issues. If one translator handled all the legal content and another handled the UI strings, a purely random sample might over-represent one and under-represent the other.

Stratified sampling divides the corpus into meaningful strata (by translator, content type, module, or source complexity) and draws proportional samples from each. This ensures your evaluation covers the full risk surface. In practice, many teams use a hybrid: stratified selection for the bulk of the sample, with a random overlay to catch unexpected patterns.

Reviewer Calibration and Inter-Annotator Agreement

MQM scores are only meaningful if reviewers apply the typology consistently. Calibration sessions, where two or more reviewers independently evaluate the same set of segments, then compare and discuss their annotations, are essential before any evaluation cycle begins.

Measure inter-annotator agreement (IAA) using metrics like Cohen's kappa or percentage agreement on severity classification. An IAA below 0.6 typically signals that the error definitions or severity guidelines need clarification. Document disagreements and update your evaluation guidelines accordingly. This is not overhead; it is the foundation of trustworthy quality data.

Automated QA Checks Every Localization Team Needs

Automation handles the mechanical, pattern-based errors that humans should never have to waste time catching. A well-configured automated QA layer eliminates entire categories of defects before a human reviewer ever opens the file.

ICU Placeholder and Variable Validation

ICU MessageFormat patterns (e.g., {count, plural, one {# item} other {# items}}) are notoriously fragile. A missing brace, a renamed variable, or a dropped plural category will cause runtime crashes or display raw code to users. Automated validators parse both source and target, flag mismatched or missing placeholders, and verify that plural rules align with the Unicode CLDR plural categories for the target locale.

HTML, Markdown, and XML Tag Integrity

Translated content embedded in HTML, Markdown, or XML must preserve the tag structure of the source. Unclosed tags, reordered nesting, or deleted attributes break rendering. Automated checks compare the tag skeleton of source and target segments, flagging any structural divergence. This is especially critical for web and help center content where a single unclosed <strong> tag can cascade formatting errors across an entire page.

Terminology and Glossary Enforcement

Term consistency is both a brand concern and a functional one, using "cancel" in one screen and "abort" in another confuses users. Automated term checks compare translated segments against an approved termbase and flag deviations. The best implementations distinguish between mandatory terms (must be used as specified) and preferred terms (recommended but flexible), reducing false positives that erode translator trust in the tool.

String Length, Truncation, and UI Fit

German text averages 30% longer than English. Finnish compound words can stretch a single UI label beyond any reasonable button width. Automated length checks compare target character or pixel length against defined limits per string, flagging potential truncation before it reaches a build. For mobile apps, where screen real estate is tightly constrained, this check alone prevents a significant share of visual defects.

Bidirectional (Bidi) and RTL Control Characters

Arabic, Hebrew, and other right-to-left scripts require correct use of Unicode bidirectional control characters (LRM, RLM, LRE, RLE, PDF, and the isolate equivalents). Misplaced or missing bidi marks cause numbers, punctuation, and mixed-direction text to render in the wrong order. Automated checks verify that bidi controls are present where expected and absent where they would cause harm, particularly in strings that mix RTL text with LTR elements like URLs, file paths, or brand names.

Spellcheck, Banned Terms, and Locale-Specific Linting

Spellcheck is table stakes, but locale-specific linting goes further. It verifies that date formats match the target locale (DD/MM/YYYY vs. MM/DD/YYYY), that currency symbols are correct, that measurement units are converted where required, and that culturally inappropriate or legally banned terms are flagged. Banned term lists are especially important for regulated industries where specific claims or phrases are prohibited in certain markets.

Screenshot and In-Context Review

Automated checks catch structural errors. In-context review catches everything else, the truncated label that technically fits the character limit but loses its meaning, the translated tooltip that overlaps an adjacent element, the cultural reference that lands wrong in context.

Pseudo-Localization and Visual QA

Pseudo-localization, replacing source strings with accented, expanded versions before translation begins, exposes hardcoded strings, concatenation issues, and layout fragility early in the development cycle. Running pseudo-localized builds through visual regression tools catches UI problems before real translations are even produced, shifting defect detection dramatically left.

Functional and Contextual Testing in Localized Builds

Screenshots and in-context review tools render translated strings in their actual UI context, either through live application builds or simulated environments. Reviewers see exactly what users will see, which surfaces issues invisible in a spreadsheet: overlapping elements, broken layouts, contextually inappropriate translations (e.g., "Save" translated as the financial term rather than the data-persistence action), and missing translations that fall back to source.

For teams managing localization across software, websites, and multimedia content, platforms such as Ollang provide the execution infrastructure to manage these review workflows across content types. Ollang centralizes in-context review and traceability, reducing manual handoffs and preserving context throughout the pipeline. You can see how in-context review fits into a broader localization pipeline when you book a demo with Ollang.

Bug Triage, Root Cause Analysis, and Regression Prevention

Finding bugs is only half the job. The other half is ensuring they do not recur.

Classifying and Prioritizing Localization Bugs

Not all localization bugs are created equal. A triage framework should classify each defect by:

• Severity: Does it block functionality, degrade UX, or merely violate a style preference?
• Scope: Does it affect one string, one language, or all languages?
• Root cause category: Is this a translator error, a source content issue, a tooling failure, or an engineering defect?

Critical and blocker bugs (crashes, data loss, legal exposure) go to the top of the queue. Major bugs (functional confusion, significant truncation) follow. Minor bugs (style inconsistencies, non-preferred terminology) are batched for the next cycle.

Root Cause Analysis: Source, Process, or Tooling?

Every recurring defect deserves a root cause investigation. Common patterns include:

• Ambiguous source text that leads multiple translators to different interpretations. Fix: improve source authoring guidelines or add context notes.
• Missing context in the translation environment. Fix: provide screenshots, character limits, and usage descriptions with every string.
• Tooling gaps where the TMS or QA tool fails to catch a known error pattern. Fix: add or refine automated check rules.
• Vendor skill gaps where a specific translator or team consistently underperforms on a content type. Fix: reassign, retrain, or replace.

Document root causes in a shared tracker. Over time, this data becomes the most valuable asset in your QA program, it tells you exactly where to invest to prevent the next wave of defects.

Regression Protocols

A regression protocol ensures that fixed bugs stay fixed. Maintain a regression test suite, a curated set of previously failed strings and scenarios, that is re-evaluated with every new delivery. Automated checks should encode past failure patterns as rules. Human reviewers should receive updated guidelines reflecting lessons learned. If a specific error type recurs after being addressed, escalate it as a process failure, not just a translation error.

Defining Release Quality Gates

Quality gates are the checkpoints that a localized delivery must pass before it ships. Without them, release decisions default to gut feel, schedule pressure, or whoever shouts loudest.

Gate Structure: What to Check and When

A typical multi-gate structure:

Each gate has explicit pass/fail criteria. A failure at any gate blocks progression until the issue is resolved or a documented exception is granted.

Thresholds by Content Type

Reuse the MQM threshold table from earlier as the backbone of Gate 2 criteria, but augment it with gate-specific rules:

• Software UI: Zero critical or major accuracy errors. No unresolved truncation flags. All placeholder checks pass.
• Legal / Regulatory: Zero errors of any severity in legally binding clauses. Full in-context review of all regulated content, not just a sample.
• Marketing: No brand name errors. Tone and style alignment confirmed by in-market reviewer.
• Multimedia (subtitles, voiceover scripts): Timing and character-per-second limits verified. No untranslated segments.

Dashboards for Continuous Monitoring

Quality gates produce data. That data belongs in a dashboard, not a spreadsheet buried in someone's inbox. Effective localization QA dashboards track:

• MQM scores by language, vendor, and content type over time
• Automated QA pass rates per delivery
• Defect density (errors per thousand words) trended by release
• Time to resolve localization bugs by severity
• Regression recurrence rate
• Gate pass/fail rates per release cycle

These metrics enable data-driven conversations with vendors, product teams, and leadership. They transform QA from a cost center into a visible quality function with measurable impact.

Frequently Asked Questions

How is MQM different from a simple "good/bad" translation review?

MQM is a structured error typology with categories and severity weights that produce a numerical score normalized by word count, enabling comparable, trended measurement across languages and vendors. Platforms like Ollang operationalize MQM so scores feed dashboards and SLAs.

How many words should we review per language in each LQA cycle?

A common industry baseline is 1,500-2,000 words per language per delivery for standard-risk content, with larger samples for high-risk material and smaller samples for mature, proven language pairs; always use stratified sampling to ensure coverage. Your TMS or QA platform can automate sampling selection and track coverage.

Can automated QA replace human linguistic review?

No, automated QA reliably catches structural and mechanical errors, but people are required to judge naturalness, context, and cultural fit. The most effective programs layer automation first, then focus human reviewers on nuanced linguistic decisions.

What should we do when a delivery fails a quality gate?

Record the failure with MQM annotations, automated reports, and screenshots, triage by severity, and require correction of critical and major defects before release. Use root cause analysis to update source content, tooling, or vendor assignments and encode those fixes into your regression suite.

Building a QA Model That Scales

Designing a defensible localization QA model is not a one-time project. It is an iterative system that improves with every release cycle as you accumulate error data, refine thresholds, expand automated checks, and tighten feedback loops with translators and vendors.

Start with the fundamentals: adopt MQM as your error typology, implement the automated checks that match your content types, define explicit quality gates with pass/fail thresholds, and build a dashboard to make quality visible. Then iterate, calibrate reviewers, analyze root causes, update regression suites, and adjust thresholds as your program matures.

For organizations managing localization across text, software, websites, video, audio, and legal documents, the operational complexity of coordinating these QA workflows across content types and languages demands purpose-built infrastructure. Ollang is the AI execution layer for enterprise localization, integrating translation quality review into a unified pipeline across every content format.

Ready to operationalize your localization QA? See how structured quality gates, automated checks, and continuous monitoring work in practice across your full localization portfolio when you book a demo with Ollang.