Beyond BLEU scores: how automated QC evaluation lets AI agents self-grade translation quality before human review

Every VP of localization faces the same problem: quality assurance that doesn't scale with volume. Sampling ten percent of a batch works until the batch is machine-generated at ten times the previous velocity. Spreadsheet-based LQA, export segments, assign a linguist, wait for scores, reconcile in a tracker, was built for a world where translation volume moved slower than review capacity. That world is gone. Content now arrives from AI pipelines faster than human review teams can spot-check, and BLEU scores only measure n-gram overlap against a reference translation, not whether the output meets a regulator, a brand guideline, or a customer in a specific market.

The point is simple: enterprise localization governance is moving from manual spot-checks to programmatic quality gates, and the mechanism that makes this possible is a QC evaluation that runs as a callable API step rather than a human process. Ollang's QC evaluation endpoint scores translations across multiple dimensions at the segment level and returns exactly which segments need a human, which means quality assurance stops being a bottleneck you schedule and becomes a step your pipeline executes.

What the QC evaluation endpoint actually scores

The endpoint assesses four named criteria: accuracy, fluency, tone, and cultural fit. Each criterion answers a question a reviewer would otherwise judge manually: does the translation preserve the source meaning; does it read naturally in the target language; does it match the intended register; and does it land correctly for the cultural context of the target market. You can choose which criteria to run for a given batch, and you can add a custom instruction to focus the evaluation, for example prioritizing technical terminology accuracy for a product manual or tone for a marketing asset.

This is not a synchronous call that blocks your pipeline. The evaluation runs asynchronously. When you call the endpoint it returns an evaluation ID, confirms the job has started, and calls a callback URL you supply when scoring finishes. That queuing pattern, "QC evaluation started. You will be notified when it's completed," plus fields like "evalId": "eval_abc123def456", "creditsUsed": 5, "isProcessing": true, is the operational detail that matters. It means QC evaluation behaves like any other backend job in your stack: fire it, track it by ID, consume the result when it lands, and avoid having a human sit idle waiting for a spreadsheet to come back from a vendor.

The output is not a single aggregate number. It returns segment-level analysis alongside the criterion scores, the granularity that decides what happens next.

Segment-level scoring changes the reviewer's job

The main operational shift is this: instead of re-reviewing an entire deliverable because a QA process flagged "issues somewhere," a segment-scored QC run tells you precisely which lines fell short and on which dimension. A 40,000-word document translation might come back with 98% of segments clean and forty segments flagged for cultural fit or tone. Under a manual LQA model, that finding typically triggers a full re-review pass, because sampling-based QA cannot isolate problems with that precision.

With segment-level scores, a native-speaking reviewer's time goes directly to the content that needs attention. AI agents and pipeline logic can route only flagged content to human eyes rather than routing entire orders. The Skills workflow that wraps this endpoint describes the sequencing: upload the file via ollang-upload and receive a projectId, create the order with ollang-order-create and receive orderId(s), monitor status with ollang-order-get until it is completed, run the quality check with ollang-qc-eval to receive scores and segment analysis, and report issues with ollang-revision to create revisions if needed. An agent can execute this end to end: create the order, poll until complete, run the QC evaluation, and when segment scores fall below threshold, either open a revision request or escalate the order for professional linguist review. All Ollang API operations (except the health check) require an API key, available at lab.ollang.com. Human reviewers stop being a filter applied to everything and become a resource applied to the fraction of content that actually needs judgment.

Asynchronous, callback-based QC vs. the LQA spreadsheet

Compare this to the standard manual LQA cycle: export a sample, assign it to a linguist or panel, wait for scored feedback (often days), reconcile scores manually against a rubric, decide what to fix, and re-export for a second pass if needed. Every step is a handoff, and every handoff is a place where status gets lost, files get versioned incorrectly, or accountability becomes fuzzy about whose sign-off covered which batch.

The API pattern collapses that cycle into a single callable step with a durable record. You get an evalId the moment the job starts, a creditsUsed figure so cost is trackable per evaluation, and an isProcessing flag your system can poll or wait on via the webhook. There is no spreadsheet to lose, no reviewer assignment to chase, and no ambiguity about which version of a file was scored, because the evaluation is tied directly to the order it ran against. A GET on the order retrieves full details by ID, including status, languages, documents, and QC results. That single retrieval call is the audit record a spreadsheet-based process was always trying and failing to be.

Auditability and accountable sign-off for regulated content

For brand-sensitive or regulated content, legal documents, financial disclosures, product safety copy, the question a compliance or legal team will ask is not simply whether someone reviewed the content, but exactly what was checked, on what criteria, and what the result was. A manual LQA spreadsheet answers that question poorly: scores live in disconnected files, criteria definitions vary reviewer to reviewer, and there is rarely a timestamped, immutable link between the specific translated segment and the judgment made about it.

A QC evaluation run through the API produces a structured, criterion-tagged, segment-level result tied to an order ID and an evaluation ID, retrievable on demand. That is raw material for an audit trail. It also means quality thresholds can be defined once as policy, for example requiring human sign-off on any segment scoring below a cultural-fit threshold in regulated markets, and enforced consistently by the pipeline rather than depending on a reviewer remembering to apply the rule. Quality becomes a governance primitive: the gate is programmable, and the record of it firing is queryable after the fact.

QC as API vs. the MQM framework

The established alternative in the localization industry is the Multidimensional Quality Metrics framework, an error-typology model where trained linguists annotate translations against a taxonomy of error categories and severities, then roll those annotations into a weighted score. MQM is rigorous, and it is the right tool when the goal is a periodic, deeply calibrated audit of vendor or engine performance across a large corpus. But it is heavyweight by design: it assumes trained human annotators applying a shared taxonomy, a review cycle measured in days, and tooling built for LSP-style project management rather than for a machine to invoke mid-pipeline.

Ollang's QC evaluation is a different category of object. It does not attempt to replace MQM's analytical depth for a formal linguistic audit. It is built to be called by a pipeline or an AI agent as a routine step, the way a CI system calls a test suite before a deployment. Agents can check core things like meaning preservation, terminology, consistency, structure, and instruction adherence across modalities, while also handling file-specific rules such as subtitle timing and length, dubbing sync and speech naturalness, or JSON/schema integrity. A shared scoring core plus modality-aware checks for subtitle timing, dubbing sync, or document structure makes the evaluation callable at the frequency an execution layer requires: every order, not every quarter.

Where this sits in the stack

In practice, the QC endpoint sits directly after order completion and directly before your routing logic. An agent working through Cursor, Claude Code, Windsurf, Cline, or Codex can trigger the entire sequence in natural language, "Run a QC check on order #12345 for accuracy and tone," or your pipeline can call the REST endpoint programmatically as part of a CI-style release gate for a translated document, a subtitle file, or an AI-dubbed video. When scores clear threshold, the order proceeds. When segments fall short, the same workflow can open a revision request or push the order into professional linguist review without a manual handoff initiating it. The pipeline already knows when a batch is ready for QA.

The governance argument

The deeper shift is that quality becomes something you can specify, measure, and enforce with the same rigor as latency or uptime. A VP of localization who can point to a programmable threshold, a segment-level record, and a retrievable evaluation ID for every order has what a sampling percentage never provided: a queryable answer to "how do we know this was good enough," asked by legal, a regulator, or a CFO. BLEU never answered that question, and neither did a spot-check. A scored, auditable, callable QC step does, and that turns quality assurance from a service you wait on into infrastructure you govern.