How to score a vendor's AI execution layer: a buyer evaluation framework for CTOs

Every localization RFP response reads the same way now. "AI-powered." "Enterprise-grade." "Integrates with your stack." Procurement teams collect these decks, put them side by side, and end up scoring vendors on how confidently they use the word "platform." That's not evaluation, it's marketing comprehension testing.

The problem compounds when the buyer is a CTO rather than a localization manager. You don't care whether a vendor's homepage says "AI-native." You care whether your build systems, CI pipelines, and your AI agents can call localization the way they call any other backend service, reliably, with typed responses, with auditable state, and without a human relaying files over email. Most vendor comparisons never test for this because most comparisons are built by people evaluating translation quality, not system architecture.

This article gives you a different tool, a five-axis rubric built around what actually distinguishes a callable execution layer from a service wrapped in a web portal. It's vendor-neutral by design, you can run it against any localization vendor's documentation. Then, in the interest of transparency, we apply it to Ollang's own documented concept model, citing the documentation directly rather than the marketing copy.

Why checklist evaluations miss the point

A typical RFP checklist asks: Does the vendor have an API? Does the vendor support machine translation? Does the vendor offer human review? Every vendor answers yes to all three, because the questions are binary and the vendors control the wording of their own answers.

An architecture-based rubric asks different questions. Not "does it have an API" but "what is exposed through that API, and can an autonomous agent invoke it without a human in the loop." Not "does it support human review" but "is review a first-class object with its own state, score, and audit trail, or is it a support ticket that happens to route to a translator." The difference between these framings is the difference between evaluating a vendor's claims and evaluating a vendor's system design.

The five-axis rubric

Score each vendor from 0-4 on each axis. A vendor that scores well on modality breadth but poorly on programmatic depth is a translation shop with a marketing page. A vendor that scores well on programmatic depth but poorly on governance is a developer toy that will fail your compliance review. You need all five.

Axis 1: Programmatic access depth

Not "has an API", what surfaces does it expose, and do they compose into a real integration path: REST endpoints, an agent-native protocol, an SDK, and prebuilt agent actions.

Axis 2: Model orchestration and BYOK

Can the vendor route work across multiple underlying models and providers, and can you supply your own credentials rather than being locked into a single opaque model the vendor has bundled into its price?

Axis 3: Governance and audit trail

Are review, approval, and quality scoring modeled as objects with state and history, reviewable at any point, or are they manual processes that happen to produce a final file?

Axis 4: Quality feedback loop

Does the system measure its own output, automated QC scoring, human-edit-percentage tracking, score progression over time, or does "quality" mean a single pass/fail checkbox at delivery?

Axis 5: Modality breadth

Does the same programmatic and governance model apply across text, documents, software strings, video, audio, and live speech, or does each modality require a different vendor, contract, or integration?

Testing programmatic depth without taking the vendor's word for it

"We have an API" is the least useful sentence in any vendor pitch, because a single authenticated endpoint that accepts a file and returns a status code technically qualifies. The real test is whether the vendor's documentation shows layered access that matches how your engineering org actually works.

Ask for four specific items. First, a REST API with documented authentication, request/response formats, pagination, and error handling, not a PDF of sample calls but a reference a developer can build against without a support call. Second, an MCP-style server, a Model Context Protocol interface that lets an AI agent invoke localization actions directly as tools, rather than a human copying output from a chat window into a ticket. This is the piece most vendor documentation still doesn't have, because it wasn't a requirement until agent workflows became normal in engineering orgs. Third, an SDK that wraps the raw API in typed methods for your primary language, ideally with utilities for the workflow specifics of localization, asset scanning, string extraction, and CMS capture, rather than a generic HTTP wrapper. Fourth, agent skills or an equivalent, file-based or natural-language action definitions that let coding agents and IDE-native assistants trigger localization work as part of their existing task flow, without you standing up a server.

When you run this test against Ollang's documentation, all four layers are present and named as distinct integration paths rather than as marketing synonyms for the same thing. The documentation covers APIs, MCP server, SDK, Skills, and platform concepts for AI-native localization. The docs explicitly cover four ways to integrate, REST API, MCP, SDK, and Skills, and describe when to use each and how they fit together, which is the layered structure this axis is scoring for rather than a single generic "developer tools" page.

Specifically, programmatic uploads, orders, projects, revisions, QC, human review, and webhooks are exposed through an API-key authenticated REST interface, while a hosted Model Context Protocol server, authenticated via OAuth 2.0 and PKCE, integrates with Claude, Cursor, Claude Code, Devin, Replit, Windsurf, and similar agent environments. The SDK layer is a TypeScript/Node.js SDK for asset scanning, i18n workflows, CMS capture, and a typed REST client, and the Skills layer is file-based Agent Skills for Cursor, Claude Code, Codex, Windsurf, and Cline that support natural-language operations with no server to run. If you're scoring an RFP response and a vendor claims parity with this, ask them to point you to a documentation page as specific as https://api-docs.ollang.com/home that names each surface and its auth model rather than describing full API access in prose.

Scoring governance as first-class objects, not add-ons

This is where most vendors fail quietly. Ask a vendor whether they support human review, and they'll say yes, because somewhere in their operation, a person reads translated output before it ships. The question that actually matters for a CTO is whether that review step is a modeled object in the system, with its own identity, state, and history, or whether it's an operational process bolted onto the outside of the pipeline.

The distinction matters for reproducibility, automation, and audit: can you query the state of a specific review at a specific point in time, can a workflow branch on the outcome of a review programmatically, and can you produce a record of who approved what, and when, without asking a project manager to reconstruct it from email.

Score this axis by checking whether the vendor's documented concept model includes review as a typed, addressable entity, something you'd expect to see named alongside the other core objects in the system (folders, projects, orders) rather than described only in a features page. In Ollang's documented model, review gates are named as a specific object type you attach to an order: you can add a Level 1 review gate to any order to route output to Ollang-managed linguists or your own LSPs and editors. That "or your own" clause matters for governance scoring, it means the review step isn't a black box the vendor controls unilaterally, it's a configurable gate you can route to your own language service providers.

Quality scoring should follow the same pattern. A vendor that reports "quality" as a single delivered score is giving you a snapshot, not a feedback loop. Ollang's documentation describes QC as a multi-dimensional, tracked object: AI QC across accuracy, fluency, tone, and cultural fit, human QC annotations, and QC score progression and human-edit-percentage analytics. Score progression and human-edit percentage are the pieces to look for specifically, they tell you whether quality is measured once or tracked as a trend you can act on, which is the difference between a report and a governance signal.

When you evaluate any vendor on this axis, don't accept "we do human review" as a sufficient answer. Ask them to show you the object model: what fields does a review gate have, what states can a QC score move through, and can you pull human-edit-percentage data through the API rather than requesting it from a client success manager.

BYOK as the real test of model-agnosticism

Every vendor now claims to be "model-agnostic." Almost none of them let you prove it. The tell is BYOK, bring-your-own-key support for the underlying model providers.

If a vendor routes all translation and generation work through models they've licensed and bundled into their pricing, you are locked into their model roadmap regardless of what they say about flexibility. You can't swap providers when a better model ships, you can't route sensitive content through a model your compliance team has separately vetted, and you can't audit which underlying provider actually processed a given piece of content. "Model-agnostic" in that setup means the vendor has picked several models on your behalf, not that you have a choice.

BYOK inverts that. If a platform's concept model documents BYOK as a first-class configuration option, something you set at the account or project level, not a special enterprise addendum negotiated separately, it signals the vendor built model orchestration as infrastructure rather than as a fixed pipeline with a marketing label. Ollang's documentation lists BYOK alongside the other core platform concepts a buyer needs to understand before integrating: folders, projects, orders, levels, workflows, review gates, LSPs, QC, BYOK, and more, treated as the documented operational model of the platform, not as separate feature bullets.

When you score a vendor on this axis, ask two follow-up questions the marketing page won't answer. Can you bring keys for multiple providers simultaneously and route by content type or sensitivity, and is that routing visible and controllable through the same API you use for everything else, or does it require a separate configuration console. If the vendor can't answer both, treat "model-agnostic" as unverified.

A worked example: applying the rubric

To make the rubric concrete, here's how it applies to Ollang's documented capabilities, scored against the same five axes, using only what the documentation states.

Programmatic access depth: High. The documentation names REST API, MCP, SDK, and Skills as distinct, separately documented integration surfaces, each with its own authentication model, API-key authentication for the REST layer, and OAuth 2.0 with PKCE for the hosted MCP server. That's a layered surface, not a single endpoint with a badge.

Model orchestration and BYOK: Present as a named platform concept rather than an enterprise-only side agreement, documented alongside folders, projects, orders, levels, workflows, review gates, LSPs, and QC as core operational objects a buyer needs to understand.

Governance and audit trail: Review is modeled as an attachable object, a Level 1 review gate can be added to any order, routing output to Ollang-managed linguists or your own LSPs and editors, which supports the reproducibility and routing-control tests above.

Quality feedback loop: Scores as a genuine loop rather than a single delivery checkpoint, because the documentation names AI QC across accuracy, fluency, tone, and cultural fit, human QC annotations, and QC score progression and human-edit-percentage analytics as tracked, not just reported.

Modality breadth: Spans AI dubbing, subtitle translation, CC, document, studio dubbing, visual translation, and website translation under the same order and workflow structure, meaning a CTO isn't managing five vendor relationships to cover five content types.

This is what an honest self-application of a rubric looks like: specific, citable, and limited to what's documented rather than what's implied. Run the same five questions against any competitor's documentation page and see how much of it survives the same scrutiny.

The argument this framework is really making

The deeper claim is not that Ollang scores well on five axes, but that most localization vendors can't be scored this way at all because their architecture wasn't built to survive the question. A vendor built as a service business with a portal on top will have an API because everyone expects one, will have "human review" because it's a defensible line item, and will call itself model-agnostic because the word costs nothing to say. None of that produces objects a CI pipeline can query, a governance team can audit, or an AI agent can invoke without a human relaying instructions in the middle.

The shift this framework measures is structural, localization stops being a project you commission and becomes infrastructure you call, with the same expectations for typed access, state, and observability you'd apply to any other backend dependency in your stack. Score for that, and the checklist vendors sort themselves out on their own.