The CTO's scorecard for evaluating a localization vendor's AI execution layer

Your last localization RFP probably asked vendors to list supported language pairs, quote a per-word rate, and describe their translation memory tooling. Every vendor answered those questions well, because every vendor built their pitch deck around them. None of those questions tell you whether the platform can be called from your CI/CD pipeline, whether it depends on a single model provider you don't control, or whether the "AI-powered" quality claims on page one survive an actual API call.

That's the gap. Localization RFPs were built for an era when the deliverable was a translated file handed off by a project manager. The deliverable now is a callable service that your systems and agents invoke directly, at whatever volume and cadence your business needs. A CTO evaluating that shift needs a different rubric, one built around model neutrality, programmatic access, and enforceable governance, not language coverage and word counts. This article gives you that rubric, and uses Ollang's documented capabilities as a concrete reference point for what a real answer to each question looks like.

Why the old scorecard misses what matters

Language coverage and per-word pricing made sense when translation was a manual service purchased per project. They tell you almost nothing about an execution layer, because they do not touch the three things that determine whether the platform will hold up in production: which model is doing the work and what happens when that model changes, degrades, or is deprecated; whether your systems can call the platform directly or whether every request still routes through a portal and a project manager; and whether quality and review controls are configurable and auditable or you are taking the vendor's word for it.

None of that shows up on a language-pair matrix. It shows up in the API documentation, the integration surface, and the governance primitives the vendor actually ships, which is where this scorecard points instead.

The five-axis scorecard

Score each vendor 1-5 on each axis. Don't accept a sales deck as evidence, require the vendor to demonstrate the claim against their own documentation or a live call.

Axis 1: Model neutrality / BYOK

The question to ask is whether the translation engine is a fixed black box, or whether you can choose, swap, and combine the underlying models.

Single-provider lock-in is a hidden dependency risk, and it is not theoretical. Meta shut down its own Llama API public preview in mid-2026, the company confirmed, and it is winding down Llama API, which had remained in public preview since launch. If your localization vendor built its entire pipeline against one provider's endpoint, that kind of deprecation becomes your migration project, on someone else's timeline, with your production content in the middle of it.

An execution layer avoids that by architecting for multiple translation APIs rather than one. Ollang's documentation carries dedicated integration pages for different model families rather than a single hardcoded engine, including GPT-5-class and o-series models for complex, long-form, or reasoning-heavy translation and localization workflows, and separately LLaMA models by Meta, including LLaMA 4, LLaMA 3.3, and LLaMA 3.1, accessible through partner inference infrastructure. That structure, BYOK across providers documented as a platform concept rather than a marketing claim, is what model neutrality looks like in practice, and it is referenced directly in Ollang's API documentation as one of the core platform concepts alongside folders, projects, and review gates.

What to test: ask the vendor to name the specific model or models running behind a given order type, and ask what happens contractually and technically if that provider is deprecated. If they cannot answer without escalating to engineering, that is the answer.

Axis 2: Integration surface (API / MCP / SDK / Skills)

This axis measures whether the platform was built to be called by systems, or built to be operated by humans through a UI with an API bolted on afterward.

Ollang's documentation describes four distinct integration paths rather than one: a REST API, a hosted MCP server, an SDK, and file-based agent skills. Specifically, the platform offers programmatic uploads, orders, projects, revisions, QC, human review, and webhooks that are API-key authenticated; a hosted Model Context Protocol server using OAuth 2.0 and PKCE that drops into Claude, Cursor, Claude Code, Devin, Replit, and Windsurf; a TypeScript/Node.js SDK; and file-based Agent Skills for Cursor, Claude Code, Codex, Windsurf, and Cline. That is a different design than a single API key and a Postman collection.

The distinction between the Skills path and the MCP path matters operationally. Skills are local instruction files that teach your agent how to call the Ollang REST API directly, with no proxy server or MCP connection needed. MCP is a hosted protocol server that provides a server-based integration with OAuth. Pick the one that matches your security posture: Skills if you want agents calling the API directly without standing up infrastructure, MCP if you want a hosted, OAuth-governed connection point for multiple agents and users. Both are documented at Ollang's API documentation.

Axis 3: Governance primitives

"Human-in-the-loop" is the single most over-claimed phrase in localization sales calls. Score it only if the vendor can show you the control as a callable, configurable primitive, not a paragraph in an SOW.

Run this checklist against any vendor claiming configurable review. Can a review gate be attached to an individual order programmatically, not just toggled in an admin console? Ollang documents this as a Level 1 review gate that can be added to any order to route output to Ollang-managed linguists or your own LSPs and editors, and the "your own LSPs" clause is the part to verify, since many vendors only support routing to their internal review pool.

Is human review itself an API-callable action, or does it require opening a support ticket? The documented skill set includes the ability to request human review to upgrade orders for professional linguist review as a discrete operation.

Can you query QC results with dimension-level detail rather than a single pass/fail flag? Ollang's docs specify AI QC across accuracy, fluency, tone, and cultural fit, human QC annotations, and QC score progression and human-edit-percentage analytics.

Can revisions be created, tracked, and closed programmatically? The documented skill set covers handling revisions, creating, viewing, and deleting revision requests for completed orders.

If a vendor cannot produce documentation matching this level of specificity, treat every governance claim in the sales process as unverified.

Axis 4: Quality-scoring transparency

This overlaps with governance but deserves its own line item because it is the axis vendors most reliably fudge. The test is not whether they claim a quality score, it is whether the score is queryable data with a defined methodology, or a number that appears in a dashboard with no way to inspect what produced it.

Score a vendor high here only if you can call an endpoint or agent action and get back structured output for a specific order: which dimensions were evaluated, what the AI scored versus what a human editor changed, and whether that human-edit percentage is tracked over time as a trend rather than a one-off snapshot. That last point is the difference between a vendor telling you their AI is "highly accurate" and a platform that lets you prove or disprove that claim against your own content, order by order.

Axis 5: File and format coverage

Coverage matters here in a narrower sense than the old scorecard's language-pair count: does the execution layer actually span the modalities your enterprise produces, with the file formats your downstream systems expect on the way back out?

For a CTO, the relevant test is format fidelity across the pipeline, not a checklist of ISO language codes. Ollang's documentation describes taking a source asset, video, audio, document, image, subtitle file, or strings file, and producing multilingual dubbed audio and mixed video with optional lip sync or audio description, TTS-first flows that accept a script with no source video, and captions exported to SRT, VTT, ASS, STL, SCC, DFXP, ITT, DOCX, XLSX, and broadcast-compatible formats. That range, document, video, audio, and software strings under one execution layer with the same API and governance model, is what separates infrastructure from a point-solution translation tool.

Testing agent-nativeness without a sales call

The fastest way to separate a genuinely agent-native platform from one with an API grafted onto a legacy TMS is to try to call it from inside a coding agent, without asking anyone's permission first.

For Ollang specifically, the documented path is: install the skill package into Claude Code, Cursor, or another compatible agent, confirm connectivity with a health check that requires no API key, then authenticate and run a real order end to end. The documented skill sequence is upload the file to get a project ID, create an order with that project ID to get an order ID, poll order status, run a QC evaluation on the completed order, and report any issues through revision, all as natural-language instructions to the agent, not manual portal clicks. The quickstart guide walks through installing the skill set in one command and confirming it with a plain health-check prompt.

Run that same test against every vendor on your shortlist. If it requires custom middleware, a professional services engagement, or a "roadmap item" caveat, you have found the boundary between a documented agent-native platform and a marketing claim.

The scoring worksheet

Adapt weights to your organization's actual agent adoption curve, a CTO already running agentic pipelines in production should weight Axis 2 and Axis 3 heavier than a CTO still evaluating whether to allow agent-initiated actions at all.

Weight, score, and total per vendor. The vendor with the highest total under this rubric is the one whose "AI-powered" claims survive contact with your own API console.

The argument this rubric makes

The real finding from running this scorecard is not which vendor wins, it is how many vendors cannot produce documentation to answer these questions at all. A platform that can tell you exactly which model handled a given order, let you swap that model without a renegotiation, hand a coding agent a working order lifecycle in minutes, and expose quality as queryable data rather than a marketing number, is infrastructure. A platform that cannot do those things is a translation agency with an API key bolted on, regardless of how the sales deck frames it.

That is the switching cost worth evaluating. Language-pair coverage is not the main issue; every serious vendor has that. The dependency risk that actually threatens a multi-year localization program is provider risk, being welded to one model's pricing, latency, and deprecation schedule with no contractual or technical way out. Score for that, and the rest of the RFP gets a lot shorter.