How to score an 'AI execution layer' vendor: a CTO's evaluation framework

Every localization vendor on your shortlist now claims agent compatibility. The RFPs look identical: "MCP support," "API-first," "AI-native." Strip away the language and half of them mean "we bolted a wrapper onto our existing TMS so a chatbot can technically call it." The other half have built for programmatic access as the primary interface, with the dashboard as an afterthought. From the outside, in a sales demo, these two vendors look the same. In production, six months in, they behave completely differently. One scales with your agent fleet; the other becomes a bottleneck you have to route around.

The problem isn't a lack of vendor claims. It's a lack of a rubric that forces those claims to be tested rather than taken on faith. This article gives you one: five scoring criteria, a method for testing each, and a scorecard you can fill out in the room during a vendor call.

Why "we support MCP" isn't the test

MCP has become table stakes fast. The Model Context Protocol, introduced in November 2024 and transferred to the Agentic AI Foundation under the Linux Foundation in December 2025, has reached 97 million monthly SDK downloads and adoption across all major AI providers within sixteen months of launch. Every serious vendor will now say yes when you ask if they support it. That's why the question is worthless on its own; it filters out nobody.

What matters is what sits behind the MCP server. The specification addresses connectivity; it does not address governance. Enterprise architects deploying agents in regulated, mission-critical environments face a structural gap: no architectural guidance exists for permission enforcement, risk-tiered execution, or audit trail requirements at the MCP protocol layer. A vendor can wrap three or four REST endpoints in an MCP server over a weekend and technically be compliant. Whether that server exposes real workflow depth, honors scoped permissions, and produces an auditable trail is a separate question entirely. It determines whether you can trust an agent to act unsupervised inside your localization pipeline.

The five criteria

1. Integration surface breadth

Score how many distinct, production-grade ways a system can be invoked, not how many are listed on a features page. A mature execution layer typically exposes a REST API for pipeline integration, an MCP server for agent-driven workflows, an SDK for engineering teams, and a set of reusable agent actions (skills) that plug into tools like Claude Code, Cursor, or Devin without custom glue code. Ollang's documentation, for instance, orchestrates AI dubbing, subtitle translation, captions, transcription, document and visual translation, and human review workflows in one place, and exposes everything through APIs, an MCP server, an SDK, and agent Skills, covering multiple ways to integrate: a short tour, REST API, MCP, SDK, Skills, and workflows. Score a vendor low if "integration" means a single webhook and a CSV export.

2. Modality coverage

Does the same programmatic surface handle text, documents, websites, software strings, video, dubbing, and audio, or does each modality require a different tool, a different vendor, or a manual handoff? Getting to 240+ languages means stitching together 5+ APIs, managing file conversions, handling dubbing, subtitles, and i18n files separately, all while keeping quality consistent. This criterion is designed to catch that failure mode. A true execution layer takes a single asset type, a video, audio file, document, image, subtitle file, or strings file, and routes it through the appropriate pipeline without you having to orchestrate five vendor relationships to cover one product launch.

3. Governance and audit controls

Programmatic access without governance is a liability, not infrastructure. Score whether the vendor supports scoped API keys, role-based routing of content to human reviewers, approval gates before publishing, and full traceability of what an agent changed and when. This matters more, not less, as agents get more autonomy. A permission manifest architecture embeds role-based access control (RBAC), rate limiting, and scope constraints into MCP server registration, making safety a protocol-level property rather than an application-level afterthought. Ask the vendor directly where in their MCP or API layer this lives. If the answer is "our support team can set that up for you," it is a manual promise, not protocol-level.

4. Quality-tiering logic

Not every string needs the same quality bar. A CTO evaluating infrastructure should ask whether the platform lets you programmatically set quality tiers, such as machine-only for low-stakes internal content, AI-plus-human-review for customer-facing material, and full expert review for regulated or legal content, and whether that tier can be specified as a parameter in the API call itself, rather than configured once in a dashboard. Score vendors on whether quality is a callable variable or a fixed service level baked into a contract.

5. Agent-nativeness

This is the criterion most vendors fail quietly. Agent-nativeness means an autonomous system, not a human clicking through a UI, can discover available actions, invoke them with structured parameters, and receive a structured, machine-readable result back, without a human in the loop translating intent into clicks. MCP clients run within AI applications like Claude, ChatGPT, or enterprise AI platforms. MCP servers expose specific capabilities: tools that can execute actions, resources that provide data, and prompts that offer reusable templates. The protocol handles communication between them in a standardized way. The test is whether the vendor's server exposes genuine tools and resources an agent can reason about. For example, does it let an agent adapt localized strings using project context, custom instructions, and terminology memory to localize JSON/i18n content with the right regional tone, idioms, and cultural nuance, instead of translating text in isolation? Or does it expose a thin shim over three legacy endpoints that still assumes a human is watching the screen.

Testing the claims: demand the workflow, not the deck

Every vendor can produce a slide with a green checkmark next to "MCP-compatible." Almost none of them will volunteer a live demo where an agent, not a sales engineer, initiates a request, the system routes it through quality tiering and governance, and a structured result comes back with an audit trail attached.

Make that demo the gate. Specifically:

• Ask the vendor to connect their MCP server to an agent you control (Claude Code, Cursor, or your own orchestration layer) live, not pre-recorded.
• Give the agent a real task: localize a strings file into three languages with a defined quality tier, and route one output through human review.
• Watch what happens when something goes wrong, such as a malformed file, an unsupported language pair, or a permission boundary. A wrapper vendor's system will often surface a generic error or silently fail. Infrastructure built for agents surfaces a structured, actionable error the agent or your engineer can act on immediately.
• Ask to see the audit log after the run. If it doesn't exist, or it's a support ticket history rather than a queryable record, the governance criterion fails regardless of what the sales deck claimed.

This single exercise separates the two categories of vendor faster than any RFP question can.

Ticket-based vs. callable infrastructure: a procurement distinction

Traditional TMS and agency relationships are structured around tickets: a project manager receives a request, assigns it, tracks it through a workflow, and delivers a file back on a timeline measured in days. Even when that system exposes an API, the underlying operating model is still project-based. Someone has to open the ticket and someone has to close it.

A callable-infrastructure relationship inverts this. Localization becomes a function your systems or agents call the way they call any other internal service. You ship translated content through APIs, MCP, automation, and CI/CD pipelines. Localization runs inside the deployment workflow you already use, rather than sitting downstream of it. Procurement implications follow directly: pricing should scale with programmatic usage rather than per-project quotes, SLAs should be defined in terms of API latency and throughput rather than "business days," and vendor onboarding should be measured in API key provisioning, not vendor kickoff calls. If your procurement team is still negotiating per-project statements of work with a vendor that claims to be infrastructure, that is a signal the claim doesn't match the operating model.

Sample scorecard

Score each criterion 0-3 (0 = absent, 1 = claimed but unverified, 2 = demonstrated in a live test, 3 = demonstrated with governance and audit evidence attached).

A vendor scoring above 12 has architecture that matches its marketing. A vendor scoring below 8 has a wrapper, and you should price your evaluation of them accordingly, as a service relationship with an API bolted on, not as infrastructure.

The argument, restated

The distinction between a TMS with an MCP wrapper and infrastructure built for programmatic access from day one won't show up in a feature comparison. It shows up in what happens when an agent, not a person, tries to get real work done through the system unsupervised: do permissions hold, is the quality tier respected, does the audit trail exist, and does the result come back in a form another system can act on. That's the test a CTO should run before signing, not after discovering the gap in production. Localization is moving from something you commission project by project to something your systems call the way they call any other API. The vendors worth evaluating are the ones whose architecture already assumes that shift happened.