AI Agent Harness Lock-In: 5 Portability Tests to Run Before You Commit

In Q1 2026, three major agent infrastructure platforms dropped in nine weeks: OpenAI Frontier, the AWS Stateful Runtime, and Anthropic's Claude Managed Agents. What happens eighteen months from now when the model we built on gets deprecated, or we need to renegotiate pricing?

I ran each platform through the same evaluation. Most teams I've talked to miss the question that actually matters: will your infrastructure survive a model change? Here's the five-pillar framework I use to find out, with real tests and code for each one.

The Harness Is Not the Model

Most platforms bundle two different things under one product name: the reasoning model and the harness. The model reasons. The harness does everything else that makes it an agent capable of executing multi-step tasks:

• Working and persistent memory – what the agent retains during a task and what carries over between sessions
• Tool execution – registering available tools, intercepting calls before they run, handling failures
• Skills – reusable multi-step behaviors built on top of tools
• Orchestration – coordinating multiple agents, routing subtasks, collecting results
• Governance – access controls, human-approval gates, audit trails

When the harness is tightly coupled to one model, you've made an architectural bet. That's not necessarily wrong, but you should make it on purpose.

Five Portability Tests

Run all five before you commit.

1. Memory: Who Does It Belong To?

Run a two-session test. Session one: have the agent learn something concrete. A service that should always be read-only. A user preference. An entity it should recognize. Kill the session. Start a fresh one and ask whether it remembers.

Then export the raw memory store:

# Export memory from your agent platform 
agent-cli memory export --session-id=<id> --output memory_dump.json 
# What you want to see: plain readable JSON 
cat memory_dump.json

Portable memory looks like this:

{
  "entity": "postgres_prod",
  "type": "database",
  "note": "Read-only account. Never write.",
  "created": "2026-04-10T09:14:22Z",
  "tags": ["production", "restricted"]
}

Locked memory looks like this:

{
  "_type": "openai.memory.EphemeralObject",
  "_model_ref": "gpt-5.4-turbo-internal",
  "_blob": "AQIDBAUGBwgJCgsMDQ4PEBESExQVFhcY..."
}

If you need the provider SDK to deserialize it, it belongs to them. Persistent memory that survives a migration must be readable in a text editor without any provider dependency. If it's opaque binary or references internal provider objects, it won't survive.

2. Tools: MCP or Provider Subclass?

Check how your tools are defined. MCP is now the de facto standard for provider-neutral tool registration. A tool defined in MCP works with any harness that implements the protocol.

MCP-compatible (portable):

{
  "name": "query_database",
  "description": "Read-only SQL query against the analytics DB.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "timeout_ms": { "type": "integer", "default": 5000 }
    },
    "required": ["query"]
  }
}

Provider-locked (rewrite required when you leave):

# This class dies with your provider contract
class QueryDatabaseTool(openai.BaseTool):
    name = "query_database"
    
    def run(self, query: str) -> str:
        return db.execute(query)

Plain JSON or YAML you can hand to another harness is portable. I've seen teams underestimate the tool-layer rewrite by a factor of three. It compounds fast when you have 30+ tools. For a practical look at how MCP connects agents to any API, the pattern holds across every provider.

3. Skills: Do They Survive a Model Swap?

Skills are where lock-in accumulates without anyone noticing. A skill is a reusable multi-step sequence — something like search-summarize-route or draft-review-send. The problem is subtle: skills get built against one model's output format and response conventions. They work perfectly until they don't.

Run this smoke test against a cheap model before you commit:

SKILLS_TO_TEST = ["search_and_summarize", "draft_review_route", "incident_triage"]

def smoke_test_skill(skill_name, model="gpt-4o-mini"):
    """
    Not checking quality. Checking whether it completes.
    """
    try:
        result = agent.run_skill(
            skill=skill_name,
            model_override=model,
            timeout=30
        )
        print(f"[PASS] {skill_name} on {model}: completed in {result.duration}s")
        return True
    except (ParseError, StepTimeoutError) as e:
        print(f"[LOCKED] {skill_name} on {model}: {e}")
        return False

for skill in SKILLS_TO_TEST:
    smoke_test_skill(skill, model="gpt-4o-mini")
    smoke_test_skill(skill, model="mistral-7b")

If it crashes: locked. If quality drops but it completes: portable. Those are completely different problems.

4. Orchestration: Grep Your Own Codebase

Orchestration is where coupling gets expensive and invisible. If your planning agent parses sub-agent outputs using provider-specific response fields, swapping one model silently breaks everything downstream. The errors show up three layers away from the actual model call.

# Run this against your orchestration layer right now
grep -rn "\.choices\[0\]" ./agents/
grep -rn "\.message\.content" ./agents/
grep -rn "openai\." ./orchestration/
grep -rn "anthropic\." ./orchestration/

If those grep results are long, you're coupled. Portable orchestration parses against schemas you control:

# Locked: parsing a provider response object directly
raw = agent_response.choices[0].message.content
result = json.loads(raw)

# Portable: your schema, not theirs
result = TaskOutput.model_validate(
    parse_task_output(agent_response, schema=SUBTASK_SCHEMA)
)

The fix isn't dramatic, but catching it late means touching a lot of code under pressure. For teams running multi-agent workflows with AWS Step Functions, this schema boundary becomes even more critical when agents span different provider runtimes.

5. Governance: Export and Read It

The global race to govern AI agents has made governance a first-class concern — but most teams still treat it as an afterthought until a compliance conversation forces the issue. Governance covers what tools can be called, what needs human sign-off, and what goes into the audit log. It should live in your harness, not baked into a provider's permission system.

Export it and look at what it references:

agent-cli governance export --format=yaml > governance_config.yaml
cat governance_config.yaml

Portable governance config:

policies:
  - name: restrict_production_writes
    applies_to_roles: [sre_agent, oncall_agent]
    deny:
      actions: [database.write, infrastructure.delete]
      resources: ["prod/*"]
    require_approval:
      - action: infrastructure.restart
        approvers: [oncall-lead]

audit:
  log_all_tool_calls: true
  retention_days: 90

Locked governance config:

openai_platform:
  policy_set_id: ps_abc123xyz
  workspace_id: ws_prod_999
  iam_role_binding: roles/openai.agentOperator
  permission_set: OPENAI_PROD_RESTRICTED

If your config references provider IAM primitives or platform-specific IDs, it stays behind when you leave. You're rebuilding from scratch. The role of AI in IAM is evolving fast — your governance layer needs to be portable enough to keep up.

How the Platforms Actually Scored

OpenAI Frontier/AWS Stateful Runtime

Well-built if you're staying on GPT. Memory in OpenAI infrastructure, tools through OpenAI SDK, orchestration layer assumes GPT conventions. The April 2026 Agents SDK update added workspace portability via Manifest and four memory tiers. Genuine improvements. But the docs say it plainly: designed for OpenAI models. The reference examples all use gpt-5.4. Know what you're signing up for.

Claude Managed Agents

The architecture is interesting. Three independent interfaces: session log, brain layer (Claude), and code execution sandbox. Each one can fail or be replaced without breaking the others. Anthropic published their reasoning: harness code encodes model limitations, and those limitations become technical debt as models improve. They built the interfaces to be swappable. The catch is the brain interface runs Claude. Changing that is a migration, not a config change.

Claude Platform on AWS (GA May 11)

Solves the governance and procurement problem better than anything else I tested. Auth through AWS IAM, billing through AWS Marketplace, audit logs in CloudTrail alongside your existing AWS services. Your governance policies for Claude agents live in a system your security team already knows. What it doesn't solve: data is processed by Anthropic outside the AWS boundary, so no Bedrock regional residency. And it's still Claude. The five tests above don't change.

Open-source (LangChain Deep Agents, Letta, CrewAI, Microsoft Agent Framework)

Model selection is a config variable. Memory in open formats. Tools in MCP. Governance is harness-owned. Less polished, more infrastructure work. That's the honest tradeoff. If you expect the model layer to keep moving, this is where you start. For teams already building compound AI systems for scalable workflows, the open-source stack plugs in naturally.

Checklist Before You Commit

These take a few hours. A migration takes months.

• Export persistent memory, open it in a text editor without a provider SDK
• Check tool definitions: MCP/open schema vs. provider SDK subclass
• Run your top three skills against a non-primary model. Do they complete?
• Grep orchestration code for provider-specific response object references
• Export governance config, check whether it references provider IAM primitives
• Ask: if this provider relationship ends tomorrow, which assets do I actually own?

The platforms that shipped this year solve real problems. If you need production-grade agents fast and you're not switching models anytime soon, the managed options are good. The five tests tell you exactly what you're trading. Run them before you commit, and the decision is deliberate. Skip them, and you'll find out later at a worse time.