How to Evaluate Coding Agents With Harbor Framework, Python, Docker

AI coding agents can write a working FastAPI endpoint in seconds, refactor a module, or track down a bug across a few files. Anyone who's used Cursor, Claude Code, or a similar tool has felt how genuinely useful that is. What's much less settled is a harder question: how capable are these things, actually, and where do they reliably fail?

That's not a rhetorical question. Coding agents pass some benchmarks impressively and stumble on tasks that, at a glance, look easier than ones they've already solved. They can produce code that passes a quick visual check but fails on an edge case a careful engineer would have caught. And critically, an agent under evaluation has an incentive problem a human engineer mostly doesn't: if the way it's graded is visible or guessable, it can satisfy the grading without actually doing the work. For example, it can pass the test without fixing the bug, or produce output that looks right without being right. Knowing whether an agent is actually good at something, not just good at appearing to be, is genuinely hard, and it's why a serious discipline of AI agent evaluation exists. Benchmarks like SWE-bench and Terminal-Bench, and the harnesses that run them, are built to solve exactly this problem.

This article is about that problem, worked from the ground up. You don't need to know anything about evaluation frameworks to start. You just need to have written software before and, at some point, asked yourself, "How would I actually know if this worked?" We'll reason through what a rigorous test of an AI agent actually requires, then look at a real open-source framework (Harbor) that implements it, and finally build two real tasks with it: a simple one from nothing, and a harder one against an existing project with a real bug in it.

The Problem, Before the Tool

How do you test whether an AI coding agent can actually do a task? Think about what that requires as an engineer before any AI-specific tooling enters the picture. 

First, you need somewhere for it to work that isn't your actual machine. If you're going to let something, human or AI, run arbitrary commands and edit files, you want to contain it. You also want every attempt to start from the exact same clean state, or your comparisons are meaningless. This is just... a container. Nothing novel yet.

Second, you need to tell it what to do, in plain language, without accidentally telling it how. This sounds obvious, but it's the part people get wrong first. If your task description says "open the file, find the loop on line 12, change <= to <," you haven't tested whether the agent can debug anything. What you have tested is whether it can follow a numbered list. The instruction has to describe the symptom, the way a real bug report does, not the fix.

Third, you need a way to check whether it succeeded, and that check cannot be visible to the thing being tested. Think about why. If the agent can read your test file, it doesn't need to solve the problem since it can just read the assertions and produce output that satisfies them directly, or edit the test itself so it passes trivially. A test the subject can see stops being a test of ability and becomes a test of "can you find the answer key." This one constraint is the single most important idea in this entire article, and everything below follows from it.

That's it. Isolated environment, plain-language task, hidden check. If you understand why those three things are necessary, you already understand Harbor conceptually.  Everything else is just how one specific framework implements them.

Harbor Is One Implementation of That Idea

Harbor is an open-source framework, built by the team behind the Terminal-Bench benchmark, for running exactly this pattern at scale: put an AI agent in a container, give it a task, and grade the result with a check it never sees.

Once you have the three-part model above in your head, Harbor's file layout stops looking arbitrary:

my-task/
├── instruction.md      # the plain-language ask
├── task.toml             # metadata/config
├── environment/
│   └── Dockerfile          # the isolated place the agent works
├── solution/
│   └── solve.sh              # a reference answer (kept hidden)
└── tests/
    ├── test.sh                  # the hidden check, runs after the agent
    └── test_outputs.py           # what the check actually verifies

Map this back to the three requirements. environment/ is the isolated container. instruction.md is the plain-language task. tests/ is the hidden check. Notice that the tests/ is a sibling folder to environment/, not something nested inside it. That's not a style choice; it's the whole point: tests/ never gets copied into the container the agent works in. The agent can't read what it's being graded on, for the same reason a student can't see the answer key during an exam.

solution/ is new — we haven't justified it yet. It holds a reference fix: what a correct answer actually looks like. Why does that need to exist if the agent is going to produce its own answer anyway? Because before you trust a test, you need to know two things about it: does a correct answer pass it, and does a wrong answer fail it. You can't know either of those without something to run through the test first. That's what solution/ is for. It is not for the agent, but for you, the task author, to validate your own work.

The Lifecycle, Still Without Writing Any Code

Harbor calls one agent's attempt at one task a trial. When you want to sanity-check a task you've built, rather than pointing a real (slow, costly) AI agent at it, Harbor gives you an Oracle agent. It does no reasoning at all; it just runs your solution/solve.sh directly and submits whatever that produces. If Oracle doesn't pass your own test, your task is broken, full stop, before any real agent has spent a single second on it.

There's a mirror image of this you'll want to, once we get further in: a NOP agent, which submits the environment completely untouched. If NOP also passes your test, that tells you something important: your test isn't actually checking anything. A check that accepts a wrong answer isn't a check.

Oracle should pass, NOP should fail. We will come back to it as the real discipline behind building a task well, not just building one that runs.

The result of a trial is a reward, usually just 1 for pass or 0 for fail, written by your test script to a specific file Harbor reads.

Build One From Scratch: A Simple FastAPI Endpoint

Let's make all of this concrete with something small enough to hold in your head at once. The task: build a FastAPI endpoint, POST /add, that takes {"a": 2, "b": 3} and returns {"result": 5}.

The instruction — plain language, states the contract, not the implementation:

# Build a FastAPI "Add" Endpoint

Create a FastAPI application at `/app/main.py` that exposes a single
endpoint:

- Method: `POST`, Path: `/add`
- Request body: `{"a": <number>, "b": <number>}`
- Response body: `{"result": <number>}`, where `result = a + b`

The app must be importable as `main:app`. Accept both integers and
floats. If a field is missing or non-numeric, a `422` is fine (FastAPI's
default validation behavior handles this).

The environment — just enough installed for the agent to do the work, deliberately not including a working solution: 

FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn pytest httpx

The reference solution, which only you and Oracle ever see: 

#!/bin/bash
cat > /app/main.py << 'EOF'
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class AddRequest(BaseModel):
    a: float
    b: float

@app.post("/add")
def add(payload: AddRequest):
    return {"result": payload.a + payload.b}
EOF

The hidden test, using FastAPI's own TestClient to hit the endpoint the way a real caller would: 

from fastapi.testclient import TestClient
from main import app

client = TestClient(app)

def test_add_integers():
    resp = client.post("/add", json={"a": 2, "b": 3})
    assert resp.status_code == 200
    assert resp.json() == {"result": 5}

def test_missing_field_returns_422():
    resp = client.post("/add", json={"a": 2})
    assert resp.status_code == 422

The verifier entrypoint, which runs after the agent is done, and translates pytest's result into the file Harbor actually reads: 

#!/bin/bash
cp "$(dirname "$0")/test_outputs.py" /app/test_outputs.py
cd /app
pytest test_outputs.py -v
if [ $? -eq 0 ]; then echo 1 > /logs/verifier/reward.txt
else echo 0 > /logs/verifier/reward.txt
fi

Two details worth stopping on, because they're easy to get backward the first time. Harbor doesn't look at pytest's exit code directly. Instead, it reads whatever's in /logs/verifier/reward.txt, so your script has to write that explicitly; skip it, and you get an error even if your tests ran fine. And don't put set -e at the top of that script. The reason is that if pytest fails, set -e would kill the script on that line, before it ever reaches the part that writes the reward file, turning a clean "fail" into a confusing error instead.

Run it:

harbor trials start -p my-task --agent oracle   # should be reward: 1.0
harbor trials start -p my-task --agent nop      # should be reward: 0.0

If both of those come back the way you expect, you've built a working task, and you've proven it's working, not just assumed so.

Leveling Up: Debugging an Existing Project

The build-from-scratch pattern above teaches the mechanics well, but it's not the most common real-world use case. More often, you're not asking an agent to write something new. Many times you're asking it to find and fix a problem in something that already exists and mostly works. That's a meaningfully different kind of task, and it's the one I'd actually recommend practicing next, on a real project.

I built one of these against a small open-source FastAPI currency converter — full code at https://github.com/iyanuashiri/frankfurter-http-error so you can clone it, run the task yourself, and see every file mentioned below in context.

The project itself is a normal small service: users authenticate with an API key, spend "credits" on requests, and the service proxies out to a public exchange-rate API (Frankfurter) for live rates. It already has a real test suite — six files covering auth, conversions, currency listing, historical rates, and history — none of it written with Harbor in mind. That matters: the task doesn't require a specially-crafted project; it works against ordinary application code.

The task folder wraps that project without modifying its shape:

frankfurter-http-error/
├── instruction.md
├── task.toml
├── environment/
│   ├── Dockerfile
│   └── backend/            ← the actual FastAPI project, untouched
│       ├── app/
│       ├── frankfurter/
│       │   └── rest_adapter.py   ← where the bug lives
│       ├── tests/                ← the project's own existing suite
│       └── pyproject.toml
├── solution/
│   └── solve.sh
└── tests/
    ├── test.sh
    └── test_outputs.py

The bug is one character in the HTTP client the app uses to talk to the external rate API: 

# environment/backend/frankfurter/rest_adapter.py
if response.status > 400:   # should be: >= 400
    error_text = await response.text()
    raise HTTPException(status_code=response.status, detail=f"API request failed: {error_text}")
return await response.json()

A 400 Bad Request from the external API — a real, common response when an invalid currency code is requested — no longer counts as an error. > 400 lets it fall straight through to return await response.json(), handing the caller whatever error payload the external API sent back, as if it were a valid exchange rate. Nothing crashes. The bug corrupts data silently, which is a more realistic and more dangerous failure mode than one that throws — and a good one for testing whether an agent actually reasons about the boundary instead of skimming the condition.

The instruction stays a symptom report, matching the actual file shipped with the task: 

When a request is made with an invalid or unsupported currency code, the external 
Frankfurter API returns an error response. Instead of surfacing that as a clear error 
to our users, something downstream is returning malformed or unexpected data without 
raising any exception. Investigate /app/frankfurter/rest_adapter.py, specifically how 
HTTP error responses from the external API are detected and handled.

No mention of 400 specifically, no mention of the operator. The agent has to trace the request path and find the exact boundary itself.

harbor trials start -p . --agent oracle   # reward: 1.0, expected
harbor trials start -p . --agent nop      # reward: 1.0, NOT expected

NOP scoring 1.0 on a task where the bug was never fixed is the clearest possible signal a test suite is hollow. The fix was adding the exact boundary case: 

@pytest.mark.asyncio
async def test_get_raises_exactly_at_400_boundary():
    """Status exactly 400 (the boundary the >= comparison must include) must raise."""
    adapter = RestAdapter()
    url = f"https://{adapter.hostname}/{adapter.ver}/latest?base=XXX&symbols=USD"
    with aioresponses() as mocked:
        mocked.get(url, status=400, payload={"message": "bad request"})
        with pytest.raises(HTTPException) as exc_info:
            await adapter.get(url)
    assert exc_info.value.status_code == 400

Along with a companion test one status code below the boundary (399, which must succeed), so the suite proves both sides of the >= — not just that something raises somewhere above 400. After that change, NOP correctly dropped to reward: 0.0.

That's really the whole lesson of debugging-task authoring, distilled: a held-out test only counts if it fails on the buggy version and passes on the fixed one. If it would pass either way, it isn't a test, it's decoration. Prove this for every task you build — Oracle passes, NOP fails — the same way you'd want to prove it for the build-from- scratch example earlier. Neither check alone tells you the task is good; you need both.

What to Actually Try

If you've followed this far, you understand the whole shape of the problem, not just Harbor's specific commands: isolate the environment, describe the symptom without describing the fix, hide the check from the thing being checked, and prove your check can fail before you trust that it can pass.

Clone the repo, run the credits task yourself, then try seeding a different bug into the same project and building your own task around it. The framework mechanics are secondary. The discipline of "can I prove this task actually measures what I think it measures" is the actual skill, and it transfers to any evaluation framework you'll encounter after this one.