Harness Engineering for AI: Why the Model Is Only Half the System

The discipline of building what surrounds the model, so AI can operate safely in production.

The Problem Nobody Puts on the Roadmap

Every AI project starts the same way. Someone wires up a call to an LLM, the demo works, and the room gets excited. Then it goes to production, and within a week:

• It hallucinates a hotel that doesn't exist.
• It quotes a price in the wrong currency.
• It answers a question it was explicitly told not to touch.
• Nobody can explain why it did what it did, because nothing was logged.

None of this is a "the model isn't good enough" problem. GPT-4-class and Claude-class models are extraordinarily capable. The gap is almost always in everything built around the model — the part that decides what the model gets to see, what it's allowed to do, whether its output can be trusted, and what happens when it's wrong.

That surrounding system has a name: harness engineering.

What Is Harness Engineering?

Harness engineering is the discipline of designing the infrastructure, constraints, tools, memory, verification, and feedback loops that let an AI system operate safely, reliably, and autonomously in production.

The analogy I keep coming back to is that a raw LLM is a Formula 1 engine. Enormous power, zero judgment. Bolt that engine into a chassis with no brakes, no steering, and no telemetry, and you don't have a race car — you have a liability. The harness is the chassis, brakes, steering, and dashboard combined. It's what turns raw intelligence into a dependable product.

Concretely, a harness is made of six layers:

Here's the same six layers as a pipeline:

The rest of this post builds that pipeline for real, using a scenario straight out of the box: a user asking an AI agent to find hotels in Paris under a fixed nightly budget.

Tools Used

• Python 3.11
• LangGraph – to model the harness as an explicit state graph rather than a single prompt
• LangChain – for the LLM wrapper and tool-calling utilities
• Pydantic – to make guardrail and verification checks type-safe instead of string-matched
• A structured logger (standard logging, swappable for LangSmith/OpenTelemetry in production)

Nothing exotic. That's the point — harness engineering is mostly disciplined software engineering applied to a non-deterministic component.

Building the Harness: A Trip-Planning Agent

The scenario: a user asks the agent, "Find me the best hotels in Paris under ₹10,000 per night." We'll build this as a LangGraph StateGraph, where every node is one layer of the harness. That's a deliberate choice — a raw prompt chain hides the harness; a graph makes every constraint, check, and loop a first-class, testable node.

1. Define the Shared State

Every node in the graph reads from and writes to one typed state object. This is the contract that keeps the graph honest — no node can silently mutate something another node depends on.

from typing import TypedDict, Optional
from pydantic import BaseModel

class HotelOption(BaseModel):
    name: str
    price_per_night_inr: float
    rating: float
    source: str  # which API/tool returned this

class TripPlanState(TypedDict):
    user_request: str
    user_id: str

    # Context & Memory
    user_profile: dict
    past_trips: list[dict]

    # Guardrails
    budget_inr: Optional[float]
    guardrail_violation: Optional[str]

    # Tools
    tool_results: list[HotelOption]

    # Model output
    draft_response: str

    # Verification
    verified: bool
    verification_notes: list[str]

    # Final
    final_response: str

2. Context and Memory Layer

Before the model sees anything, the harness decides what it's allowed to see. Here we pull the user's saved preferences and past trip history from a store (a vector DB or a plain Postgres row — the interface matters more than the backend).

def load_context(state: TripPlanState) -> TripPlanState:
    profile = user_store.get_profile(state["user_id"])
    past_trips = trip_store.get_recent(state["user_id"], limit=5)

    return {
        **state,
        "user_profile": profile,
        "past_trips": past_trips,
    }

This is a single-responsibility node: it fetches context and nothing else. It doesn't call the LLM, doesn't validate anything, doesn't touch tools. That separation is what makes the graph testable — you can unit-test load_context with a fake user_store and never touch an LLM.

3. Guardrails and Constraints Layer

This layer runs before the model generates anything expensive, and it fails fast if a precondition isn't met — no fallback, no silent guessing at the budget.

import re

class GuardrailViolation(Exception):
    pass

def apply_guardrails(state: TripPlanState) -> TripPlanState:
    match = re.search(r"under\s*₹?([\d,]+)", state["user_request"])
    if not match:
        raise GuardrailViolation(
            "No budget detected in request — refusing to proceed without a constraint."
        )

    budget = float(match.group(1).replace(",", ""))

    if budget <= 0:
        raise GuardrailViolation("Budget must be a positive number.")

    if "paris" not in state["user_request"].lower():
        raise GuardrailViolation("Destination outside supported scope for this agent.")

    return {**state, "budget_inr": budget}

Note what this is not doing: it isn't asking the LLM to "please respect the budget" in a system prompt and hoping. The budget is extracted and validated in code, before the model is in the loop at all. Prompts are guidance; guardrails are enforcement.

4. Tools and Integrations Layer

The model doesn't know real-time hotel prices — nor should it guess them. This node calls a real API and hands the model facts instead of letting it hallucinate them.

def search_hotels(state: TripPlanState) -> TripPlanState:
    raw_results = hotel_api.search(
        city="Paris",
        max_price_inr=state["budget_inr"],
        currency="INR",
    )

    results = [
        HotelOption(
            name=r["name"],
            price_per_night_inr=r["price"],
            rating=r["rating"],
            source="hotel_api_v2",
        )
        for r in raw_results
    ]

    return {**state, "tool_results": results}

5. The AI Engine Node

Only now — with a validated budget and real tool data in hand — does the model get involved. Its job is narrow: turn structured facts into a readable recommendation. It is explicitly not asked to invent prices or hotels.

from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage, HumanMessage

llm = ChatOpenAI(model="gpt-4o", temperature=0.2)

def generate_recommendation(state: TripPlanState) -> TripPlanState:
    hotels_text = "\n".join(
        f"- {h.name}: ₹{h.price_per_night_inr}/night, rated {h.rating}/5"
        for h in state["tool_results"]
    )

    system = SystemMessage(content=(
        "You are a trip-planning assistant. Recommend hotels ONLY from the "
        "list provided below. Never invent a hotel, price, or rating that "
        "is not in the list. If the list is empty, say so plainly."
    ))
    human = HumanMessage(content=(
        f"User request: {state['user_request']}\n\n"
        f"Available hotels (budget ≤ ₹{state['budget_inr']}/night):\n{hotels_text}"
    ))

    response = llm.invoke([system, human])
    return {**state, "draft_response": response.content}

6. Verification and Testing Layer

The model just produced text. The harness doesn't trust it — it checks it. Specifically, this node confirms every hotel name mentioned in the draft actually exists in tool_results, which is the single most common failure mode (hallucinated entities) for this kind of agent.

def verify_output(state: TripPlanState) -> TripPlanState:
    notes = []
    known_names = {h.name.lower() for h in state["tool_results"]}

    mentioned = extract_hotel_names(state["draft_response"])  # simple NER/regex helper
    hallucinated = [name for name in mentioned if name.lower() not in known_names]

    if hallucinated:
        notes.append(f"Hallucinated hotels detected: {hallucinated}")

    over_budget = [
        h for h in state["tool_results"]
        if h.name.lower() in [m.lower() for m in mentioned]
        and h.price_per_night_inr > state["budget_inr"]
    ]
    if over_budget:
        notes.append(f"Budget violation: {[h.name for h in over_budget]}")

    return {
        **state,
        "verified": len(notes) == 0,
        "verification_notes": notes,
    }

7. Feedback Loop Layer

Whether verification passes or fails, the harness logs the outcome back into the user's history. This is what lets the next run start from a better context node — the loop that turns a single interaction into a system that improves.

def record_feedback(state: TripPlanState) -> TripPlanState:
    trip_store.log_interaction(
        user_id=state["user_id"],
        request=state["user_request"],
        response=state["draft_response"],
        verified=state["verified"],
        notes=state["verification_notes"],
    )

    final = (
        state["draft_response"]
        if state["verified"]
        else "I couldn't verify a safe recommendation — please refine your search."
    )
    return {**state, "final_response": final}

8. Wiring the Graph

This is where the harness becomes visible as a structure, not a paragraph of prompt instructions:

from langgraph.graph import StateGraph, END

graph = StateGraph(TripPlanState)

graph.add_node("load_context", load_context)
graph.add_node("apply_guardrails", apply_guardrails)
graph.add_node("search_hotels", search_hotels)
graph.add_node("generate_recommendation", generate_recommendation)
graph.add_node("verify_output", verify_output)
graph.add_node("record_feedback", record_feedback)

graph.set_entry_point("load_context")
graph.add_edge("load_context", "apply_guardrails")
graph.add_edge("apply_guardrails", "search_hotels")
graph.add_edge("search_hotels", "generate_recommendation")
graph.add_edge("generate_recommendation", "verify_output")
graph.add_edge("verify_output", "record_feedback")
graph.add_edge("record_feedback", END)

trip_agent = graph.compile()

9. Observability, Wrapping the Whole Thing

Observability isn't a node in the graph — it's a cross-cutting concern that watches every node. The simplest version is structured logging at each transition; in production this is where you'd plug in LangSmith, OpenTelemetry, or your APM of choice.

import logging, time, functools

logger = logging.getLogger("harness")

def observed(node_fn):
    @functools.wraps(node_fn)
    def wrapper(state):
        start = time.monotonic()
        try:
            result = node_fn(state)
            logger.info("node=%s status=ok duration_ms=%.1f",
                        node_fn.__name__, (time.monotonic() - start) * 1000)
            return result
        except Exception as exc:
            logger.error("node=%s status=error error=%s", node_fn.__name__, exc)
            raise
    return wrapper

Wrap every node with @observed before adding it to the graph, and you get per-node latency, error rate, and a full trace of what fired for a given request — without touching the business logic inside each node.

Running It

result = trip_agent.invoke({
    "user_request": "Find me the best hotels in Paris under ₹10,000 per night",
    "user_id": "user_9231",
})

print(result["final_response"])

Trace of what actually happens, layer by layer:

1. Context and memory – loads the user's saved preference for boutique hotels from a past trip.
2. Guardrails – extracts budget_inr=10000, confirms Paris is in scope, fails fast if either is missing.
3. Tools – calls the real hotel API, gets back four options under budget.
4. AI Engine – drafts a recommendation using only those four hotels.
5. Verification – confirms every hotel named in the draft exists in the tool results and is within budget.
6. Feedback – logs the interaction, returns the verified response (or a safe fallback if verification failed).

Every one of the frustrations from the intro — hallucinated hotels, wrong currency, no explanation — is closed by a specific layer, not by a bigger prompt.

Why This Matters Beyond One Demo

This is also why harness engineering, not model selection, is where most of the engineering effort in an AI product actually goes. Swapping GPT-4o for Claude or Gemini in the generate_recommendation node above is a one-line change. Building the context, guardrail, tool, verification, and feedback layers around it is the real project.

TL;DR: Harness Engineering for AI

An LLM alone is just an engine — powerful, but no brakes. Harness Engineering is the system built around it to make it production-safe: memory (what it knows), guardrails (what it can't do), tools (real data, not guesses), verification (catching mistakes before users see them), feedback loops (learning from each run), and observability (knowing what happened).

Example: a hotel-search agent where the budget is validated in code, prices come from a real API, and every answer is checked against that data before it's shown — no guessing, no fallback text.

Bottom line: most of the real engineering effort goes into the harness, not the model. An engine alone doesn't ship — a car does.

If you're building AI systems and only budgeting time for prompt engineering, you're designing an engine and skipping the car.

This post is part one of the My Learning Series. Keep watching this space.