Engineering Production Agentic Systems: Part 2: The Guardrails

Tool Surface, Authorization Scopes, and Audit-Trail Engineering

This is Part 2 of a three-part field manual on engineering production agentic systems. Part 1 took context engineering. This part takes guardrails. Part 3 takes human-in-the-loop topology. The conviction underneath all three: production agentic systems are won on these three architectural disciplines — not on model choice.

The Opening Claim

MCP gateways have made tool exposure cheap. Wrapping a hundred internal APIs as agent-callable tools is now a one-day job, sometimes a one-hour job. Tool correctness, though — making sure the agent picks the right tool, with the right parameters, under the right authorization, with the right human review — is not cheap. It is the work nobody talks about in agentic-AI demos, and it is where production systems either hold up or collapse.

In regulated industries, the failure modes are not abstract. A misrouted shipment is one thing; a misposted ledger entry, a fetched wrong-customer record, an inadvertently triggered KYC alert is another. In Part 1, I named two of the four failure modes I keep watching teams hit: tool-authorization sprawl and audit-trail opacity. They are guardrail problems. The pipeline can produce the cleanest possible context, but if the agent then calls the wrong tool with the wrong parameters and no one can trace what happened, the cleanliness of the context did not save you.

This part is about how the guardrails are built. Tool surface as a contract, not a function. Authorization as a runtime decision, not a config file. Audit trail as a first-class artifact, engineered into the pipeline rather than bolted on. Reversibility as a property of the action, not a property of the agent.

The implementation that backs this article lives in the Function Identifier component of the Distribution layer and the metrics/state machinery of the Orchestration layer. Code references and the schema for the audit-trail event model are below.

A Tool Is Not a Function — It Is a Contract

The single biggest mistake teams make when exposing tools to agents is treating a tool the same way they treat an internal API endpoint. An endpoint has a name and a schema. A tool has a name, a schema, and three more fields that determine whether the agent should be allowed to call it in this turn, at this risk class, with this human-review depth.

The five required fields:

1. Name

The agent-visible identifier. Not the underlying API endpoint name. Names should encode intent (what the agent means to accomplish) rather than operation (what the wrapped API happens to do). reroute_container is an intent. POST /shipments/{id}/route is an operation. Agents reason better over intents because intents are how humans describe goals; operations are how systems describe implementations.

2. Schema

The parameter shape. Typed and bounded where possible. Enums beat freeform strings every time. A reason_code: enum field constrains the agent to a fixed vocabulary the downstream system can actually act on; a reason: str field invites hallucinated explanations that look plausible and fail at the API boundary.

3. Scope

The role permitted to call this tool. Role-bound, least-privilege by default. The Logistics Analyst role does not get reroute_container because rerouting is the Supply Chain Manager’s decision. Scope is not a runtime filter the agent reasons over; it is a runtime filter the Function Identifier applies before the tool is exposed to the agent at all.

4. Risk Class

What failure costs. Low, medium, high, critical. Drives logging depth, review frequency, and (combined with reversibility) the HIL gate.

5. Reversibility

Can the action be undone? Three classes: reversible (a read or a search), cost-reversible (a notification that costs goodwill to retract, a reroute that costs fuel), irreversible (a delete, a publish, a trade).

The five fields together compose into a sixth — the HIL gate — which determines whether the action runs immediately, requires async notification, requires synchronous human approval, or is disallowed for agent invocation entirely. The HIL gate is not a separate configuration; it is emergent from scope × risk × reversibility. Part 3 takes the HIL topology in depth. This part is about how the contract itself gets enforced.

The Function Identifier as Gate

The contract is enforced by the Function Identifier component of the Distribution layer. Every agent turn, before tools are exposed to the agent at all, the Function Identifier reads each tool’s contract, applies the role policy and the risk policy, and emits three things: the constrained tool surface for this turn, the HIL gates that any tool call must cross, and the pre-trace skeleton the audit-trail event will populate.

This is the seam between the pipeline and the guardrails that I telegraphed in Part 1. The pipeline’s Distribution layer doesn’t just inject context into the prompt — it also injects the tool surface that the agent is allowed to see. The agent never reasons about whether it should be allowed to call delete_shipment_record; that tool simply does not exist in this turn’s tool catalog, because the Function Identifier filtered it out before the agent saw the prompt.

The relevant code from the repo’s context_distribution.py:

# From context_distribution.py — FunctionCallingIntegration
def identify_required_functions(self, context, available_functions):
    function_selection_prompt = """
    Based on the following context, determine which functions (if any)
    should be called. For each, provide the appropriate parameters.

    Context:        {context}
    Available functions: {functions}

    Respond with a JSON array of function calls.
    If no functions should be called, respond with []
    """
    # Tool surface (`available_functions`) is already filtered upstream
    # by role + risk policies before reaching this method.
    return self.llm.predict(prompt.format(...))

Two things to defend about this design.

Tool filtering happens before the LLM sees the prompt. The available_functions parameter passed into this method is not the full catalog — it is already the constrained surface computed by the upstream filter. The LLM’s job is to choose among allowed tools, not to decide which tools should be allowed. This separation matters because LLMs are bad at policy enforcement (they can be coaxed) and good at choice among constrained options. Put policy enforcement where it can’t be coaxed.

Tools are constrained per turn, not per session. The same role-bound user might see different tool surfaces at different process steps. A Supply Chain Manager doing Initial Assessment doesn’t get reroute_container — that lives at Resolution Planning. The Function Identifier reads the current process step and adjusts the surface accordingly. Static per-role tool catalogs are a tell for a guardrails system that hasn’t been instrumented for workflow yet.

Naming as the First Guardrail

The single cheapest guardrail intervention is naming. Most teams underestimate this because naming feels like a documentation choice rather than a runtime control. It is both.

Names that read like API endpoints invite the agent to reason about how to call them rather than whether to call them. POST_orders_status_update invites parameter speculation. update_order_status is better. mark_order_resolved is better still, because it commits to an intent the agent can only invoke when the intent actually applies.

The discipline holds across the tool catalog. Intent verbs beat operation verbs. notify_customer_of_delay is an intent; send_email is an operation. escalate_to_manager is an intent; create_jira_ticket is an operation. The intent name carries enough constraint that the agent’s miss-selection rate drops sharply — in my own production work, roughly an order of magnitude — relative to operation-named tools.

There is one anti-pattern that needs to be named because it is so common: tools that take a freeform command or action parameter. The temptation is to expose a single execute_shipment_action tool that takes a command: str field, on the grounds that this is more flexible than exposing twelve narrow tools. It is more flexible. It is also a guardrail liability. The agent will fill the freeform field with hallucinated values; the schema cannot reject them at the contract layer; the only thing that can catch the bad command is downstream system rejection, by which point the audit trail is already polluted with a half-formed action. Always prefer twelve narrow tools to one wide one.

Authorization Scopes

Authorization at the tool layer is the runtime enforcement of the scope field in the tool contract. Two questions every tool call must answer at runtime: Is this caller in scope for this tool? and Are this caller’s specific parameters within their scope on this tool?

The first question is straightforward: the Function Identifier checks the caller’s role against the tool’s scope field at filter time. If the role isn’t permitted, the tool isn’t in the surface; the agent never sees it.

The second question is the harder one. A Supply Chain Manager is in scope for reroute_container in general, but which containers? Their own customers’ containers, certainly. A peer manager’s containers? Almost certainly not. The schema field container_id: str cannot enforce this on its own — the type system says the parameter is a string, not that the caller has authority over the specific container the string identifies. The enforcement happens at the call boundary, not the contract boundary.

The pattern that works in production: every tool call passes through a parameter-scope check between contract validation and underlying API invocation. The check is policy-driven, queries an authorization service (typically the same one the human UI uses), and returns either allowed or denied with reason. Denied calls become audit-trail events the same as allowed ones — Part 5 below — but they never reach the underlying system.

# Pattern, not from repo — runtime parameter-scope check at the call boundary
def invoke_tool(tool_name: str, caller: Principal, params: dict) -> Result:
    contract = catalog.get(tool_name)
    contract.schema.validate(params)                        # contract check
    decision = authz.check(caller, contract, params)        # parameter scope check
    audit.emit(
        tool=tool_name,
        caller=caller,
        params=params,
        decision=decision
    )
    if decision.denied:
        raise Forbidden(decision.reason)
    return downstream.call(contract.endpoint, params)

The discipline here is to keep the four steps — schema validation, scope check, audit emission, downstream call — as four explicit steps with explicit ordering. It is tempting to fold the scope check into the underlying API’s existing authorization. Don’t. The agent’s authorization context is not the underlying API’s authorization context; you want one place where the agent-specific policy is enforced, and that place lives inside the platform, not inside whatever fifteen-year-old API the platform happens to be calling.

Audit Trail as a First-Class Artifact

Of the four failure modes in Part 1, the one teams most often discover only after a regulatory or compliance event is audit-trail opacity. The agent did something. Someone asks what it did, why, on whose behalf, and with what authorization. The team realizes they have application logs, model traces, and tool-call records — none of which compose into the question the auditor is actually asking.

The fix is to engineer the audit trail as a first-class artifact, emitted by the pipeline itself, with a schema that’s stable across stages and a storage tier that’s append-only and queryable.

Eight required fields per event. trace_id correlates events across one request’s lifecycle. actor identifies who acted — agent, role, or human-in-the-loop approver. stage names the pipeline phase that emitted the event. action describes what was done — a tool call, a prompt injection, an approval gate decision. payload carries the parameters or content (with PII fields hashed for sensitive scopes). approval captures the HIL gate result, if any. outcome records success, error, or denial. prev_event_id maintains the causal chain so the full trace can be reconstructed.

One concrete event, captured at the moment of a tool call:

{
  "event_id": "evt_2026_05_12_a8c4f1",
  "trace_id": "req_SCE_2026_001_b3",
  "timestamp": "2026-05-12T14:22:08.471Z",
  "stage": "distribution.tool_call",
  "actor": {
    "kind": "agent",
    "role": "supply_chain_manager"
  },
  "action": {
    "tool": "reroute_container",
    "risk_class": "medium"
  },
  "payload": {
    "container_id": "MSCU7654321",
    "to_port": "Antwerp"
  },
  "approval": {
    "gate": "human",
    "approver": "user_472",
    "status": "granted"
  },
  "outcome": {
    "status": "success",
    "duration_ms": 1240
  },
  "prev_event_id": "evt_2026_05_12_a8c4f0"
}

Three design choices that matter.

Append-only storage. Audit-trail events are never updated, never deleted. Corrections are themselves events, with a corrects: prev_event_id field. This is the same discipline a financial ledger uses; for the same reason.

Stage names as a fixed taxonomy. The set of valid stage values is the union of pipeline stages and orchestration phases — acquisition.gather, refinement.enrich, distribution.tool_call, orchestration.approval. Auditors and engineers should never see a stage: "misc" event.

Pre-trace before action. Every potentially-emitting action writes a pre-event into the trace before invoking, and updates it with the outcome afterward. If the action never returns (crash, timeout, network partition), the pre-event remains as evidence that the attempt was made. This is what makes the audit trail reliable in the failure modes that matter most.

The audit trail lives in the Orchestration layer’s metrics collector rather than in any individual pipeline layer. That placement is deliberate — audit is cross-cutting, and concentrating its concerns in one place is what lets you query “show me everything that happened in this trace” without joining across four services.

Reversibility Classes and the HIL Gate Matrix

Reversibility is the field that closes the loop between guardrails (this part) and human-in-the-loop topology (Part 3). The combination of risk class and reversibility class determines the depth of human review any tool call must cross.

Reversibility × Risk → Required HIL Approval Depth

The matrix is read by the Function Identifier when computing the HIL gate for any tool call. None means the call runs immediately, logged. Notify means the call runs immediately, with an async notification to a human. Approve means the call blocks on synchronous human approval before running. Dual-approve means two humans must approve. Dual + audit adds a second-line review (compliance, risk, or audit function) on top of dual approval. Disallow means the agent cannot invoke this combination at all — it is a human-only action.

Three observations.

Reversible-and-low-risk is the safe zone. Read operations, search queries, status lookups. The agent runs these freely. This is where most of the agent’s actual work should live in a well-designed system; if you find your tool catalog dominated by cost-reversible and irreversible operations, you have a tool surface design problem, not a HIL design problem.

The diagonal is where the cost lives. As risk and irreversibility both rise, the human-review burden rises with them. This is the right shape — the matrix should make sure that the cheap-to-undo, low-stakes actions move fast, while the expensive-to-undo, high-stakes actions move slowly through proper review. The discipline is encoding the matrix at all, not the cell values you start with.

The values in the matrix are organization-tuned. The matrix I have shown reflects what works in regulated industries with material consequences for irreversible action — supply chain, banking, healthcare. A consumer-facing application might shift the entire matrix one cell to the left. A capital-markets application might shift it one cell to the right. The values are policy; the matrix is architecture.

Regulated-Industry Constraints — Enough to Know They Need a Seat at the Table

Three quick notes on regulated industries, not exhaustive, just enough to make clear the architecture has to leave room for compliance from day one.

Different regulatory regimes touch different parts of the guardrails layer. SOX wants every financial-system action attributable to an approving human; the audit trail’s actor and approval fields are the surface where SOX evidence lives. GDPR and similar data-residency regimes constrain what payload can contain and where it can be stored; the audit trail’s payload-hashing discipline is what makes this manageable. PCI-DSS constrains how and where cardholder data can transit; the Verify stage in the pipeline is where PCI scope is established. The EU AI Act and similar emerging frameworks layer in disclosure, explainability, and reviewability requirements that further shape the event schema.

The trap to avoid is treating compliance as a post-hoc audit layer. Once an agent has acted, you cannot retroactively make the audit trail richer than it was at action time. The architecture has to emit compliance-quality evidence as a side effect of normal operation, not as a periodic export job. This means compliance lives in the guardrails layer — in the contract, in the gate, in the event schema — not as a wrapper around the pipeline.

The good news is that the same architecture that solves the engineering problems in this article also solves most of the compliance problems. The five-field tool contract, the Function Identifier gate, the event schema, and the reversibility matrix together produce the kind of structured evidence regulators ask for. The work is in committing to the discipline; the artifact is largely a by-product.

Closer

Part 1 closed by saying the pipeline is the moat. Part 2 closes by saying the pipeline is the moat and the guardrails are the moat. Both are architectural disciplines that compound: a pipeline without guardrails produces clean context that gets squandered on the wrong tool calls; guardrails without a pipeline produces a tightly-controlled tool surface starved of the context to choose well within it.

The four artifacts of this part — the tool contract, the Function Identifier gate, the audit event schema, the reversibility matrix — are reusable across domains in a way that the pipeline isn’t. The pipeline gets shaped by what data the system has; the guardrails get shaped by what actions the system can take. Different problem spaces.

This pattern is what makes MoJoCo, the agentic modernization platform I have been designing hands-on for eighteen months, hold up under the scrutiny of regulated-industry buyers. The deterministic reverse-engineering tools (ARC, MAM, CAST) sit underneath as the action surface. The pipeline from Part 1 produces reasoning-grade context above them. The guardrails from this part filter, scope, and audit every action the multi-agent layer takes against those tools. The three pieces compose into a system that can defensibly run autonomously in environments where most agentic systems can’t.

One failure mode remains: agent-loop divergence. The pipeline can produce great context; the guardrails can constrain the tool surface to safe choices — but if the agent’s reasoning loop diverges, generating plausible-but-wrong subgoals and burning turns on phantom subtasks, the cleanliness of the upstream pieces did not save you either. Part 3 takes that on: loop bounding, HIL joints as a design vocabulary, termination conditions, and the feedback loop that closes the system.

Production agentic systems are won on context engineering, guardrails, and human-in-the-loop topology — not on model choice. Two of the three are now closed out. The third is the topology that determines when the agent acts at all.

Part 3 — The Topology: loop bounding, HIL joints, termination discipline, and the closing feedback loop — drops next.