The Missing Primitive in Data Platforms: Agent Contracts for Tool Calls

Analytics agents are moving from answering questions to doing things — running SQL, resolving metrics, fetching lineage, creating exports, and triggering workflows. This shift breaks a common assumption in GenAI projects: that production will be fine if the agent’s prompt is good. In reality, once an agent can call tools, you are operating a distributed system whose behavior can drift with every model upgrade, prompt change, routing adjustment, or schema change.

Most teams respond by adding a few guardrails, tuning prompts, or rate-limiting tool access. That helps, but it doesn’t address the failure mode that matters most in data platforms: the same question leading to different tool behavior over time. A small change can turn a safe metric lookup into raw SQL, increasing retries and introducing silent correctness drift without any explicit error. Traditional data platforms solved this problem with data contracts, which consist of SLOs, explicit interfaces, controlled rollouts, and ownership.

Agents need the same discipline, but applied to tool-call behavior. This is not a table schema or an API signature. This article proposes a missing primitive in the data platform: the agent contract. It is a short, enforceable specification per tool that defines success criteria, cost SLOs, golden traces, allowed data, governance boundaries, rollback triggers, canary releases, and retry or loop limits. Prompts can guide behavior, but contracts make behavior testable and stable.

Why Prompts Aren’t Enough

Prompts are necessary, but they are not a control plane. Once an analytics agent can call tools, you inherit failure modes that prompts cannot reliably prevent, especially under change.

• Silent behavior drift: The same question can shift from the semantic layer to raw SQL or from one dataset to another after a routing tweak, model upgrade, or schema change.
• Governance bypass: If one path is blocked, the agent may try another tool or broaden queries to compensate, crossing policy boundaries.
• Retry loops and storms: When a tool fails, agents often retry in multiple ways, increasing load and cost unless hard ceilings are enforced.
• Budget violations: A prompt can ask the agent to be fast, but it cannot enforce concurrency limits, p95 latency targets, or per-tenant budgets.
• Unbounded blast radius: Large queries, exports, and multi-step flows can leak data or trigger expensive workloads quickly.
• Incorrect success criteria: A SQL query executing successfully is not success in analytics. Wrong grain, joins, or timeframes can produce plausible but incorrect answers.

Agent contracts address these issues by moving key constraints out of model instructions and into platform enforcement. The gateway decides what is bounded, allowed, and safe to release.

What Is an Agent Contract?

An agent contract is a one-page, enforceable specification attached to a tool capability (e.g., warehouse_query, lineage_lookup, or export_job). It is enforced by the platform around the agent — typically at a tool gateway or interceptor — before a tool runs (validation, budgets, policy), after it runs (evidence and verification), and during releases (regression gates and canaries).

Think of it as a tool call that is allowed only under specific conditions and must behave within defined bounds. A good agent contract answers four questions:

1. What is allowed? (datasets, output modes, data classes)
2. What are the reliability bounds? (retries, timeouts, circuit breakers, max steps)
3. How do we roll it out safely? (canaries, golden traces, rollback rules, regression triggers)
4. What does success mean? (beyond “the tool returned a response”)

In practice, contracts have four layers:

• Governance: Specify allowed scopes and enforcement (aggregation-only modes, classification tags, row or column controls).
• Release discipline: Prevent drift through canary rollouts, golden traces for trajectories, and automatic rollback triggers.
• Functional correctness: Define “done” in analytics terms (required filters, metric bindings, validation checks).
• Reliability: Bound execution (retries, timeouts, safe fallbacks, idempotency).

Agent Contract Template

This template is applied per tool.

Agent Contract: <tool_name> (vX.Y)

Purpose:

• One sentence describing what the tool is for
• When to use it and when not to

Inputs:

• Required structured inputs
• Forbidden inputs (e.g., raw PII, unbounded free text)

Success Criteria:

• Conditions that must be true for a tool call to be considered successful
• Conditions that require abstention or denial

Allowed Data Scope:

• Dataset allowlist or denylist, or tag-based restrictions
• Allowed data classes (internal, PII, confidential, public)
• Required enforcement: column masks, row filters, aggregation-only modes

Retry and Loop Controls:

• Max tool invocations per user request
• Circuit breakers (deny or degrade on repeated errors or budget exhaustion)
• Max retries and backoff

Evidence and Observability:

• Safe fingerprints: dataset IDs, redaction summaries, SQL hashes, plan hashes
• Required log fields: tool_call_id, request_id, policy_decision_id
• Required user-visible explanation fields (citations, metric bindings)

Failure Handling:

• Policy denial behavior (explain constraints; never propose workarounds)
• Timeout handling (return cached results, ask to narrow scope, or deny)
• Ambiguity handling (ask clarifying questions)

Latency and Cost:

• Cancellation rules and hard timeouts
• Cost budgets (row caps, bytes-scanned limits, export size caps)
• p50 and p95 latency targets

Rollout Rules:

• Auto-rollback triggers (retry spikes, golden trace failures, latency regressions, denial spikes)
• Approval requirements for expanding scope (new data classes or tools)
• Canary scope (e.g., 15% traffic, selected tenants, internal users)

Example Contract

warehouse_query

Purpose:
Execute bounded, parameterized SQL for exploration when the semantic layer cannot satisfy the request.

Success Criteria:

• Datasets are within the allowed scope
• Time windows, partition filters, and row limits are enforced
• Queries pass static checks and are parameterized

Governance:

• Classification checks are required before execution
• PII columns are disallowed unless explicitly masked or aggregated
• Column- and row-level security enforcement signals are required

Fallbacks:

• On denial, explain allowed alternatives such as approved metrics or aggregation-only views
• If cost caps are exceeded, suggest narrowing filters or using summary metrics

Latency and Cost:

• Hard timeouts and cancellation must be enforced
• Bytes scanned are capped; execution is canceled if exceeded

Loops and Retries:

• Max total tool invocations per user request across all tools (e.g., 3)
• Max one retry on transient errors

Golden Traces:

• “Refund count for the last 12 hours”
• “Bottom three regions by purchase rate”
• Expected path: policy_check → query_plan_check → execute → summarize

Unbounded joins or full table scans are explicitly disallowed.

Golden Traces: Regression Tests for Tool-Call Behavior

Golden traces make contracts enforceable. They don’t test whether the model got the “right” answer; they test whether the system behaved acceptably.

Each test should include:

• Allowed variance (what may change without failing)
• Governance outcomes (redaction requirements, allow or deny modes)
• Expected trajectory (tool-call sequence)
• Budgets (max invocations, retries, or cost caps)

Example: Revenue for the last 60 days by product group

• Must not call export_job
• Must attach semantic_metric and metric binding
• Only fails if retries increase, datasets expand, or latency regresses beyond the threshold

How to Adopt Without Boiling the Ocean

Start with the tools that have the largest blast radius.

• Write contracts for semantic_metric and warehouse_query first
• Add required evidence fields such as dataset IDs, fingerprints, and policy decision IDs
• Add auto-rollback rules and canaries
• Add contract enforcements at the tool gateway (retries, timeouts, loop ceilings)
• Create 10–30 golden traces and run them on every prompt, router, or model change

The hardest part is not writing the template — it’s defining success criteria that represent trustworthy analytics, not just a tool that returned a result.

Conclusion

We want agents to feel magical to users but boring to operators. Tool calls should be governed, predictable, and testable like any other production system. When they are, agents stop being a source of surprise incidents.

Prompts describe intent, but contracts enforce reality. When the next model upgrade or schema change arrives, agent contracts help keep tool behavior stable, within budget, and compliant.