Engineering AI Accountability: Control the Execution Path, Not the Model

An approved model does not make an AI system accountable. The consequential behavior emerges from the model's interaction with runtime context, retrieved data, tools, permissions, orchestration rules, and human approval controls.

That changes the assurance target. Engineering teams still need model evaluation, dataset documentation, bias testing, red teaming, and release approval. They also need to prove that the deployed system operated within its delegated authority and that they can reconstruct the execution path that produced an outcome.

The practical shift is from governing a model artifact to governing a socio-technical execution boundary.

Move the Assurance Boundary Outward

Production AI systems increasingly assemble decisions at runtime. The orchestrator selects context, the retriever introduces enterprise data, policy services constrain actions, tools change external state, and reviewers approve or challenge recommendations. The model participates in the decision; it does not determine the system boundary.

A stale vector index can produce a harmful recommendation even when the model performs perfectly. An overprivileged tool can convert a hallucination into an unauthorized financial transaction. A poorly designed approval queue can reduce human oversight to rubber-stamping. None of these failure modes exist within model weights.

In architecture reviews, evaluate system capability as a function of several interacting controls:

Capability = f (Model, Context, Data, Tools, Permissions, Policy, Workflow)

This formulation sets a boundary rather than generating a score. Changing any term alters the application's risk profile without altering the model. For instance, a single foundation model can power three distinct risk profiles:

• Internal Summarizer: Read-only access with zero external side effects.
• Customer Service Agent: Reads account history, issues messages, and processes limited refunds.
• Procurement Agent: Compares suppliers, negotiates contract terms, and executes purchases.

Model governance applies to all three. System governance must distinguish their different authority levels, blast radiuses, and evidence requirements.

Governing Execution Paths, Not Isolated Operations

Traditional identity and access management (IAM) evaluates operations in isolation: May this principal read this record? May this role invoke this API? Agentic workflows introduce risk through path composition.

If an agent can read sensitive customer records and send external emails, both permissions might pass individual IAM checks. However, combining them creates a data exfiltration risk. Static role-based access control (RBAC) cannot reliably detect path-based failures because enforcement depends on prior events, data classification, business purpose, and accumulated context. The execution path itself must become a first-class governance object.

Path-aware enforcement can be implemented using deterministic runtime rules:

• Context labeling: Tag data with classification metadata as it enters the context window.
• Metadata propagation: Pass sensitivity and purpose tags through intermediate artifacts.
• Accumulated state evaluation: Evaluate proposed tool calls against accumulated context labels.
• Transaction binding: Bind high-impact actions to transaction-specific approvals.
• Approval invalidation: Invalidate pending approvals if arguments (recipient, amount, payload) change.
• Fail-closed policy: Deny execution when required provenance metadata is missing.

Using deterministic controls to evaluate proposed model actions prevents enforcement responsibilities from falling on the probabilistic model itself.

Separating Technical Access from Delegated Authority

An IAM role specifies which resources an identity can read or invoke, but it does not define which business decisions an agent may make. A procurement agent may require read access to supplier catalogs, price histories, and contract schemas; these permissions do not imply authority to select a vendor, accept commercial terms, or release payments.

Represent delegated business authority as an explicit, machine-enforceable contract:

agent_id: procurement-agent-17
purpose: supplier_comparison

data_scope:
  allow:
    - approved_supplier_catalog
    - purchase_history

tools:
  allow:
    - search_catalog
    - request_quotation
  deny:
    - create_purchase_order
    - release_payment

decision_authority:
  allow:
    - recommend_supplier
  deny:
    - accept_commercial_terms

execution_limits:
  autonomous_spend_gbp: 0
  subagent_delegation: false

approval:
  required_for:
    - supplier_selection
    - purchase_order
  expires_after_minutes: 15

revocation:
  mode: immediate

An authority schema must address six core operational questions:

1. Purpose: What business objective is permitted?
2. Data and tools: Which resources and interfaces are authorized for that purpose?
3. Decision scope: Which choices may the agent recommend versus decide?
4. Execution scope: Which side effects may it trigger autonomously?
5. Constraints: What financial limits, approvals, and segregation-of-duties rules apply?
6. Ownership: Which accountable principal can revoke this authority?

Fine-grained authority models introduce policy maintenance overhead, whereas coarse roles obscure authority inside technical permissions. Organizations should standardize reusable authority profiles for common risk tiers, applying transaction-level constraints for consequential actions.

Operationalizing Human Oversight

Inserting a manual approval gate does not guarantee meaningful oversight. A reviewer evaluating hundreds of decisions daily with seconds per case will default to confirming automated outputs—especially when the UI displays only the final recommendation.

Effective human-in-the-loop controls require structured operational conditions:

• Contextual delivery: Present recommendations alongside supporting evidence, provenance tags, and explicit uncertainty flags.
• Scope-bound approvals: Bind approvals strictly to exact proposed parameters. An approval for a £500 adjustment must not authorize a £5,000 transaction or a modified payload.
• Actionable workflows: Provide clear mechanisms to challenge, override, or escalate recommendations.
• Telemetry monitoring: Track review velocity, queue pressure, override rates, and approval patterns.
• Adaptive capacity control: Suspend or re-route automation if reviewer capacity drops below design thresholds.

Applying risk-tiered review avoids operational bottlenecks: fully automate low-impact, reversible tasks; sample medium-risk workflows for quality control; and mandate explicit approval for high-impact or irreversible side effects.

Connecting Design Time Intent to Runtime Evidence

Design documentation states how a system should operate; runtime evidence proves how it executed. This distinction is vital when agents select tools dynamically, retrieval contexts shift, and workflows execute asynchronously. Post-incident analysis cannot reconstruct a decision using only a model version and final output.

A robust architecture segregates concerns into three distinct planes: Governance (purpose, policy, authority), Execution (agent, model, context, tools), and Evidence (provenance, audit traces, outcomes).

For every consequential execution, record typed events with a stable correlation ID to capture:

• Initiation context: Initiating identity, business purpose, and active authority profile.
• Component metadata: Model, prompt template, policy engine, and tool versions used.
• Lineage and logic: Retrieved source items, retrieval timestamps, policy rules evaluated, and tool arguments generated.
• Oversight and state: Reviewer identity, exact approval scope, executed state changes, and downstream business outcomes.

Use structured, typed event schemas rather than unstructured trace logs for audit compliance. Unstructured logs remain valuable for real-time debugging, but schema drift makes them unreliable for formal control testing.

Preserving Audit Evidence Securely

Maximum observability differs from maximum accountability. Raw agent traces often contain personally identifiable information (PII), proprietary documents, system prompts, credentials, and confidential tool responses. Logging all context indiscriminately introduces security and compliance risks.

Design audit systems for maximum verifiability with minimum necessary disclosure:

• Cryptographic references: Store immutable content hashes (SHA-256SHA-256) and versioned source IDs instead of duplicating whole payload documents.
• Segmented telemetry: Separate short-term operational logs from long-term, restricted-access audit records.
• Data minimization: Redact credentials, API keys, and unneeded PII before writing events to storage.
• Policy claims: Record structured policy decision inputs and outputs instead of raw contextual prompts.
• Tiered payload retention: Reserve full payload persistence for high-risk transaction classes that explicitly require complete replay capabilities.

Defining the Governed System Boundary

Different engineering domains maintain distinct perspectives on the system boundary:

• Model teams: View prompts, weights, and inference outputs.
• Application teams: Focus on orchestration, context assembly, and retrieval chains.
• Security teams: Monitor identities, network gateways, microservices, and data stores.
• Compliance teams: Track business purpose, data processing, policy checks, and end-user impacts.
• Operations teams: Manage throughput, system dependencies, error rates, and incidents.

System governance aligns these perspectives around the end-to-end business outcome.

Lifecycle Integration Guide

Integrate accountability controls directly into the software development lifecycle:

1. Map business consequences: Categorize system actions by reversibility, financial limits, data sensitivity, and recovery time.
2. Define authority contracts: Establish explicit schemas covering data access, tool invocation, decision scope, execution boundaries, and revocation owners.
3. Threat-model execution paths: Evaluate multi-step attack vectors, including prompt injection via retrieval data, confused deputy scenarios, parameter substitution, and tool-chaining exploits.
4. Enforce at the side-effect boundary: Validate tool arguments, policy state, and accumulated context immediately before executing external actions.
5. Calibrate review capacity: Design human approval interfaces to display clear evidence, enforce time-on-task standards, and automatically throttle automation if queues overflow.
6. Emit structured evidence: Log typed events linked by correlation IDs to enable independent audit reconstruction.
7. Verify revocation controls: Test kill-switches to ensure teams can revoke agent authority, invalidate pending approvals, and cancel queued tasks instantly.

Architecture Review Checklist

• What real-world state changes can this system initiate?
•  What runtime data enters the context window via retrieval and tools?
•  What external side effects can each tool produce?
• What explicit decision boundaries govern the agent's actions?
• What sequence of individually permitted actions could produce a prohibited outcome?
• Where do deterministic policy engines validate proposed operations?
• Does human oversight remain effective during peak transaction volumes?
• Is approval cryptographically bound to the exact execution parameters?
• Can authority be revoked instantly to stop queued and in-flight operations?
• Can an independent auditor reconstruct a historical execution path without developer intervention?
• Is audit evidence stored securely without unnecessarily duplicating sensitive context?

Model governance verifies the safety and training lineage of a technical component. However, it cannot confirm whether a running system stayed within its delegated authority, executed valid context, maintained human oversight, or initiated authorized business actions.

Complete system assurance requires governing the full execution path: purpose, context, policy, tools, human controls, actions, and audit evidence. Placing deterministic controls at the action boundary ensures AI systems remain accountable, observable, and defensible in production.