Beyond the Black Box: Implementing “Human-in-the-Loop” (HITL) Agentic Workflows for Regulated Industries

The Technical Hook

Autonomous agents exhibit failure patterns analogous to those in distributed systems: not through isolated catastrophic errors, but via a cascade of locally justifiable actions that collectively result in globally unsafe states. Prompt injection in AI systems parallels a forged remote procedure call (RPC) syntactically valid input that traverses multiple processing layers before inducing an unauthorized state transition. 

As illustrated in Figure 1, this architectural risk is mitigated by the "Commit Boundary," which prevents adversarial inputs from reaching sensitive executors by validating every intent against a deterministic schema. When extended with capabilities such as tool invocation and long-term planning, these agents manifest failure modes like confused deputy scenarios and privilege escalation, which are neutralized by the layered enforcement framework depicted in the diagram.

The Architecture Pattern

In the initial phases of agent development, teams typically prioritize "tool correctness" (ensuring the agent invokes the correct API) and "model correctness" (verifying the accuracy of generated text). However, in regulated domains, this prioritization is misaligned. The primary architectural consideration must be: where is the commit boundary established, and what deterministic controls govern state transitions across that boundary?

A proven architectural pattern for high-compliance sectors such as financial services, the Defense Industrial Base (DIB), and industrial control systems follows this sequence:

Agent → Policy Gate → Human Review → Executor

As illustrated in the high-level sequence (Agent → Policy Gate → Human Review → Executor), the architecture depicted in Figure 2 operationalizes the 'Commit Boundary' design pattern. This pattern establishes a structural separation between probabilistic agent decision-making and deterministic system operations through a layered policy enforcement framework. By integrating human-in-the-loop (HITL) oversight as a validation gate before the final executor, the system ensures that every state transition in a regulated environment is bounded, traceable, and attributable.

The "Commit Boundary" Design Pattern

The commit boundary demarcates the transition from advisory output to executable action. Within agentic workflows, direct modification of production state by the agent must be prohibited. Instead, the agent generates a structured action request subject to the following deterministic stages:

1. Typed and validated against a fixed schema to ensure syntactic and semantic integrity, 
2. Scored and classified according to predefined risk tiers using rule-based or statistical models, 
3. Submitted for human evaluation when risk thresholds exceed defined limits, 
4. Processed exclusively by an execution service operating under least-privilege principles and supporting idempotent operations, 
5. Persisted in an immutable log to maintain a verifiable audit trail resilient to model iteration or retraining.

This approach does not hinder AI deployment; rather, it applies established engineering rigor—commonly enforced in database schema changes, payment processing, or privileged system modifications to agentic systems. It mandates formalization of intent, systematic evaluation, conditional approval, and controlled state mutation, ensuring compliance, traceability, and operational safety.

Implementation Step 1: Typed Action Schemas as Governance Gates

Without a deterministic specification of an agent’s intended state transition, auditability is unattainable. Unstructured natural language does not constitute a verifiable audit record, but it introduces ambiguity and risk. The foundational improvement lies in routing all state-modifying operations through rigorously defined, typed schemas, establishing the schema as the boundary layer between probabilistic decision-making and deterministic system enforcement. 

Presented below is a streamlined Pydantic model for a TypedActionRequest. This design is intentionally prescriptive: it decouples agent intent from system execution, ensuring that state transitions, particularly those involving Controlled Unclassified Information (CUI) or sensitive financial data, proceed after passing validation checks. By embedding policy logic directly into the schema, we provide auditors and incident responders with a verifiable record of causal provenance: identifying exactly what was triggered, the justification provided, and the evidence used to authorize the action.

from pydantic import BaseModel, Field, HttpUrl, model_validator
from enum import Enum
from typing import Any, Dict, List

class DataSensitivity(str, Enum):
    PUBLIC = "public"
    INTERNAL = "internal"
    CUI = "cui"  # Controlled Unclassified Information (NIST 800-171) 
    PII = "pii"

class ActionType(str, Enum):
    WRITE = "write"
    READ = "read"
    DELETE = "delete"
    ACCESS_GRANT = "access_grant"
    TRANSFER = "transfer"
    CONFIG_CHANGE = "config_change"

class TypedActionRequest(BaseModel):
    """
    The formal 'intent' packet that crosses the commit boundary.
    Separates probabilistic agent 'thought' from deterministic execution. 
    """
    actor: str = Field(..., description="Authenticated principal for the agent session")
    action: ActionType
    target_system: str = Field(..., description="System-of-record (e.g., Jira, GitHub, SAP)")
    sensitivity: DataSensitivity
    justification: str = Field(..., min_length=20, description="Auditable reasoning")
    evidence_urls: List[HttpUrl] = Field(default_factory=list, description="Links to tickets/logs")
    
    # Critical for distributed safety: prevents the agent from re-running an action
    idempotency_key: str = Field(..., min_length=16) 

    @model_validator(mode="after")
    def enforce_governance(self) -> "TypedActionRequest":
        """
        Enforces policy as code. This ensures that no state transition 
        occurs without a verifiable audit trail. [cite: 750, 761]
        """
        # Rule: State-changing actions MUST have associated evidence (e.g., a ticket URL)
        if self.action != "read" and not self.evidence_urls:
             raise ValueError("State-changing actions must include evidence_urls for audit integrity")
             
        # Rule: Restrict sensitive data (CUI/PII) to specific hardened targets
        if self.sensitivity in {DataSensitivity.CUI, DataSensitivity.PII}:
            if "public" in self.target_system.lower():
                raise ValueError(f"High-sensitivity {self.sensitivity} cannot target public systems")
                
        return self

The Necessity of Schemas as the Sole Auditable Conduit to Probabilistic Systems

In regulated contexts, compliance does not derive from inspecting model weights or outputs. It arises from examining system behavior specifically: the identity of the requester, the data accessed, the control policies applied, approval lineage, and the precise operation executed. 

Typed schemas enable:

• Deterministic interpretation (eliminating ambiguity in action semantics), 
• Reproducible change evaluation (structured diffs support accurate review), 
• Uniform logging (consistent field presence across events), 
• Enforceable policy integration (attribute-based routing and controls via explicit fields such as action type and data sensitivity), 
• Long-term stability (while models evolve frequently, the schema remains a constant reference).

This approach directly supports compliance with security frameworks emphasizing access governance, audit integrity, configuration control, and sensitive data handling, such as NIST SP 800-171, which mandates protection of Controlled Unclassified Information (CUI) in nonfederal information systems (NIST Computer Security Resource Center).

Implementation Step 2: Tiered Risk Routing — Managing Reviewer Fatigue Through Deterministic Logic

Deploying "Human-in-the-Loop" (HITL) as a universal mandate requiring approval for every agent action proves ineffective in operational environments. This approach leads to reviewer overload, processing delays, habitual approvals without scrutiny, and ultimately results in de facto reversion to full automation.

A more sustainable solution is tiered risk routing: a deterministic mechanism that evaluates action requests using a quantifiable risk score, automatically executing low-risk actions while escalating medium- and high-risk actions to designated human review levels. Crucially, risk assessment must be derived from explicit, traceable data attributes rather than subjective judgment.

The following example outlines a concrete risk_score function. It classifies actions into one of four pathways — AUTO, PEER_REVIEW, SECURITY_REVIEW, or LEGAL_COMPLIANCE based primarily on data sensitivity and action category, with incremental risk adjustments applied for privileged access modifications, financial transactions, and irreversible operations.

from enum import Enum
from typing import Tuple

class ReviewTier(str, Enum):
    AUTO = "auto"
    PEER_REVIEW = "peer_review"
    SECURITY_REVIEW = "security_review"
    LEGAL_COMPLIANCE = "legal_compliance"

def calculate_risk_tier(req: TypedActionRequest) -> Tuple[int, ReviewTier, str]:
    """
    Scans a proposed action and routes it to the appropriate governance tier.
    Returns: (score 0-100, tier, rationale)
    """
    score = 0
    rationale = []

    # 1. Sensitivity Bias: CUI/PII/PCI are first-class routing signals
    sensitivity_map = {
        "internal": 10,
        "cui": 35,  # Controlled Unclassified Information (NIST 800-171)
        "pii": 45,
        "pci": 60
    }
    score += sensitivity_map.get(req.sensitivity.value, 0)
    rationale.append(f"Sensitivity: {req.sensitivity}")

    # 2. Action Impact: Destructiveness and privilege changes increase risk
    action_risk_weights = {
        "read": 0,
        "write": 15,
        "config_change": 35,
        "access_grant": 50  # High-risk: alters the security posture
    }
    score += action_risk_weights.get(req.action.value, 0)
    rationale.append(f"Action: {req.action}")

    # 3. Contextual Overrides: Large transactions or admin requests
    if req.parameters.get("amount_usd", 0) >= 10000:
        score += 20
        rationale.append("Large financial transfer")
    
    if req.parameters.get("grants_admin", False):
        score += 25
        rationale.append("Admin privilege escalation")

    if req.dry_run:
        score -= 15  # Mitigating factor: non-mutating validation
        rationale.append("Safe-mode (Dry Run)")

    score = max(0, min(100, score))

    # 4. Deterministic Routing Logic
    # Hard Escalation: Any state change to CUI data triggers a Security Review
    if req.sensitivity == "cui" and req.action != "read":
        return score, ReviewTier.SECURITY_REVIEW, "Sensitive state-change mandate"

    if score < 20:
        return score, ReviewTier.AUTO, "Low-impact automated execution"
    elif score < 50:
        return score, ReviewTier.PEER_REVIEW, "Standard peer oversight"
    elif score < 75:
        return score, ReviewTier.SECURITY_REVIEW, "High-risk security audit required"
    
    return score, ReviewTier.LEGAL_COMPLIANCE, "Critical compliance review required"

As demonstrated by the decision logic in Figure 3, these tiered outcomes provide the structural foundation for governed automation. To ensure this system remains robust in a production environment, there are two critical implementation considerations to address:

• Preserve the rationale string: The rationale string generated during scoring must be preserved within the review package and audit log. This enables clear responses to queries such as, “Why was Security Review triggered?” with an objective, reproducible justification.
• Strict parameter schemas: Unstructured key-value inputs introduce hidden risk through invalidated fields. Instead, model parameters as a controlled API: define schemas, maintain versioning, document allowable fields, and reject unrecognized keys — particularly in high-risk systems.

Mapping System Architecture to Regulatory Requirements

This phase operationalizes policy by transforming external regulatory mandates into enforceable system-level constraints, ensuring architectural alignment with compliance objectives.

NIST SP 800-171 Rev. 3 as a Constraint Model at the Commit Boundary

NIST SP 800-171 Rev. 3 establishes security requirements for safeguarding Controlled Unclassified Information (CUI) in nonfederal information systems (NIST Computer Security Resource Center). Compliance is not achieved through documentation alone but through architectural enforcement of authentication, authorization, auditing, and data handling controls.

Architectural mechanisms enabling compliance: 

• Access control and least privilege: The Executor operates as a distinct service identity with minimal, role-specific permissions. The agent does not possess production credentials, thereby limiting the potential impact of compromise or misuse during unauthorized access attempts.
• Audit and accountability: A verifiable audit trail is generated through structured data elements - TypedActionRequest, risk assessment score, human approval decision, and execution outcome. This transforms ambiguous autonomous actions into auditable, deterministic state transitions.
• Configuration management and change control: Configuration modifications are formalized as CONFIG_CHANGE actions, requiring passage through the commit boundary. This converts untracked configuration updates into governed, inspectable change operations.
• Controlled CUI handling: Data sensitivity labels (e.g., PUBLIC, INTERNAL, CUI) function as multi-purpose controls, influencing routing decisions, determining log redaction policies, and restricting permissible execution endpoints based on classification.

NIST IR 8596 as Constraints on AI Cybersecurity Outcomes

NIST IR 8596 functions as a cybersecurity framework profile tailored to artificial intelligence, aligning AI-specific risk factors with measurable cybersecurity objectives. (NIST Publications) Its primary operational implication underscores a frequently overlooked engineering principle: AI implementations extend beyond statistical models that constitute complex, interconnected systems requiring comprehensive, end-to-end security measures.

Architectural implications include:

• Deployment of a Policy Gate as a dedicated enforcement point for mitigating AI-unique threats, including prompt injection, unauthorized tool invocation, data leakage via agent actions, and uncontrolled autonomous execution paths. 
• Implementation of the Human Review Queue is not as a default fallback mechanism reliant on human judgment, but as a programmatically triggered control governed by deterministic decision rules. 
• Elevation of the Audit Trail to a core security component, enabling pattern detection, forensic analysis, and iterative control refinement through reliable, structured records - eliminating reliance on heuristic interpretation of agent behavior.

Colorado SB24-205 as Constraints on High-Risk Decision Systems

Colorado’s SB24-205 (Consumer Protections for Artificial Intelligence) establishes legal obligations for systems classified as “high-risk,” mandating reasonable safeguards against known or foreseeable risks of algorithmic discrimination, with enforceability beginning on the date specified in the legislation. (Colorado General Assembly).

From an engineering perspective, compliance translates into: 

• Mandatory traceability and governance mechanisms for agentic workflows involved in high-stakes domains such as credit, employment, insurance, or housing that remain effective across model iterations and system updates. 
• Reliance on architecture-defined intent typing, rule-based routing logic, and immutable audit logging to generate auditable evidence of input factors, proposed actions, approval authorities, and executed outcomes forming the foundational data layer required to support fairness assessments and regulatory inquiries following adverse events. (Colorado General Assembly).

Open Problems and Challenges

Reviewer Fatigue Stems From Systemic Design Limitations, Not Individual Capacity Constraints

When every operational decision triggers a manual review ticket, cognitive load accumulates, leading to reviewer exhaustion and approval decisions driven by habit rather than scrutiny. While tiered routing mitigates volume, it must be complemented by:

• Action aggregation and differential summaries: present reviewers with compact, change-focused diffs instead of full execution logs.
• Idempotent retry mechanisms: eliminate redundant approvals caused by transient infrastructure failures by ensuring repeat executions do not trigger new review cycles.
• Statistical sampling with retrospective audits: for low-risk automated operations, implement a policy to validate risk models and detect policy drift without introducing latency.

Evaluating Long-Horizon Agent Behavior Lacks Deterministic Tractability

Even with well-defined commit boundaries, agents operating over extended timeframes introduce evaluation challenges. The primary risk is not isolated, faulty actions but sequences of individually valid steps that, collectively, violate safety invariants, mirroring emergent failure modes in distributed systems.

Effective countermeasures include:

• Resource and action budgets: impose limits on state-modifying operations per session, resource, or time interval.
• Policy-enforced invariants: embed logical constraints in approval gates (e.g., prohibiting simultaneous privilege escalation and audit disablement).
• Causal trace propagation: attach persistent trace identifiers to all actions, enabling accurate reconstruction of execution lineage independent of model-generated memory.

Architect’s Note

In a corporate setting, achieving audit-ready AI requires designing agent systems as untrusted components operating within trusted, governed workflows. Persuasion does not stem from claims of model sophistication; it derives from demonstrable system properties - specifically, that all state transitions are bounded, traceable, and attributable, and that operational integrity is preserved even under model failure, adversarial input, or stochastic error.

Speed need not be sacrificed at the point of execution, provided human oversight is strategically allocated. Focus human judgment where its marginal impact is greatest: handling sensitive data, executing privileged actions, transferring value, and implementing irreversible operations. All other processes should be engineered for safe automation via deterministic validation gates, rigidly defined schemas, and idempotent execution mechanisms. If human review queues accumulate indeterminate cases, the root cause typically lies in underspecified routing logic, overly permissive action definitions, or systemic gaps being offset by manual intervention.

Furthermore, the audit trail must be treated as a first-class deliverable. It should be structured for efficient querying, cryptographically immutable, and aligned with business-level identifiers rather than low-level technical logs. When accountability is demanded, “Why did this occur?”, the response must not rely on reconstructing intent from chat histories or inferring significance from raw tool invocations. 

Instead, a unified, time-ordered record must exist, linking typed user intent, policy-based decision logic, required human approvals, execution outcomes, and resulting state changes — all anchored by identity, idempotency keys, and temporal sequence. This is the foundation for deploying agentic systems in regulated environments without exposing organizational risk to opaque, unverifiable processes.

References and Further Reading

• arXiv 2601.17548: Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems (Maloyan & Namiot, Jan 2026) - A comprehensive systematization of 42 attack techniques and the relative failure of current defenses against adaptive injection.
• NIST SP 800-171 Rev. 3: Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations (National Institute of Standards and Technology, May 2024) - The federal security requirement for safeguarding CUI in non-federal systems, serving as the primary constraint model for industrial AI.
• NIST IR 8596 (IPD): Cybersecurity Framework Profile for Artificial Intelligence (Cyber AI Profile) (National Institute of Standards and Technology, December 2025) - Guidelines for managing AI-specific risk factors using the NIST Cybersecurity Framework 2.0.
• OWASP LLM Top 10: OWASP Top 10 for Large Language Model Applications (OWASP Foundation, 2025) - The definitive industry classification for AI vulnerabilities, specifically addressing "Excessive Agency" (LLM08) and "Prompt Injection" (LLM01).
• Colorado SB24-205: Concerning Consumer Protections in Interactions With Artificial Intelligence Systems (Colorado General Assembly, May 2024) - The first comprehensive U.S. state law mandating "reasonable care" and impact assessments for high-risk AI decision systems.
• FINRA 2026 Report: Annual Regulatory Oversight Report: Dedicated Generative AI Section (FINRA, Dec 2025) - Clarifies that firms must maintain recordkeeping and supervision even for "agentic" automated support workflows.