Stop Paying Your AI Agent to Do the Same Job Twice

The key detail that makes this actually save money, rather than just being a fancy cache: tiers 1 and 2 run as plain code in the webhook handler, before the agent is ever invoked. The decision of "do we need the AI here?" is made without AI.

Why Hashing Alone Isn't Enough

A naive cache would just hash (task_type, params) and check for an exact match. That handles literal repeats — someone re-running the identical command but it misses the far more common case: near-duplicate requests.

Think about it from the requester's side. 

1.   /collect-data --source=warehouse-a --range=2026-07 and
2.  /collect-data --source=warehouse-a --range=2026-07 --format=csv 

That's why there is a second tier: turn the task into a short natural-language description,

task: collect-data; range=2026-07; source=warehouse-a

embed it, and search for the closest prior tasks of the same type using cosine similarity in Postgres with pgvector. If something is very close (above a "full match" threshold), we serve it directly. If it is close but not close enough to the same source, different range, say, we treat it as a partial hit: we know part of the answer, and we hand that to the agent as context so it only has to fill the gap.

Schema

The whole cache lives in one table, plus an execution log for observability:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE task_knowledge (
    id              BIGSERIAL PRIMARY KEY,
    task_type       TEXT NOT NULL,
    signature_hash  TEXT NOT NULL UNIQUE,   -- exact-match lookup
    params          JSONB NOT NULL,
    description     TEXT NOT NULL,          -- text fed to the embedding model
    embedding       vector(1024),           -- semantic-match lookup
    result          JSONB NOT NULL,
    covered_scope   JSONB NOT NULL DEFAULT '{}'::jsonb,
    missing_scope   JSONB NOT NULL DEFAULT '{}'::jsonb,
    status          TEXT NOT NULL DEFAULT 'complete',  -- complete | partial
    ttl_seconds     INT NOT NULL DEFAULT 86400,
    executed_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_embedding ON task_knowledge
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

Two columns do a lot of the conceptual work: covered_scope and missing_scope. Every cached result knows what it actually answers and what it doesn't; this is what lets a partial hit be useful instead of all-or-nothing.

The Signature Has to Be Genuinely Deterministic

The exact-match tier is only as good as the hash is stable. {"source": "warehouse-a", "range": "2026-07"} and {"range": "2026-07", "Source": "warehouse-a "} need to hash identically, or the cache silently misses on trivial formatting differences. So normalization happens before hashing:

def normalize_params(params: dict) -> dict:
    normalized = {}
    for key, value in params.items():
        norm_key = key.strip().lower()
        if isinstance(value, str):
            value = value.strip()
        normalized[norm_key] = value
    return normalized

def build_signature(task_type: str, params: dict) -> str:
    payload = {"task_type": task_type.strip().lower(), "params": normalize_params(params)}
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

sort_keys=True matters more than it looks without it; dict key order leaks into the hash, and two functionally identical requests produce different signatures.

The Lookup Flow

async def handle_task(ctx: TaskContext) -> None:
    task_type, params = ctx.command.task_type, ctx.command.params

    if not ctx.command.force:
        exact = await kb_service.exact_lookup(task_type, params)
        if exact.kind == "exact":
            return await _serve_cached(ctx, exact.result, "cache-exact")

        semantic = await kb_service.semantic_lookup(task_type, params)
        if semantic.kind == "semantic_full":
            return await _serve_cached(ctx, semantic.result, "cache-semantic")

        if semantic.kind == "semantic_partial":
            return await _run_agent_and_finish(
                ctx, scope="partial",
                prior_result=semantic.result,
                missing_scope=semantic.missing_scope,
            )

    # nothing usable in cache, or --force was passed
    await _run_agent_and_finish(ctx, scope="full", prior_result=None, missing_scope=None)

Notice the order: cheapest and most certain first. By the time you are calling the agent, you already know either this is genuinely new or here is what exactly is missing; the agent never has to rediscover context it already had access to in a prior run.

Scoping the Agent Call Is the Actual Cost Saver

It is tempting to stop at caching the full result and skip the agent on hits. That alone helps, but the bigger win is what happens on a partial hit. Instead of:

Do the whole task from scratch

the agent gets:

Here's what we already know. Here is specifically what is missing. Fill only that.

payload = {
    "task_type": task_type,
    "params": params,
    "scope": scope,                 # "full" or "partial"
    "prior_result": prior_result,   # trusted context on a partial run
    "missing_scope": missing_scope, # exactly what to compute
}

A well-scoped prompt on a partial hit is dramatically cheaper than a cold-start prompt has less context to establish, less reasoning to redo, and fewer tool calls in many cases. This is the difference between caching the whole answer or not and actually decomposing the task so the agent's effort is proportional to what is genuinely new.

Freshness Matters as Much as Matching

A cache with no expiry is a correctness bug waiting to happen; data pipelines especially. ttl_seconds is set per task type (data pulls might be valid for a day, static reference lookups for a month), and every lookup checks staleness before it is considered a hit at all:

def _is_fresh(executed_at: datetime, ttl_seconds: int) -> bool:
    age = (datetime.now(timezone.utc) - executed_at).total_seconds()
    return age <= ttl_seconds

And because the cache is wrong is always a possibility someone needs to escape from, the slash command supports a --force flag that skips all three tiers and always re-runs the agent; cheap insurance against a bad cache entry blocking someone.

What This Actually Buys You

For a workflow where a meaningful fraction of requests are repeats or near-repeats:

• Exact hits cost nothing – a single indexed hash lookup instead of an agent invocation and a Jenkins run.
• Semantic hits cost nothing – same, just via vector similarity instead of literal equality.
• Partial hits cost a fraction of a full run – the agent's context and reasoning scope shrink to just the gap.
• The system gets better over time – every agent run, full or partial, ends with an upsert into the knowledge base, so the next similar request has a better chance of hitting tier 1 or 2.

None of this requires touching the AI agent's internals or model choice. It is entirely a decision layer sitting in front of it, which is exactly why it is cheap to build and safe to roll out incrementally: worst case, everything falls through to tier 3 and behaves exactly like the system did before.

Where This Pattern Breaks Down

Worth being honest about the limits:

• Highly unique tasks (every request meaningfully different) get no benefit; you are just adding a cache lookup with no hits.
• Semantic thresholds need real tuning. Too loose, and you serve stale near-misses as if they were exact. Too tight, and tier 2 never fires, and you've built a vector index for nothing. This needs actual production traffic to calibrate, not guesswork.
• Partial-scope decomposition only works if your agent (or its skill definitions) can meaningfully interpret "do just this part." Some tasks are not decomposable; collecting one row of a dataset is not a well-defined sub-task if the pipeline processes the range as a single unit. In those cases, a partial hit should probably just be a lower similarity threshold for a full re-run, not a scoped one.
• Correctness > cost. If being wrong is expensive (financial data, compliance), skew every tuning knob toward fewer cache hits, not more.

The Broader Point

AI agents are excellent at reasoning over genuinely new problems and bad economics for repeated ones. Most production agent workflows I have seen treat every request as novel by default, which is the expensive default. Adding a deterministic lookup layer in front — one that is cheap enough to always check and specific enough to trust — turns running the agent from the default action into the fallback action. That one inversion is where most of the savings come from.