DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Threat Modeling Context-Aware AI: A STRIDE Walkthrough
  • A Practical Pipeline for Identifying Sensitive Columns Before Test Data Masking
  • Building a Mortgage Agent With FRED Data, FastAPI, and LLM Tool Calling
  • Using LLMs to Automate Data Cleaning and Transformation Pipelines

Trending

  • Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense
  • Prompt Caching for LLMs, Part 1: Exact-Match Caching in Production
  • Architecture Review Boards for Generative AI: A Framework for Governance, Risk Management, and Responsible Adoption
  • Going All In On the JVM 25 Virtual Threads

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. AI/ML
  4. The Alert Is Not the Incident: LLM Triage on a Multi-Engine Data Platform

The Alert Is Not the Incident: LLM Triage on a Multi-Engine Data Platform

Incidents cross engine boundaries; alerts do not. Discover how bounded LLM context assembly accelerates incident triage while keeping remediation safely in human hands.

By 
Thirupathi Reddy Anneda
·
Nov. 19, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
24 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

At 2:40 in the morning, the page said a scheduled job had failed. It was accurate. It was also close to useless.

The failed job was the last in a chain that started in a source system, moved through a change-data-capture topic, landed in one warehouse, joined against a reference set in a completely different engine, and finally fed the dashboard an executive would open in about five hours. The failing job was the only component with an alert on it because it was the only component anyone had thought to instrument end to end. The actual fault was four hops upstream, in a partition that had stopped advancing an hour earlier without tripping anything.

Whoever picks up that page spends the first twenty-five minutes not fixing anything. They are reconstructing a picture that exists nowhere except in the heads of maybe six people, three of whom are asleep in a timezone eleven hours away.

I have spent about eighteen years in enterprise data and integration work, most of it on always-on platforms with hard SLAs, and I now run production engineering for a reporting ecosystem that spans several database engines, a streaming layer, and a scheduler that ties them together. When we started applying LLMs to operations, the useful result was not the one I expected. It was not that the model fixed things. It turned out that the twenty-five minutes of reconstruction was an information-assembly problem, and information assembly is exactly what these models are good at.

Alerts Are Per-Engine, Incidents Are Not

Every engine in a heterogeneous stack ships with competent monitoring for itself. The warehouse tells you about query queuing. The streaming platform tells you about consumer lag. The scheduler tells you about exit codes. Each one is correct, and each one is scoped to its own boundary.

Incidents do not respect those boundaries. Nearly every serious one I have worked on crossed at least two engines, and the alert that fired was almost never attached to the component that broke. It was attached to the first component downstream that could not tolerate what broke.

The usual reaction is to add more alerts. We did that for years. The result is an on-call rotation that receives more signals, correlates them no faster, and develops a fine instinct for ignoring the noisy ones. Detection was never the constraint. Correlation was.

The Unglamorous Prerequisite

Before any model was involved, we had to make the platform's structure machine-readable, because you cannot ask a model to reason about a topology it cannot see. That meant a lineage and ownership manifest, maintained as a checked-in artifact rather than a wiki page.

YAML
 
# lineage/exec_daily_summary.yaml

asset: exec_daily_summary

sla:

  ready_by: "06:00 PT"

  severity: p1

upstream:

  - id: cdc.orders            { type: stream,    lag_alert_min: 15 }

  - id: wh.orders_staged      { type: warehouse, engine: primary }

  - id: ref.geo_hierarchy     { type: warehouse, engine: secondary, refresh: weekly }

  - id: job.summary_rollup    { type: scheduled, window: "04:15-05:00 PT" }

owners:

  primary: data-platform-oncall

  business: revenue-reporting

known_failure_modes:

  - "weekly ref refresh lands late, joins produce null geo, rollup succeeds with wrong totals"


That last block matters more than it looks. Failure modes that a team knows about but has never written down are precisely the ones a new on-call engineer rediscovers at 3 am, and precisely the ones a model has no way to infer.

Give the Model Evidence, Not the Alert

The failure pattern I see most often when teams try this is handing the LLM the alert text and asking what is wrong. That produces confident, generic, wrong answers, because the alert text contains almost none of the information needed to answer.

What works is spending the tokens on retrieval first. When a page fires, an assembler walks the lineage graph and collects the state of everything upstream inside the relevant window, plus what changed recently.

Python
 
def assemble_context(asset_id, fired_at, lookback=timedelta(hours=6)):

    node = lineage.load(asset_id)

    evidence = {"asset": asset_id, "fired_at": fired_at.isoformat(),

                "sla": node["sla"], "known_failure_modes": node["known_failure_modes"]}



    for up in lineage.walk_upstream(node, depth=4):

        if up["type"] == "stream":

            evidence.setdefault("streams", []).append(

                consumer_lag(up["id"], since=fired_at - lookback))

        elif up["type"] == "warehouse":

            evidence.setdefault("tables", []).append(

                {"id": up["id"], "last_write": last_write_ts(up["id"]),

                 "row_delta": row_delta(up["id"], lookback)})

        elif up["type"] == "scheduled":

            evidence.setdefault("jobs", []).append(

                job_history(up["id"], runs=5))



    evidence["recent_changes"] = deploys_and_ddl(node, since=fired_at - timedelta(days=2))

    return evidence


recent_changes earns its place. A large share of production incidents are somebody's change, and a model given the deploy log alongside the telemetry finds that correlation reliably, which a tired human at 3 am often does not.

Constrain the Output, or It Will Write You an Essay

Free-form model output is unusable in an operational context. It cannot be graded, routed, or compared across incidents. So triage output is a fixed schema, and the prompt is explicit that abstaining is an acceptable answer.

Python
 
TRIAGE_SCHEMA = {
    "probable_cause_asset": "id from the provided lineage, or null",
    "reasoning": "max 3 sentences, cite specific evidence fields",
    "confidence": "high | medium | low",
    "contradicting_evidence": "what in the context argues against this",
    "suggested_first_check": "one command or query the on-call should run",
    "escalate_to": "owner team from lineage, or null",
}

SYSTEM = """You are triaging a data platform incident.
Use only the evidence provided. Do not infer components not in the lineage.
If the evidence does not distinguish between causes, set confidence to low
and name the check that would distinguish them.
Return only JSON matching the schema. No prose outside the JSON."""


The contradicting_evidence field was the single best addition. Forcing the model to argue against its own conclusion surfaces ambiguity that a confident one-line answer hides, and it gives the on-call engineer something to check rather than something to believe.

The suggested_first_check field is deliberately a check, not a fix. That distinction is the whole safety posture.

Grade It Before Anyone Trusts It

We ran this in shadow mode for a full quarter before it was allowed to write into the incident channel. Every page produced a triage output that went into a table, and when the incident was resolved, the actual root cause got recorded next to it.

SQL
 
select confidence,
       count(*) as n,
       avg(case when probable_cause_asset = actual_root_cause then 1 else 0 end) as hit_rate
from triage_evaluation
where resolved_at is not null
group by confidence;


The result was the thing that made the case internally. High-confidence predictions were right most of the time. Low-confidence predictions were right at close to chance, which is fine, because low confidence is the model telling you it is guessing. The failure worth worrying about is a system that is confidently wrong, and that is measurable, cheaply, before you depend on it.

If you cannot produce that table, you do not have an AI operations capability. You have a vibe.

What It Does Not Get To Do

It does not restart jobs. It does not scale clusters. It does not touch a production system. Not because auto-remediation is unthinkable, but because the cost asymmetry in a reporting platform is brutal: a wrong number that reaches a decision-maker is far worse than a delayed number, and an automated action taken on a misdiagnosed cause is an excellent way to produce a wrong number quietly.

Triage compresses time to understanding. Humans still decide, and humans still act. That boundary has not moved, and I am not in a hurry to move it.

The Same Shape Shows Up in Security Triage

The other place this pattern landed cleanly was application security. A scanner across a large portfolio produces far more findings than any team can act on, and the hard part is not detection; it is deciding which findings are reachable in a deployed configuration and which are noise in a dependency nobody calls.

The structure is identical. Assemble evidence, which here means the finding plus the call path plus the deployment context plus whether the affected code is actually on a request path. Constrain the output to a schema. Grade the prioritization against what the security team independently decided. Let it rank; never let it close.

What I Would Tell Someone Starting

Do the lineage work first. It is tedious, has no demo, and is the entire foundation. A model given a rich, accurate picture of your system and a narrow question performs well. The same model given an alert string and a broad question performs like a search engine that occasionally lies.

The winning move was never smarter answers. It was cutting the reconstruction phase out of every incident.

Engine Data (computing) large language model

Opinions expressed by DZone contributors are their own.

Related

  • Threat Modeling Context-Aware AI: A STRIDE Walkthrough
  • A Practical Pipeline for Identifying Sensitive Columns Before Test Data Masking
  • Building a Mortgage Agent With FRED Data, FastAPI, and LLM Tool Calling
  • Using LLMs to Automate Data Cleaning and Transformation Pipelines

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook