Details
Why observing whether an AI system is running is no longer enough and how teams can determine whether its outputs are right.
Traditional observability connects metrics, traces, and logs to explain why distributed systems behave as they do. AI expands that responsibility. Teams must still monitor infrastructure and application health, but they must also evaluate model outputs, trace agent decisions and tool calls, measure token usage and cost, and protect new AI attack surfaces. Datadog’s Eddie Araujo explains why full-stack observability must now answer two distinct questions: Is the system running correctly, and is the answer itself correct?
Traditional dashboards can show healthy service-level objectives, available error budgets, low latency, and zero application errors while an AI system produces an incorrect answer.
The session’s budgeting-agent example completes every step successfully but adds only one of four transactions. Nothing in the conventional operational signals identifies the incorrect result.
Metrics show how a system changes over time, traces follow requests across services, and logs record individual events. Their value comes from connecting them across the entire environment.
As applications expand across cloud infrastructure, containers, serverless systems, databases, microservices, and AI components, correlating those signals becomes considerably more difficult.
AI is not a single component added to an application. Its stack can include GPUs, model-serving infrastructure, models, retrieval systems, data layers, orchestration, tools, and agents.
Each layer introduces distinct performance, cost, reliability, quality, and security questions that conventional application monitoring may not answer.
Traditional software changes can often be assessed through deterministic tests that clearly pass or fail. AI behavior is probabilistic, and a prompt or model change may appear operationally healthy while degrading output quality.
Without structured evaluations, teams may not discover a problem until a customer receives the wrong response, submits a support ticket, or silently loses trust.
Teams need to trace each agent step, tool call, and model response while evaluating the final output against defined quality criteria.
Those evaluations can cover factual correctness as well as organization-specific requirements such as brand alignment, appropriate language, and other measures relevant to the application.
The session presents four areas of AI observability:
GPU monitoring for infrastructure utilization and cost
Agent observability for applications built internally
Agent Console for third-party AI tools adopted by employees
AI Guard for risks such as prompt injection, data exfiltration, and unauthorized agent behavior
Together, these extend the traditional “what happened and why?” discipline across the AI stack.
Operational health and output quality are different
Practical implication: Do not treat successful requests, low latency, and zero errors as proof that an AI application is working correctly.
AI outputs need structured evaluation
Practical implication: Define measurable quality criteria for important use cases and evaluate production outputs against them.
Agent workflows need step-level tracing
Practical implication: Capture model calls, tool use, retrieval activity, orchestration steps, token consumption, latency, and final responses.
Production failures should improve future tests
Practical implication: When an incorrect output is found, convert that failure into a repeatable evaluation before releasing the fix.
Every AI layer has a different failure mode
Practical implication: Monitor infrastructure utilization, application behavior, output quality, third-party tool adoption, cost, and security separately while correlating them in one operational view.
AI usage must be discoverable before it can be governed
Practical implication: Inventory which AI tools employees use, who uses them, what they cost, and whether they produce measurable productivity gains.
Open instrumentation can remain part of the architecture
Practical implication: Teams using OpenTelemetry can retain open instrumentation while collecting AI-related data such as traces, token usage, latency, and evaluations.
Q: Why are traditional observability signals insufficient for AI?
A: Metrics, traces, and logs can establish that an AI workflow executed successfully, but they do not automatically establish that its final answer was accurate, appropriate, or useful.
Q: What is the difference between monitoring and observability?
A: As presented in the session, monitoring identifies what is happening, while observability helps explain why it is happening by connecting signals across the system.
Q: What does the AI stack include?
A: The session describes infrastructure and GPUs, model serving and access, retrieval and data systems, and the application layer containing agents and orchestration.
Q: What is agent observability?
A: Agent observability traces an AI application’s steps, tool calls, model responses, latency, token consumption, and other behavior. It also applies evaluations that measure whether the resulting output meets defined quality requirements.
Q: How can teams detect an AI answer that is wrong despite zero errors?
A: They need evaluations that score the response itself. These evaluations can measure correctness and other requirements specific to the application or organization.
Q: How should teams respond to an incorrect production output?
A: The session recommends detecting and diagnosing the failure, creating an evaluation that reproduces it, implementing a fix, and validating the change before redeployment.
Q: What should teams monitor beyond latency and error rate?
A: Possible measures include token usage and cost, correctness, hallucination-related failures, brand consistency, response language, tool behavior, and other application-specific quality standards.
Q: Can teams use OpenTelemetry for AI agent observability?
A: Yes. According to the speaker, agents instrumented with OpenTelemetry can still provide traces and related information such as token usage, latency, and evaluations.
Q: Can observability data be returned to an agent through MCP?
A: Yes. The speaker describes workflows in which agent traces are retrieved from the observability system and fed back to agents, potentially creating an automated feedback loop.
Q: What is the difference between internally built and externally adopted agents?
A: Internally built agents require detailed workflow tracing and output evaluation. Externally adopted tools such as coding assistants require organizational visibility into adoption, usage, cost, governance, and productivity.
Corey Noles: Hello, everyone, and welcome to today’s DZone webinar, “From Cloud to AI: What Full-Stack Observability Means Now.”
I’m Corey Noles, and I’ll be your host for today’s session with Datadog.
Before we begin, I have a quick housekeeping note. Today’s session is being recorded, and we’ll share the on-demand recording with registrants after the event.
If you have questions during the presentation, please submit them through the Q&A tab. We’ll address as many as possible during the Q&A portion at the end.
Let’s take a closer look at the challenge at the center of today’s discussion.
Enterprise technology environments are becoming more complex. Applications now span cloud and hybrid infrastructure, distributed services, and increasingly AI workloads that introduce new behaviors and dependencies across the technology stack.
As these environments become more connected, visibility across the full stack becomes increasingly important. Looking at individual systems or layers in isolation can make it harder to understand what is affecting performance, troubleshoot an issue, or see how a change in one area affects another.
Today, we’ll examine how AI is changing the observability challenge and what full-stack visibility needs to look like as enterprise environments continue to evolve.
To guide us through today’s discussion, I’m pleased to introduce Eddie Araujo, Senior Sales Engineer at Datadog.
Eddie, thank you for joining us. I’ll hand it over to you.
Eddie Araujo: Thank you, Corey. I’m happy to be here, and thank you to everyone joining us today.
I’d like to begin with a picture.
This is what a full-stack dashboard commonly looks like. We have our infrastructure, application traces, and logs in one place.
In this example, our SLOs are green. Our error budgets are healthy, and requests are flowing. According to every signal we have spent the past 15 years learning to trust, this system is healthy.
It could also be completely wrong.
The system could be serving confident, well-formed, and fast answers that happen to be false. Nothing on this dashboard would tell you that.
Everything here answers one question: Is it running?
Today, I want to discuss the question it cannot answer and why AI has made that question the one that matters most.
We’ll begin with what observability has always been for. We’ll then look at the surface it must cover today across cloud, applications, and AI. Next, we’ll examine where AI rewrites the rules.
We’ll finish with one question that ties everything together: not merely whether the AI is running, but whether it is right.
Let’s ground the discussion.
Stripped down to its fundamentals, observability involves taking the data you are already collecting and moving from “what is happening?” to “why is it happening?”
Monitoring tells you what is happening. Observability tells you why.
That distinction is the whole game. Keep it in mind because “Why is it happening?” is exactly the question AI will make more difficult to answer.
Until now, we have generally answered that question through three signals.
Metrics show the shape of the system over time: what is healthy, what is trending, and what may be about to break.
Traces show the path of a single request across every service it touches.
Logs provide the ground truth of what happened, event by event.
These are three views of one system.
The most important point is that the skill was never simply collecting these signals. We have been collecting them for a long time. The difficult part was connecting them.
When you connect and correlate these three signals, you get something like this.
This slide is busy, but that is the point. Full-stack observability covers a great deal of ground, and nobody can hold all this information in their head at once, at least not for very long.
Full-stack observability must now extend from code to customer and from development to business outcomes. Every surface produces signals continuously while also changing underneath you.
Connecting three signals was never the hardest part. Connecting them across this type of environment is.
This is what that complexity looks like at the service level.
Here, we have one request moving through the frontend, backend, databases, and approximately a dozen services to return one answer to a user.
Something appears to be wrong. Elasticsearch is producing errors at a rate of approximately 15.6%.
Is Elasticsearch the underlying problem, or is it struggling because the recommendations service upstream is failing at approximately 8% and placing additional pressure on it?
To answer that question, you must connect the metric on one node with the traces for the request and the logs from the affected service—all in real time while traffic continues flowing.
That is what connecting three signals means at scale. Multiply it by every request type the organization serves, and you have today’s architecture.
That architecture has not stopped growing.
We began with servers, followed by cloud, containers, serverless systems, and distributed microservices. Every era added more surface area.
Some SREs watching this may think this is the hardest part of the job.
I would say it is the hardest part of the job so far because we now have a new entry on the timeline.
Every previous era made the system larger. There was more to watch, but teams were still observing the same general kinds of behavior.
AI is different. It breaks some of the assumptions our tools were built on, changing what observability must do.
AI is not merely something bolted onto an application. It has a complete stack of its own.
At the bottom, you have infrastructure and GPUs providing raw compute.
Above that, you have the models, including how they are served and accessed.
Next are the retrieval and data layers feeding the models.
At the top is the application layer, where agents and orchestration connect everything.
That is a great deal of new surface area. Every layer is something teams now need to watch.
If that were the entire story, AI would simply be the next step on the infrastructure staircase—larger, certainly, but fundamentally the same type of problem.
That is only half the story. The other half changes everything.
Here, we’re looking at a budgeting agent. Someone gives it a straightforward instruction: calculate my total grocery spending by adding up all my purchases.
If we examine the workflow, we see that it ran through approximately 24 spans, including triage, PII detection, the budgeting agent, and a summarizer.
At a glance, there were no errors. Every step succeeded, and the system appears perfectly healthy.
Now look at the result.
There are four transactions: Whole Foods for approximately $84, Safeway for $112, Trader Joe’s for $63, and Costco for $147.
The agent’s answer says the total grocery spending is $84.32.
I would bet that something like this has happened to nearly everyone who has used an AI agent.
The agent took the first transaction, listed all four, added only one, and presented that value as the total. Its answer was wrong by more than $300.
It did not necessarily hallucinate or invent the transactions. Every amount it retrieved was real. It reasoned incorrectly and then stated the wrong answer with complete confidence.
The worst part is that nothing flagged the problem.
There were zero errors. Every span was green.
If I ran the workflow again, I might receive a different answer. That is what nondeterministic behavior can mean in production.
The system ran successfully, but the answer was wrong.
We should ask how that happened. How did a wrong answer pass through while everything being monitored remained green?
Traditional software delivery follows a familiar process. You make a change and run tests. Those tests pass or fail, and either way, the system tells you.
That feedback loop is the foundation of continuous integration, monitoring, alerting, and related practices. They assume the system can tell you when something is wrong.
Shipping AI changes that.
You modify a model or prompt and then run the application. Everything appears green, but how do you know whether the application worked or whether the agent is better?
The short answer is that you may not know immediately. Often, you do not know until customers tell you.
The feedback loop has not disappeared. It has moved to the worst possible place: after a customer has already received the incorrect answer.
The signal used to be a failing test before production. Now it may be an angry support ticket—or a customer who quietly stops trusting the application and never says anything.
The budgeting agent completed 24 green spans with zero errors. The only person who could initially identify that it was wrong was the user who requested a grocery total and received an answer that was incorrect by more than $300.
Every signal, dashboard, alert, and test we have built answers the same question: Is it running?
They do not necessarily answer the question that matters most in the AI era: Is the answer right?
That is the gap full-stack observability must now close.
There is good news: We have solved a similarly shaped problem before.
When cloud computing emerged, it made systems distributed and difficult to govern. Observability helped make cloud environments manageable and governable.
AI now needs similar treatment.
AI is already a serious production concern. Its risk surface has grown, its systems have become genuinely more complex, and spending has increased enough to attract the attention of finance teams.
This is no longer only a science project.
However, it resembles a class of problem we have addressed before, and the same discipline can help answer it.
Observability has always moved teams from “what happened?” to “why did it happen?” That has not changed.
The surface area has changed, but the job remains the same.
We still want to make and deploy changes with the confidence previously supplied by deterministic tests, established design patterns, and CI/CD guardrails.
AI components are unpredictable, however, and changes can be difficult to quantify. Teams need structured scoring and comparison tools to make informed decisions about improving performance and results.
Consider the broken feedback loop in which you change a prompt or model, run it, and do not know whether it improved.
You can rebuild that loop deliberately.
First, detect what is wrong in production, even when nothing produces a conventional error.
Next, diagnose the root cause across the agent’s different steps.
Then generate an evaluation that tests specifically for that failure so it cannot pass through as a healthy result again.
After that, fix the problem, release the change, and return it to production with greater confidence.
The process becomes: detect, diagnose, evaluate, fix, and ship. Then the cycle begins again.
The feedback loop that AI disrupted is deliberately reconstructed.
Under this model, the budgeting agent’s failure would be caught rather than passing silently.
AI is not one component; it is an entire stack. Every layer can fail differently and therefore requires an appropriate form of observability.
Starting at the bottom, we have GPU monitoring.
GPUs are often the most expensive infrastructure component and among the easiest resources to waste.
Are the GPUs saturated or sitting idle? Is a training run stalling because of the hardware or the workload?
At this layer, you monitor utilization and cost to ensure your most expensive infrastructure is performing useful work.
Next are the agents you build—your own applications. This is agent observability.
The budgeting agent belongs at this layer. Teams trace every step, tool call, and model response and run evaluations that can catch incorrect answers before customers do.
This is the layer that answers, “Is it right?” It is also the layer that would catch the budgeting agent’s incorrect total.
Then there are the agents your organization does not build but purchases and adopts.
These may include Claude, Copilot, Cursor, ChatGPT, or other tools used by employees.
Datadog Agent Console is intended to help answer who is using these tools, what they cost, and whether they are making people more productive.
You cannot govern what you cannot see. Most organizations do not have complete visibility into the AI tools currently in use. This layer provides that visibility.
Woven through these layers is AI Guard.
This provides security and trust protections against risks such as prompt injection, data exfiltration, and an agent being persuaded to perform an action it should not perform.
Every layer we have discussed creates a new attack surface that teams must monitor.
We now have four areas with different failure modes and different observability requirements, brought together in one platform.
The same discipline that has always moved us from “what?” to “why?” now extends across the complete AI stack.
That is what full-stack observability means once AI becomes part of the environment.
We began with a dashboard that was entirely green while the system’s output was wrong.
This entire discussion has focused on closing the gap between “Is it running?” and “Is it right?”
The technology surface has continued to grow and will keep growing. But the job has not changed, nor has the discipline used to accomplish it.
Observability made cloud governable. AI now needs the same treatment.
Q&A
Corey Noles: Thank you, Eddie. That was a helpful explanation of how full-stack observability is expanding as enterprise environments become more distributed and AI assumes a larger role.
We’ll now move into the Q&A portion. You can continue submitting questions through the Q&A tab.
Our first question is: If we instrument our agents with OpenTelemetry, can we still obtain token cost and evaluation data without giving up open standards?
Eddie Araujo: You do not need to give up open standards to receive that information.
You can instrument your agents using OpenTelemetry and still receive the same quality of tracing, including token, latency, and evaluation data.
Corey Noles: Next question: If I’m building my own agents, can I feed observability data back into them—perhaps through MCP—so the agents can act on it?
Eddie Araujo: Absolutely. Some organizations are doing that today.
You can develop agents that generate traces, retrieve those traces from the observability system, and then feed that information back to the agents.
That allows you to begin automating parts of the process and create an automated feedback loop.
Corey Noles: One more question: For LLMs in production, what are people alerting on beyond latency and error rate?
Token cost and hallucinations seem important, but how do you monitor those the way you monitor something such as P95 latency?
Eddie Araujo: I love that question because it gets to one of the foundational questions about AI: What can we do with it?
Teams may monitor latency as well as the correctness of responses.
I have also seen customers evaluate brand alignment. Does the response remain consistent with the organization’s standard branding?
Another example is language. A customer may ask a question in one language while the model responds in English or another language.
You may want to track whether the response language matches the language used in the original question.
The possibilities are extensive. It ultimately depends on what matters to your organization. Customers are already evaluating and monitoring criteria like these today.
Corey Noles: Eddie, thank you for the presentation and for answering these questions.
If we could not address your question today, the Datadog team can follow up. Be sure to explore the additional resources shared during the webinar as well.
As we conclude, several themes are worth remembering.
AI adds another layer of complexity to already distributed enterprise environments, making isolated monitoring increasingly difficult to rely upon.
Full-stack observability now needs to connect what is happening across cloud infrastructure, applications, and AI workloads. Teams need that context to identify issues faster and understand how each part of the environment affects the others.
Eddie, before we close, how can attendees learn more or continue the conversation with Datadog?
Eddie Araujo: If today’s discussion sparked your interest and you would like to continue the conversation, scan the QR code or visit the link displayed beneath it.
One of our representatives will contact you, answer your specific questions, and continue the discussion from there.
Corey Noles: Thank you, Eddie, and thank you to everyone who joined us today.
Keep an eye on your inbox for the on-demand recording. We hope to see you at a future DZone webinar.
Presenters:
Eddy Araujo
Senior Enterprise Sales Engineer at Datadog
Join Now for More Content & Events
For event and sponsorship inquiries, please email: [email protected]
