From DevOps to AIOps: How Agentic AI Tamed Our Multi-Substrate Chaos

The Mess We Started With

When I took over the team, we had two substrates and a long list of problems that cut across both.

On-prem ran on VMware. The core application was vendor-licensed, but over the years the team had layered so much custom code on top of it that the vendor could barely support it anymore. Upgrades were risky. Every change required manual configuration. Dependencies were undocumented. Libraries were old enough that half the team was nervous about touching them. There was no real monitoring — we found out about problems when users complained.

AWS was in better shape, but not by much. The team was reactive by habit. Tickets came in and sat in a queue. Developers waited days for environment provisioning. On-call rotations were brutal because there was no good way to correlate what was happening across both substrates during an incident. The on-prem side had its own telemetry, AWS had its own, and nobody had a clean view of both at once.

The team was capable. They were just spending the majority of their time on toil — manual interventions, ticket triage, configuration changes that should have been automated years ago. That is the environment we decided to change.

What DevOps Actually Looked Like

We called it DevOps. In practice, it was good engineers doing repetitive work because the systems weren't capable of doing it themselves.

Provisioning a new environment meant opening a ticket, waiting for someone with the right access to pick it up, manually running through a checklist, and hoping nothing had changed since the last time the runbook was updated. Runbooks were the real source of truth — and they were wrong half the time.

Incident response meant someone getting paged, logging into two different observability tools, trying to mentally correlate what they were seeing across VMware and AWS, and making judgment calls with incomplete information at 2 am.

IT dependency was constant. Developers couldn't self-serve anything meaningful. Every access request, every config change, every environment spin-up went through a human. That queue never got shorter.

This wasn't a people problem. The team was doing exactly what the system required them to do. The system was the problem.

Why We Moved to Agentic AI

We had tried the standard fixes. Better runbooks. Improved monitoring. More automation scripts. These helped at the margins but didn't change the underlying dynamic: humans were still the execution layer for work that didn't require human judgment.

The shift to agentic AI was a deliberate decision to change that. An AI agent doesn't just surface a recommendation — it acts. It perceives its environment, reasons over a goal, executes a sequence of actions, observes the results, and adapts. Within a defined authority boundary, it owns the work.

The key distinction from traditional AIOps tooling is autonomy. We had anomaly detection before. We had dashboards. What we didn't have was something that could take a detected anomaly and resolve it without waking someone up.

What We Built and What Changed

Self-Service Request Handling

The ticket queue was the most visible symptom of how reactive our operation was. Developers submitted requests and waited. The requests weren't complex — environment provisioning, access grants, config changes, dependency updates — but each one required a human to pick it up, context-switch, and execute.

We built an agentic request layer in front of our service catalog covering both substrates. A developer requests a dev environment on AWS — the agent parses the intent, validates against policy, provisions via Terraform, runs validation checks, and delivers a verified environment with an audit trail. A request that touches the on-prem VMware substrate follows the same flow, adapted to that environment's provisioning model.

Requests that used to sit in a queue for a day or more now resolve in minutes. Developer self-service went from a goal to a reality. The ticket queue dropped significantly. The engineers who used to service that queue are doing more interesting work.

Cross-Substrate Incident Response

Incidents were hard because the blast radius of a problem rarely stayed on one substrate. Something degrading on the VMware side would show up as anomalous behavior on the AWS side, and the on-call engineer had to stitch that together manually with two separate observability tools.

Our incident response agent ingests signals from both substrates continuously. When an anomaly appears, it correlates across both environments, maps the probable fault tree, and executes initial remediation — service restarts, traffic rerouting, config rollbacks — before a human is involved. If it hits a decision it isn't authorized to make, it escalates with a structured brief that tells the on-call engineer exactly what it found and what it did.

Mean time to detect dropped materially. Mean time to resolve dropped further. The number of incidents that require a human for resolution is a fraction of what it was. The on-call rotation is no longer brutal.

Proactive Monitoring and Drift Detection

The on-prem environment had no real monitoring. Problems surfaced through user complaints. We set that as a baseline, but we went further — we deployed a drift-detection agent that continuously compares the actual state of both substrates with the desired state defined in our IaC. When it finds drift, it remediates automatically for low-risk changes and flags for human review for anything higher risk.

The shift from reactive to proactive operations didn't happen because we wrote better runbooks. It happened because we deployed something that watches continuously and acts when it sees a problem, rather than waiting for a human to notice.

Governance: What the Agents Can and Cannot Do

Every agent operates within an Authority Envelope — a defined set of actions it can take autonomously, the conditions under which it must escalate, and an immutable audit log of every action it takes and why.

Irreversible actions — production data changes, security group modifications, anything touching the vendor-licensed on-prem application's core configuration — require a human approval gate. The agent prepares the action and presents its reasoning. A human authorizes it.

This boundary matters. The goal isn't to remove humans from operations. It's to make sure humans are applied to decisions that actually require judgment, and agents handle everything else.

We also red-team the agents regularly — testing edge cases, contradictory signals, and misclassification scenarios. These exercises built more confidence in the system than any amount of documentation could.

What Changed for the Team?

The toil that used to consume the majority of engineering hours is largely gone. The ticket queue no longer defines the team's day. The 2 am context switching between two observability tools is mostly gone. Developers self-serve. Incidents resolve faster and with less human intervention.

What's left is the work that actually requires engineers: improving the agents, designing for resilience, building new capabilities, making architectural decisions that compound over time.

The team is more engaged. The work is more interesting. That's not incidental — it's the point.

Where This Goes Next

The on-prem VMware environment is a constraint we're still working within. Agents help manage it, but the longer-term path is modernization — reducing the custom code surface area, retiring the dependencies that make the vendor relationship difficult, and moving more workload to AWS where the operational model is cleaner.

The agentic platform accelerates that work too. With agents handling day-to-day operations, the team has capacity to work on the modernization that was always the right answer but never had priority over keeping the lights on.

The move from DevOps to AIOps is not about replacing engineers. It's about changing what engineers spend their time on. Agentic AI handles the execution. Engineers handle the judgment. When you get that division right, the whole operation gets faster, more reliable, and more sustainable — without adding headcount.

That's the case for agentic AI. Not as a technology trend. As a practical answer to a real operational problem most enterprise teams are already living with.