Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident
incopilot is an open-source Python CLI that automates the first 15 minutes of production incident triage — collecting logs, detecting failure patterns, and incidents.
Join the DZone community and get the full member experience.
Join For FreeEvery production incident follows the same painful ritual.
An alert fires at 2 am. An engineer wakes up, SSHs into a server, and begins the manual loop — pulling logs, scanning for errors, guessing what to check next. This loop can take 15 to 45 minutes before the real diagnosis even begins. Multiply that across every incident, every team, every month, and you have thousands of engineering hours lost every year to work that is repetitive, stressful, and largely automatable.
I've been on that on-call rotation. I know what it costs — not just in time, but in cognitive load, in missed context, and in the compounding pressure of an active incident. So I built incopilot: a CLI tool that automates the entire first-pass triage so engineers can skip straight to actual problem-solving. This post walks through the architecture, the design decisions, and exactly how to build it yourself.
Why the First 15 Minutes Are the Hardest
The first phase of any incident isn't debugging — it's searching. You don't know what broke, which service is responsible, or where to look. You're making guesses based on incomplete information under time pressure. This is exactly the kind of work AI handles well: pattern recognition across large volumes of text, rapid signal extraction, and structured summarization.
The key insight behind incopilot is simple: logs contain the answer almost every time. The problem is that engineers spend those first 15 minutes manually finding where the answer is buried. An automated tool can do that search in under 60 seconds.
Architecture
incopilot is built in Python and structured around four core modules that handle the full pipeline from log collection to report generation.
incopilot/
cli.py # argument parsing + console output
collectors.py # journalctl, docker logs, file, bundle
analyzer.py # pattern detection + line normalization
reporter.py # report.md / report.json generation
config.py # patterns, golden-signal map, safe-command listThe collectors layer pulls logs from multiple sources — systemd journal, Docker containers, or raw log files. The analyzer applies pattern detection to identify high-signal events: OOM kills, 5xx error spikes, disk exhaustion, network timeouts, and service restarts. The reporter maps findings to the Four Golden Signals (latency, traffic, errors, saturation) and generates a structured incident brief in both markdown and JSON format.
The design is deliberately read-only. incopilot never writes to your system, never restarts services, and never executes commands that modify state. This makes it safe to run during an active incident when the last thing you need is an automated tool making changes.
Setup
Getting started takes under two minutes:
git clone https://github.com/AutoShiftOps/incopilot.git
cd incopilot
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtQuick Test Without Real Services
You don't need a live environment to test incopilot. The repo includes a script that generates realistic sample logs so you can see the full pipeline working immediately:
python scripts/demo_generate_sample_logs.py
python -m incopilot file --path sample.log
ls out/This generates a sample log file with injected failure patterns, runs the analyzer, and outputs your first incident report. The report.md file is formatted for immediate paste into an incident doc or Slack message.
Running Against Real Systems
For systemd journal triage — the most common production use case:
python -m incopilot journal --unit nginx --since "30 min ago"For Docker container triage:
python -m incopilot docker --container my-api --since 1hFor incidents needing both sources simultaneously:
python -m incopilot bundle \
--unit nginx \
--container my-api \
--since-journal "30 min ago" \
--since-docker 1hWhat the Output Looks Like
Every run produces two files in the out/ directory. report.md is formatted for immediate use — paste it directly into your incident document, Slack channel, or PagerDuty note. report.json is structured for programmatic use — POST it to a webhook, attach it to a JIRA ticket, or pipe it into a downstream automation.
The report identifies detected failure patterns, maps them to the appropriate Golden Signal, lists the specific log lines that triggered each detection, and provides a plain-language summary of what the evidence suggests.
Design Decisions Worth Explaining
Why Python over Go or Bash? Python gives us the flexibility to integrate LLM summarization later without rewriting the tool. The current version uses pattern matching; the next version will use an LLM to generate the incident summary paragraph. Python makes that transition straightforward.
Why read-only? The tool is designed to be run by any on-call engineer regardless of their familiarity with the system. A read-only tool cannot make a bad incident worse. That constraint is a feature, not a limitation.
Why local execution? incopilot runs on the host or inside the container — no external API calls, no data leaving your environment. This matters for organizations with data residency requirements or security-sensitive production environments.
What to Improve Next
The current version is production-ready for pattern detection and report generation. The roadmap includes per-service pattern packs for nginx, postgres, Java, and Node.js applications, Slack and Teams webhook posting via a --webhook flag, unit tests with GitHub Actions CI, and an optional LLM summarization layer for human-readable incident narratives.
All of this is open source at https://github.com/AutoShiftOps/incopilot. PRs and issues welcome.
Published at DZone with permission of Sudhakararao Sajja. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments