Testing, Deployment, and Maintenance Resources

DZone's Featured Testing, Deployment, and Maintenance Resources

Member Spotlight: Srinivas Chippagiri

By Dominique Pugh

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here. More

Designing a Polling Pipeline for Rate-Limited APIs Without Losing Data

By CARTER WANG

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records. I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them. None of this is exotic. It is the set of things I wish had been in one place before I wrote the first version. Failure Mode 1: The Fixed Interval The naive poller runs on a fixed schedule — every 60 seconds, every source, forever. This is wrong in both directions simultaneously. High-volume sources produce more records between polls than a single page returns, so you lose everything past the page boundary. Low-volume sources return nothing on the vast majority of calls, burning quota that the high-volume sources needed. With a shared quota across sources, the fixed interval guarantees you are under-sampling exactly where the data is. The fix is to let observed yield drive the interval. If the last poll came back full, poll sooner. If it came back empty, back off. TypeScript interface PollState { sourceId: string; intervalMs: number; consecutiveEmpty: number; } const MIN_INTERVAL = 30_000; const MAX_INTERVAL = 1_800_000; // 30 min function nextInterval(state: PollState, recordsFound: number, pageSize: number): number { // Page came back full — we are probably truncating. Poll aggressively. if (recordsFound >= pageSize) { return MIN_INTERVAL; } if (recordsFound > 0) { // Productive but not saturated. Ease toward the current rate. return clamp(state.intervalMs * 0.8, MIN_INTERVAL, MAX_INTERVAL); } // Empty. Back off geometrically, but bounded. const factor = 1.5 ** Math.min(state.consecutiveEmpty, 6); return clamp(MIN_INTERVAL * factor, MIN_INTERVAL, MAX_INTERVAL); } function clamp(n: number, lo: number, hi: number): number { return Math.min(hi, Math.max(lo, n)); } The condition worth highlighting is recordsFound >= pageSize. A full page is not a success signal — it is evidence that records may exist beyond what you fetched. Treat it as a truncation warning and either paginate immediately or shorten the interval. Systems that treat a full page as "great, got everything" lose data proportional to source velocity, which means they lose the most from the sources that matter most. After deploying adaptive intervals, our total request volume dropped by roughly 40% while records captured went up, because quota moved from silent sources to active ones. Failure Mode 2: The Watermark Boundary The standard incremental pattern stores the timestamp of the newest record seen and requests everything after it: SQL SELECT * FROM source WHERE created_at > :watermark ORDER BY created_at This loses records, for three independent reasons. Clock skew. The upstream timestamp is assigned by their server, not yours. Distributed writers mean created_at is not guaranteed to be monotonic in the order records become visible to the API.Visibility lag. A record can be written at T but not appear in query results until T+2s due to replication or index lag. Poll at T+1s, advance the watermark past T, and that record is never returned.Ties. Multiple records sharing the same timestamp, split across a page boundary, silently drop the remainder when you use a strict >. The fix is to rewind the watermark by a safety margin and deduplicate downstream: TypeScript const OVERLAP_MS = 120_000; // tune to observed upstream lag function queryWindow(watermark: number): { since: number; until: number } { return { since: watermark - OVERLAP_MS, until: Date.now(), }; } This deliberately re-fetches records you already have. That is the point — you trade a small amount of redundant bandwidth for the guarantee that a record has to be missed by two consecutive polls to be lost permanently. Size OVERLAP_MS From measurement, not intuition. Log the delta between upstream created_at and your own ingest time, take the 99th percentile, and double it. Ours sat around 40 seconds at p99, so 120 seconds gives real headroom without meaningful cost. Failure Mode 3: Deduplication That Isn't Overlap windows only work if dedup is correct, and ID-based dedup handles less than people assume. Upstream records get edited. A post whose body changes after ingestion keeps its ID, so an INSERT ... ON CONFLICT DO NOTHING on the ID silently discards the update. Meanwhile, some sources reuse or re-issue identifiers across deletion, so the ID alone is not always a stable key either. Hash the fields that constitute a meaningful change, and store both: TypeScript import { createHash } from "node:crypto"; function contentHash(record: SourceRecord): string { // Only fields whose change means "this record is materially different." // Volatile counters (score, comment count) are excluded on purpose. const canonical = JSON.stringify({ id: record.id, title: record.title ?? "", body: record.body ?? "", author: record.author ?? "", }); return createHash("sha256").update(canonical).digest("hex"); } SQL INSERT INTO records (source_id, external_id, content_hash, payload, first_seen_at, last_seen_at) VALUES ($1, $2, $3, $4, now(), now()) ON CONFLICT (source_id, external_id) DO UPDATE SET content_hash = EXCLUDED.content_hash, payload = EXCLUDED.payload, last_seen_at = now(), revision = records.revision + 1 WHERE records.content_hash IS DISTINCT FROM EXCLUDED.content_hash; Two details carry most of the weight. The WHERE clause means the overlap window costs nothing when nothing changed — no write amplification, no spurious updated_at churn, and downstream consumers watching for changes don't get woken up by re-fetches. And excluding volatile counters from the hash is what makes this viable at all. Include a score that ticks every few seconds and every single poll registers as a change, which defeats the purpose and turns your overlap window into a write storm. Failure Mode 4: Retry Storms and Undetected Gaps When a rate-limited API returns 429, the instinct is to retry. Retry immediately, and you extend the penalty. Retry on a fixed delay across N workers, and they resynchronize into a thundering herd that hits the limit together, forever. Most rate-limited APIs tell you what you need in response headers. Read them instead of guessing: TypeScript interface RateLimitState { remaining: number; resetAt: number; } function parseRateLimit(res: Response): RateLimitState | null { const remaining = res.headers.get("x-ratelimit-remaining"); const reset = res.headers.get("x-ratelimit-reset"); if (remaining === null || reset === null) return null; return { remaining: Number(remaining), // Some APIs send seconds-until-reset, others an absolute epoch. Normalize. resetAt: Number(reset) > 1e9 ? Number(reset) * 1000 : Date.now() + Number(reset) * 1000, }; } async function backoff(attempt: number, res?: Response): Promise<void> { const retryAfter = res?.headers.get("retry-after"); if (retryAfter) { const ms = Number(retryAfter) * 1000; if (Number.isFinite(ms)) return sleep(ms); } // Full jitter — see AWS Builders' Library on backoff. const ceiling = Math.min(60_000, 1000 * 2 ** attempt); return sleep(Math.random() * ceiling); } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); Full jitter matters more than the backoff curve. Exponential backoff with a deterministic delay keeps clients synchronized; randomizing across the whole interval is what actually spreads load. Retry-After, When present, it is authoritative — honor it over your own calculation. The subtler problem is that these failures are invisible. A source that throws 429 for six hours produces no errors in your application logs if the poller catches and backs off correctly. It just quietly returns nothing, and your data has a hole in it that nobody notices until someone asks why a report looks wrong. Detect gaps explicitly rather than inferring them from error rates: TypeScript async function detectGaps(sourceId: string): Promise<Gap[]> { const { rows } = await db.query(` SELECT window_end AS gap_start, LEAD(window_start) OVER (ORDER BY window_start) AS gap_end FROM poll_log WHERE source_id = $1 AND status = 'ok' AND window_start > now() - interval '48 hours' ORDER BY window_start `, [sourceId]); return rows .filter(r => r.gap_end && r.gap_end - r.gap_start > GAP_THRESHOLD_MS) .map(r => ({ sourceId, from: r.gap_start, to: r.gap_end })); } This requires logging every poll attempt with its requested window and outcome — a table most people skip because it feels like overhead until the first time it tells you about four hours of missing data you would otherwise never have found. Feed detected gaps into a low-priority backfill queue that runs when quota is available. What Actually Matters If you take three things from this: Log the window, not just the result. Without a record of what range each poll requested, gaps are undetectable. This is the single highest-value thing to add to an existing poller.Overlap deliberately, dedup by content. A safety margin plus content-hash dedup converts "we might have missed something" into "a record must be missed twice to be lost."Treat a full page as a warning. It is the most common silent truncation point in production pollers, and it is one comparison to check for. None of these are difficult. They are just easy to omit, and their absence produces a system that looks healthy while quietly dropping records. I work on the ingestion layer at SignalMelo, which polls public discussion sources continuously — the patterns above came out of production incidents there. Disclosure: it's a commercial product I help build. More

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

By Soham Sengupta

The Man Who Invented Selenium Also Built Its Replacement

By Arun Kulkarni

Diagnosing Test Failures at Scale: Correlating Execution Evidence Across Ephemeral Runners

By Anil Reddy Putlacheru

A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

The previous article put Genkit flows in Kubernetes pods with senro and let jev, a decision model, route a support ticket. The decisions there were small: which team, how urgent, send or hold. This one hands jev something bigger to judge: software. A software factory takes a feature request and produces a Go module: a design, a plan, the code, a green build, passing tests, and a pull request description. Gemini writes every one of those. jev decides, at every gate, whether the work moves on, goes back, or stops for a human. Each stage runs as a pod, each stage is a Genkit flow, and the whole thing is a few hundred lines of Go. Everything below ran against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5 and a local kind cluster. The code is in genkit-k8s-flows-jev-software-factory. Why a Factory Needs a Judge, Not a Second Writer The usual way to review model-written code is to ask another chat model. It works, up to a point. The reviewer writes a paragraph, you parse the paragraph, the paragraph is sometimes wrong in the same way the code is, and a well-written pull request description can talk the reviewer into "looks good to me." A decision model does not write. It takes a state (the request, the plan, the code, the test log) and answers typed questions with probabilities, in a few hundred milliseconds. There is no prose to parse and nothing to be talked into: BuildGreen is a number, and a red test log drags it down whatever the pull request says. Chat model as reviewerDecision model as reviewerVerdictA paragraph you parseChoice[CodeVerdict]: ship, rework or human, with a probability eachEvidenceMentioned, maybeA Noul per question: covers the request, tests are meaningful, build is greenFeedback for the next roundWhatever it felt like writingEach question that scored "yes" is one line of feedbackLatency and costSeconds, thousands of output tokensMilliseconds, tokens for the state onlyCan be persuaded by the authorYesNo: it never reads instructions, only state The split that falls out of this is the whole design: Gemini builds, jev decides. And senro is what gives each build step a pod, a workspace, a log, a retry policy, and a secret, and each decision a place in the graph. What We Are Building A feature request goes in. A reviewed Go module comes out, or a brief for a human saying why not. The factory is three senro pipelines and a small state machine over them: RunStepWhereWhoWhat it leaves behinddesigncleancoordinatornonean empty workspace with request.md in itdesigndesignpodGemini and jev, in a loopdesign.json, DESIGN.md and one record per review rounddesignplanpodGeminiplan.json, PLAN.mdimplementimplementpodGeminithe module's filesimplementvet, testpodgo vet, go testvet.log, test.log and their exit codesimplementreviewpodjevreview.json: ship, rework or humanreworkfix, then vet, test, reviewpodGemini, Go, jevthe same, after a fixfinishpr then publishpod, coordinatorGeminiPR.md and the module under out/<request>/srcfinishbrief then notifypod, coordinatorGeminiBRIEF.md for a human jev decides three times. Inside the design step, it reviews each design Gemini writes, and the loop revises until jev says ready, says reject, or three rounds are up. After test, it reviews the code with the logs in hand and picks ship, rework or human. And the state machine in main turns each verdict into the next senro run, with rework bounded at two. What You Need Go 1.27 or newer, Docker, kind, and kubectlGEMINI_API_KEY and TYPESAFE_API_KEY in your environmentthe senro CLI if you want the terminal UI (optional) Shell go get github.com/firebase/genkit/[email protected] go get github.com/xavidop/[email protected] go get github.com/xavidop/mamori Step 1: A Cluster and an Image With a Go Toolchain The cluster setup is the one from the previous article: kind, a namespace, a ServiceAccount with the permissions senro's Kubernetes executor needs (including get on pods/exec), and three environment variables, since senro never reads your kubeconfig. The repo's scripts/cluster-up.sh does it and source scripts/cluster-env.sh exports them. The one difference is the image. A func step needs only sh and tar in the pod, but this factory also runs go vet and go test as plain command steps, so every step uses the official Go image, pinned to its digest as senro requires: Go const stepImage = "docker.io/library/golang:1.27-alpine@sha256:f2c872bcee949b7be9c490efd02b48a8a2fbf947cdf533f1c58525ec9d8170fd" func pods() senro.ExecutorTarget { return k8s.Pod(stepImage, k8s.Namespace("factory"), k8s.Resources( map[string]string{"cpu": "500m", "memory": "512Mi"}, map[string]string{"cpu": "2", "memory": "2Gi"}, )) } Your pipeline binary is still not in that image. senro cross-compiles it and sends it into each pod, exactly as before. Step 2: The Design Loop, With jev Inside the Flow The first gate is the one that saves the most money: a bad design produces bad code, a green build and a confident pull request. So the design is reviewed before anything is planned, and the review is a loop inside one Genkit flow. The questions are a Go type. Every yes/no question is one a revision can act on: a "yes" becomes a line of feedback for the next round. Go type DesignVerdict string func (DesignVerdict) Criteria() map[DesignVerdict]string { return map[DesignVerdict]string{ "ready": "Complete, unambiguous, scoped to the request, and testable as described", "revise": "The right direction, with gaps a revision can close", "reject": "The request is too vague or too large to design from; a human has to clarify it first", } } type Risk int func (Risk) Levels() []string { return []string{"Low", "Medium", "High"} } type DesignReview struct { Verdict systemonex.Choice[DesignVerdict] `json:"verdict" jsonschema_description:"Is this design ready to implement?"` RequestClear systemonex.Noul `json:"request_clear" jsonschema_description:"Does the request say what to build precisely enough that two engineers would build the same thing?"` ScopeCreep systemonex.Noul `json:"scope_creep" jsonschema_description:"Does the design add things the request did not ask for?"` Underspecified systemonex.Noul `json:"underspecified" jsonschema_description:"Is any component described too loosely to implement without guessing: missing fields, signatures, edge cases or error behaviour?"` MissingTests systemonex.Noul `json:"missing_tests" jsonschema_description:"Does the test plan leave any requirement of the request untested?"` Risk systemonex.Score[Risk] `json:"risk" jsonschema_description:"How risky is implementing this design as written?"` } And the loop. Gemini designs; jev reviews; if the verdict is revise, Gemini gets the previous design plus the feedback and tries again: Go f.Design = genkit.DefineFlow(g, "designFeature", func(ctx context.Context, request string) (DesignResult, error) { var res DesignResult feedback := "" for round := 1; round <= maxDesignRounds; round++ { prompt := "Request:\n" + request if round > 1 { prompt += "\n\nYour previous design:\n" + asJSON(res.Design) + "\n\nReviewer feedback:\n" + feedback } d, _, err := genkit.GenerateData[Design](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You are a senior Go engineer writing a short design for exactly what the request asks. "+ "Stay inside the request: no extra features. Components are the Go types and functions you will create. "+ "The test plan names concrete, deterministic test cases. Be brief."), ai.WithPrompt("%s", prompt)) if err != nil { return res, err } res.Design = *d out, resp, err := genkit.GenerateData[DesignReview](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has a feature request and a design written for it."), ai.WithPromptParts(ai.NewDataPart(map[string]any{"request": request, "design": res.Design}))) if err != nil { return res, err } // jev's verdict, then our rules on top. verdict := string(out.Verdict.Choice) var notes []string if out.RequestClear.Probability < 0.4 { verdict = "reject" // nobody should design from a request two engineers would read differently } if out.ScopeCreep.Probability >= 0.5 { notes = append(notes, "remove everything the request did not ask for") } if out.Underspecified.Probability >= 0.5 { notes = append(notes, "specify every component fully: fields, signatures, edge cases and error behaviour") } if out.MissingTests.Probability >= 0.5 { notes = append(notes, "add a test case for every requirement in the request") } if verdict == "ready" && len(notes) > 0 { verdict = "revise" } res.Rounds = append(res.Rounds, DesignRound{Round: round, Verdict: verdict, Ready: out.Verdict.Probabilities["ready"], Feedback: notes /* ... */}) res.Verdict = verdict if verdict != "revise" { break } feedback = fmt.Sprintf("The reviewer rated the design %q (probability of ready %.2f). Do the following:\n- %s", verdict, out.Verdict.Probabilities["ready"], strings.Join(notes, "\n- ")) } return res, nil }) Two things to notice. The verdict is jev's, but the rules around it are yours: a request that two engineers would read differently is rejected whatever the design looks like, and a design with a flagged problem is not ready even if jev leaned that way. And the feedback is built from the questions, not from prose: a decision model never writes a sentence, so the sentences live in your code, one per question, and the next round gets exactly the ones that applied. The whole loop is one flow, so it is one senro step, one pod, one log, with every round recorded in design.json. Step 3: Plan, Implement, and Let Go Have Its Say The plan and the implementation are ordinary Gemini calls with typed outputs: a Plan with files and steps, then a Code with every file in full. The step writes the files into the workspace, refusing anything that would escape it: Go type Code struct { Files []File `json:"files"` Notes string `json:"notes,omitempty"` } func writeCode(ctx senro.Ctx, code Code) error { root := wsPath(ctx, ".") for _, f := range code.Files { rel := filepath.Clean(f.Path) if rel == "." || strings.HasPrefix(rel, "..") || filepath.IsAbs(rel) || strings.HasPrefix(rel, factoryDir) { return fmt.Errorf("refusing to write outside the module: %q", f.Path) } path := filepath.Join(root, rel) if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { return err } if err := os.WriteFile(path, []byte(f.Content), 0o644); err != nil { return err } fmt.Fprintf(ctx.Stdout(), "wrote %s (%d bytes)\n", rel, len(f.Content)) } return nil } Then the compiler and the tests run, as command steps in the same pod image, with no model anywhere near them: Go func goStep(w *senro.WorkflowBuilder, id, after, cmd string, ws *senro.WorkspaceRef) *senro.StepBuilder { script := "mkdir -p .factory; " + cmd + " > .factory/" + id + ".log 2>&1; echo $? > .factory/" + id + ".exit; " + "cat .factory/" + id + ".log; exit $(cat .factory/" + id + ".exit)" return w.Step(id, exec.Command("sh", "-c", script)). Needs(after). Mount(ws.At("/repo", senro.RW)). WorkDir("/repo"). Env("HOME", "/tmp"). Env("GOCACHE", "/tmp/gocache"). Env("GOFLAGS", "-mod=mod"). Env("GOPROXY", "off"). Env("GOTOOLCHAIN", "local"). Env("CGO_ENABLED", "0"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). ContinueOnError() } goStep(code, "vet", "implement", "go vet ./...", ws) goStep(code, "test", "vet", "go test -count=1 ./...", ws) Each one leaves its output and exit code under .factory/ in the workspace, so the reviewer can read them. GOPROXY=off and GOTOOLCHAIN=local keep the pod honest: the request said standard library only, and the pod cannot download anything to pretend otherwise. And ContinueOnError is the important one: a red test is a failed step, but the review still runs. jev gets to see why the build failed, which is exactly the evidence a rework needs. Step 4: The Code Review Gate The second gate reads everything a human reviewer would open: the request, the plan, every source file, and what go vet and go test printed. Go type CodeVerdict string func (CodeVerdict) Criteria() map[CodeVerdict]string { return map[CodeVerdict]string{ "ship": "Builds, tests pass, does what the request asked, no obvious defects", "rework": "Fixable problems: failing or missing tests, a missed requirement, sloppy code", "human": "Something a model should not decide: security, data loss, or the request was misunderstood", } } type Quality int func (Quality) Levels() []string { return []string{"Poor", "Acceptable", "Good", "Excellent"} } type CodeReview struct { Verdict systemonex.Choice[CodeVerdict] `json:"verdict" jsonschema_description:"Should this change ship?"` Quality systemonex.Score[Quality] `json:"quality" jsonschema_description:"How good is the code?"` CoversRequest systemonex.Noul `json:"covers_request" jsonschema_description:"Does the implementation do everything the request asked for?"` TestsMeaningful systemonex.Noul `json:"tests_meaningful" jsonschema_description:"Do the tests exercise the behaviour the request cares about, rather than just compile?"` BuildGreen systemonex.Noul `json:"build_green" jsonschema_description:"According to the logs and exit codes, did go vet and go test both pass?"` } The flow sends the whole ReviewInput as a data part, reads the answers, and applies the factory's rules on top. A red build never ships, whatever the overall verdict: Go f.Review = genkit.DefineFlow(g, "reviewChange", func(ctx context.Context, in ReviewInput) (Review, error) { out, resp, err := genkit.GenerateData[CodeReview](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has the feature request, the implementation plan, every source file, "+ "and the output and exit codes of go vet and go test. Exit code 0 means success."), ai.WithPromptParts(ai.NewDataPart(in))) if err != nil { return Review{}, err } r := Review{ Verdict: string(out.Verdict.Choice), Confidence: out.Verdict.Confidence, Quality: out.Quality.Label(), CoversRequest: out.CoversRequest.Probability, TestsMeaningful: out.TestsMeaningful.Probability, BuildGreen: out.BuildGreen.Probability, Model: systemonex.ResponseInfo(resp).Model, } if in.VetExit != 0 { r.Reasons = append(r.Reasons, "go vet failed") } if in.TestExit != 0 { r.Reasons = append(r.Reasons, "go test failed") } if r.CoversRequest < 0.6 { r.Reasons = append(r.Reasons, "the implementation does not cover the request") } if r.TestsMeaningful < 0.5 { r.Reasons = append(r.Reasons, "the tests do not exercise the requested behaviour") } if out.Quality.Level() == 0 { r.Reasons = append(r.Reasons, "code quality is poor") } if r.Verdict == "ship" && len(r.Reasons) > 0 { r.Verdict = "rework" } return r, nil }) The Reasons are what the fix flow gets, together with the logs. Again: the decision model decides, your code says it in words. Step 5: Three Pipelines and a State Machine The module under construction lives in a persistent workspace, so every run of one request works on the same tree. senro insists a persistent workspace is bounded, and refuses one without both bounds: Go func repo() *senro.WorkspaceRef { return senro.Workspace("factory-repo", senro.Scope(senro.ScopePersistent), senro.MaxAge(24*time.Hour), senro.MaxSize(256<<20)) } The three pipelines are short. Every AI step gets the same shape through a helper, with only the secrets it needs: Go func aiStep(s *senro.StepBuilder, ws *senro.WorkspaceRef, secrets ...string) *senro.StepBuilder { s = s.Mount(ws.At("/repo", senro.RW)).Timeout(stepTimeout).Retry(3, retry.OnInfra()) for _, name := range secrets { s = s.SecretEnv("SENRO_"+name, name) } return s } func buildPipeline(outDir string, rework bool) *senro.Pipeline { name, fn := "implement", "factory/implement" if rework { name, fn = "rework", "factory/fix" } p := senro.New(name) ws := repo() code := p.Workflow("code", senro.On(pods())) aiStep(code.Step(name, senro.Func(fn, NoParams{})), ws, "Gemini") goStep(code, "vet", name, "go vet ./...", ws) goStep(code, "test", "vet", "go test -count=1 ./...", ws) aiStep(code.Step("review", senro.Func("factory/review", NoParams{})), ws, "TypeSafe").Needs("test") handoff := p.Workflow("handoff", senro.Needs("code"), senro.On(senro.Local())) handoff.Step("collect", senro.Func("factory/collect", DirParams{Dir: outDir})). Mount(ws.At("/repo", senro.RO)) return p } finish declares both endings and lets a run parameter pick one, the same senro.When pattern as the previous article: Go func finishPipeline(outDir, reason string) *senro.Pipeline { p := senro.New("finish") ws := repo() ship := p.Workflow("ship", senro.On(pods()), senro.When(senro.ParamIs("verdict", "ship"))) aiStep(ship.Step("pr", senro.Func("factory/pr", NoParams{})), ws, "Gemini") publish := p.Workflow("publish", senro.Needs("ship"), senro.On(senro.Local()), senro.When(senro.ParamIs("verdict", "ship"))) publish.Step("publish", senro.Func("factory/publish", DirParams{Dir: outDir})). Mount(ws.At("/repo", senro.RO)) escalate := p.Workflow("escalate", senro.On(pods()), senro.When(senro.ParamIs("verdict", "human"))) aiStep(escalate.Step("brief", senro.Func("factory/brief", BriefParams{Reason: reason})), ws, "Gemini") // ...and a notify workflow on the coordinator that collects BRIEF.md return p } And main is the state machine. Each run ends with a record jev produced, and the record picks the next run: Go state, reason, reworks := "design", "", 0 for { switch state { case "design": if err := run(designPipeline(request, out)); err != nil { exit(err) } var d DesignResult read(out, "design.json", &d) if d.Verdict == "ready" { state = "implement" } else { state, reason = "human", fmt.Sprintf("the design review ended %q after %d round(s)", d.Verdict, len(d.Rounds)) } case "implement", "rework": // A red vet or test is a failed step, so the run is partial; the // review still ran (ContinueOnError) and its verdict is what counts. if err := run(buildPipeline(out, state == "rework")); err != nil && !partial(err) { exit(err) } var r Review read(out, "review.json", &r) switch r.Verdict { case "ship": state = "ship" case "rework": if reworks >= *maxRework { state, reason = "human", fmt.Sprintf("jev asked for rework %d times: %v", reworks+1, r.Reasons) } else { reworks++ state = "rework" } default: state, reason = "human", fmt.Sprintf("the code review ended %q: %v", r.Verdict, r.Reasons) } case "ship", "human": if err := run(finishPipeline(out, reason), senro.WithParams(senro.Params{"verdict": state})); err != nil { exit(err) } return } } Every arrow in the state diagram is one of those cases, and every run is a directory under runs/ with its plan, its events and one log per step. Step 6: Run It The first request is a token-bucket rate limiter, the kind of package every service ends up with: Plain Text # Feature request: a token-bucket rate limiter package We need a small, dependency-free Go package `ratelimit` (module path `github.com/acme/ratelimit`) that other services can import. - `New(rate float64, burst int) *Limiter`: `rate` is tokens added per second, `burst` is the bucket capacity. - `(*Limiter) Allow() bool`: takes one token if available, otherwise returns false without blocking. - `(*Limiter) AllowN(n int) bool` for batches. - Safe for concurrent use from many goroutines. - Time must be injectable so tests are deterministic (no sleeping in tests). - Standard library only. Tests included, run with `go test ./...`. Shell go run . -request requests/token-bucket.md The first run is the design. Read back with senro attach --run <id> --ui=plain: Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.44) request_clear=0.62 scope_creep=0.36 underspecified=0.56 missing_tests=0.38 risk=Low design stdout | - specify every component fully: fields, signatures, edge cases and error behaviour design stdout | round 2: jev-1.13.0 decided "ready" (p_ready 0.57) request_clear=0.57 scope_creep=0.49 underspecified=0.28 missing_tests=0.34 risk=Low design succeeded plan stdout | module github.com/acme/ratelimit, 3 files, 11 steps plan succeeded collect succeeded run succeeded Two rounds. jev found the first design underspecified (0.56), the feedback said so in one sentence, and the second design named every field, signature, and edge case. The ready probability moved from 0.44 to 0.57, and the design that came out of it is the one Gemini implemented; this is its component list in DESIGN.md: Plain Text ## Components - type Limiter struct { mu sync.Mutex; rate float64; burst float64; tokens float64; last time.Time; now func() time.Time } - concurrency-safe state tracking ... - func New(rate float64, burst int) *Limiter - public constructor delegating to newWithClock using time.Now; clamps negative rate to 0.0 and negative burst to 0 ... - func newWithClock(rate float64, burst int, now func() time.Time) *Limiter - internal constructor enabling time injection for tests; ... panics if now is nil. - func (l *Limiter) Allow() bool - convenience method returning l.AllowN(1) ... - func (l *Limiter) AllowN(n int) bool - thread-safe token deduction; edge cases: returns true immediately if n <= 0 without modifying state; returns false immediately if n > int(l.burst); advances tokens by elapsed.Seconds() * l.rate capped at l.burst ... The plan turns that into three files and eleven steps, and the second run executes it. Plain Text implement stdout | wrote go.mod (42 bytes) implement stdout | wrote ratelimit.go (1371 bytes) implement stdout | wrote ratelimit_test.go (3172 bytes) implement succeeded vet succeeded test stdout | ok github.com/acme/ratelimit 0.009s test succeeded review stdout | jev-1.13.0 decided "ship" (confidence 0.66) quality=Excellent covers_request=0.87 tests_meaningful=0.95 build_green=0.98 review succeeded collect succeeded run succeeded Gemini wrote the module, go vet and go test ran in the pod against it, and jev read the three files and the logs and said ship, with the build at 0.98 and the tests at 0.95. The module it reviewed: Go // AllowN reports whether n tokens can be consumed immediately. func (l *Limiter) AllowN(n int) bool { l.mu.Lock() defer l.mu.Unlock() if n <= 0 { return true } if float64(n) > l.burst { return false } curr := l.now() elapsed := curr.Sub(l.last) if elapsed > 0 { l.tokens += elapsed.Seconds() * l.rate if l.tokens > l.burst { l.tokens = l.burst } l.last = curr } if l.tokens >= float64(n) { l.tokens -= float64(n) return true } return false } with six tests beside it: clamping, a nil clock panic, burst capacity, AllowN edge cases, deterministic replenishment against an injected clock, and a race test over concurrent goroutines. No sleeping anywhere, as the request demanded. The third run has both endings declared and runs one: Plain Text pr stdout | # Implement token-bucket rate limiter package pr stdout | ## Summary pr stdout | Introduces `github.com/acme/ratelimit`, a thread-safe, zero-dependency token-bucket rate limiter. ... pr stdout | ## Tests pr stdout | ok github.com/acme/ratelimit 0.009s pr stdout | ## Risk pr stdout | Low. Mutex contention under high concurrency is mitigated by keeping critical sections limited to basic in-memory arithmetic. ... pr succeeded publish stdout | published 3 files to .../out/token-bucket/src publish succeeded brief skipped_condition notify skipped_condition run succeeded Three senro runs, seven pods, two Gemini designs, two jev design reviews, one implementation, one vet, one test, one jev code review and one pull request, in a little over a minute and a half. out/token-bucket/ holds every record and the module under src/. Step 7: When jev Says No The second request is the kind every team gets: Plain Text # Request The app feels slow lately, especially in the afternoon. Can you make it faster? Nothing specific, just whatever you think is best. Thanks. Shell go run . -request requests/make-it-faster.md Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "reject" (p_ready 0.13) request_clear=0.05 scope_creep=0.87 underspecified=0.70 missing_tests=0.79 risk=Low design stdout | - remove everything the request did not ask for design stdout | - specify every component fully: fields, signatures, edge cases and error behaviour design stdout | - add a test case for every requirement in the request design succeeded plan stdout | design ended "reject", nothing to plan plan succeeded collect succeeded run succeeded Gemini did what a chat model does with a vague request: it invented a feature, in this case pprof routes and a latency middleware. jev put the probability that the request is clear at 0.05, and the factory's rule turned that into reject in the first round, before a plan, a line of code or a second model call. The finish run then takes the other ending: Plain Text pr skipped_condition publish skipped_condition brief stdout | ### What was asked brief stdout | A user submitted a vague performance complaint stating the app "feels slow lately, especially in the afternoon" and asked to "make it faster" without providing specific endpoints, metrics, or reproduction steps. brief stdout | ### Why it stopped brief stdout | The design was rejected on Round 1 (Readiness: 13%, Clarity: 5%). The factory attempted to introduce Go `net/http/pprof` diagnostic routes and custom in-memory latency middleware, triggering a rejection for high scope creep (87%) ... brief stdout | ### What a human should decide brief stdout | 1. **Clarify the issue:** Contact the user to identify affected routes, user workflows, and target latency SLOs. brief stdout | 2. **Authorize observability work:** Determine if the team should prioritize a dedicated ticket for production profiling/metrics ... brief succeeded notify succeeded run succeeded The brief is written by Gemini, from the record jev produced. The two models keep their jobs even in the failure path. A Request jev Cannot Be Talked Into The third request is a glob matcher: *, **, ?, character classes with ranges and negation, backslash escapes, and an error for an unterminated class. It is precise, it is testable, and the factory never got to write it: Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.68 scope_creep=0.35 underspecified=0.49 missing_tests=0.32 risk=Medium design stdout | round 2: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.65 scope_creep=0.40 underspecified=0.34 missing_tests=0.22 risk=Medium design stdout | round 3: jev-1.13.0 decided "revise" (p_ready 0.47) request_clear=0.59 scope_creep=0.66 underspecified=0.33 missing_tests=0.20 risk=Medium design stdout | - remove everything the request did not ask for design succeeded Gemini's design transpiles the glob into a Go regular expression. jev never called it ready, the ready probability never crossed 0.5 in three rounds, and in two of those rounds none of the specific questions crossed the line either, so the feedback Gemini got was the generic "tighten the approach". The loop cannot converge on a complaint it has no question for, and after three rounds the factory stops and writes the brief: Plain Text ## What was asked Implement a standard-library Go package (`github.com/acme/glob`) providing `Compile(pattern string) (*Glob, error)` and `(*Glob) Match(path string) bool`. ... ## Why it stopped The design review failed after 3 rounds, ending in a "revise" verdict (`p_ready` plateaued at 0.46). Automated reviewers flagged persistent specification risks and scope creep regarding the chosen approach (transpiling globs into Go `regexp`), specifically around complex regex metacharacter escaping and separator handling in negated character classes. ## What a human should decide 1. **Architecture:** Approve regex transpilation (accepting edge-case escaping risks) or mandate a dedicated AST/rune-based glob parser. 2. **Path boundaries:** Explicitly define exact semantics for `**` edge cases and `/` exclusions inside negated classes. That brief is the factory working as designed. A decision model only answers the questions you ask, and when it keeps saying no without any of them firing, the honest move is to stop and ask a person, with the evidence attached. The fix for the next version of the factory is one more field on DesignReview: a question about the approach itself, so that "this should be a parser, not a regex" becomes a sentence Gemini can act on. What the Combination Gives You ConcernWho handles itDesign, plan, code, fix, pull request textGemini, through genkit.GenerateData and GenerateTextIs the design ready, should the code shipjev, through the same GenerateData call with a decision typeFeedback for the next roundYour code, one sentence per question jev answered "yes"Compiling and testinggo vet and go test, as command steps in the pod, with no model involvedA red build reaching the reviewerContinueOnError on the Go stepsThe module between runsA bounded persistent workspaceWhich ending runssenro.When(senro.ParamIs("verdict", ...))Credentialsmamori resolves, senro delivers a file per pod, per stepEvidenceruns/<id>/ per run, plus .factory/ records that travel with the code Conclusion A software factory is not one big agent. It is a handful of small, typed calls with a judge between them, and the judge is the part most people leave out. jev makes that part cheap: a design review is one call that answers six typed questions in a few hundred milliseconds, and the questions are Go fields you choose. Put the loop inside a Genkit flow, the flow inside a senro step, the step in a pod, and the verdict in a run parameter, and the factory is a state machine you can read in one screen. The code: genkit-k8s-flows-jev-software-factoryThe previous article: Genkit flows on Kubernetes, routed by a decision modelDecision models in Genkit Go and Flowssenro's Kubernetes executor and persistent workspaces

By Xavier Portilla Edo

CORE

Your Event Storming Is Not Your Software Design

The Policy That Started the Argument We were looking at a purple policy between ListingSold and StopListingFee. Read aloud, the flow expressed a simple business expectation: when a listing is sold, the applicable recurring listing fee should stop. The interesting question was where that policy belonged. If Billing owns it, Billing receives the fact that a listing was sold and determines what that means for its fee model. Listing reports the sale without naming the operation Billing should perform. If Listing owns the coordination, it requests that Billing stop the fee, which means Listing knows that capability exists. A third option is a separate process manager that receives the sale event and coordinates the response. At the level of a simple event-policy-command sequence, these versions can look almost identical. Once ownership is made explicit, the differences matter. Billing depends on the sale event’s contract in the first version. The coordinating component depends on Billing’s command contract in the others. Billing still validates changes to its own fee model in every version. Responsibility for detecting an incomplete outcome, such as a fee that continues accruing after a sale, also needs an explicit owner. We were still discussing the business rule, but we were now deciding how software responsibilities and contracts would support it. This was already a discussion about where coupling belongs and how it is managed. We had moved into architectural reasoning without changing the business outcome we wanted. I later discussed this on LinkedIn and encountered the objection that EventStorming is not software design at all. I disagree, but the objection helped me distinguish two claims. EventStorming can host software design. It does not follow that the arrangement of a board is sufficient justification for the boundaries of the resulting system. This article is about that gap. EventStorming Already Includes Design The official EventStorming material explicitly recognizes Big Picture, Process Modeling, and Software Design formats. The patterns collection also addresses collaborative design activities. EventStorming is not limited to collecting domain events on a long wall of paper. The title therefore makes a narrower claim than it might suggest. “Your Event Storming” means the model your group produced, with its particular clusters, policies, and arrows. That model may contain exploratory ideas, agreed decisions, and reasons for those decisions. What cannot be assumed is that the notation alone captures all the reasoning needed to justify the resulting software structure. A policy between two notes expresses a relationship: when this happens, that response should follow. Unless ownership and the relevant assumptions are also made explicit, the sequence leaves important questions unanswered. Who owns the rule? Which contract does the response depend on? What happens if the response is rejected or never completes? A Software Design session can address those questions. It can expose invariants, clarify responsibilities, and help participants explore alternative models. The policy discussion above is an example of that work. The problem is automatic inference: treating a cluster as sufficient reason for a service, a purple note as a prescribed message handler, or a temporal sequence as an execution topology. This is related to the architectural decisions a domain model leaves open. Modeling can establish constraints and support architectural choices without uniquely determining how the system should be deployed or operated. Those choices can be discussed during the workshop. They still need reasons beyond the appearance of the board. Same Flow, Different Ownership For this example, Listing owns the listing lifecycle, and Billing owns the fee lifecycle. A sold listing should no longer accrue the applicable recurring fee. The precise rule, especially when stopping accrual becomes effective, still needs clarification. For now, consider three ways to allocate responsibility for the response. Alternative A: Billing owns the reaction. Listing records the sale and makes a ListingSold event available across an agreed boundary. Billing receives that fact, interprets it using its own rules, and invokes the appropriate fee behavior. Listing does not need to name StopListingFee. Billing now depends on the meaning and structure of the event it consumes. The event publisher needs an explicit compatibility policy, although that does not require exposing Listing’s internal domain event unchanged. Billing keeps the relationship between sale facts and fee behavior within its own model. That is useful when the rule belongs to Billing and evolves with its charging policies. The design still needs to account for delivery failures, processing failures, and missed reactions. Suppose Listing records a sale but the corresponding fee remains active. Who detects that discrepancy, and who resolves it? Billing may own both the reaction and reconciliation, but that responsibility must be explicit. Successful publication alone does not establish that the business outcome is complete. Alternative B: Listing owns coordination. Listing requests StopListingFee from Billing. Listing therefore knows a capability Billing exposes. Billing remains responsible for validating the request and protecting its own rules. The request could be synchronous or queued. Either way, the coordinating component needs to distinguish acceptance from completion and decide how to handle rejection, an unknown outcome, or a response that does not arrive within the expected time. This option is justified only if Listing’s responsibility genuinely includes coordinating the fee change. Emitting the original event is not enough to establish that ownership. Nor does the statement “the sale is complete only when the fee stops” settle the question by itself. That requirement establishes a business outcome; deciding which component should be accountable for achieving it is still design work. Listing-owned coordination becomes more plausible when Listing already owns the broader completion process and can manage it without absorbing Billing’s fee rules. If it starts calculating charges or reproducing Billing’s eligibility logic, the coordination boundary is leaking. Alternative C: A process manager owns coordination. A separate component tracks the cross-component outcome and requests work from the responsible participants. Listing still owns the listing lifecycle. Billing still owns fee behavior. The process manager owns the progress of the interaction. This helps when a sale triggers several participants, steps have deadlines, or unresolved cases need escalation. The manager can record that the sale was observed, that a fee-stop request was issued, and whether completion was confirmed. That visibility has a cost. The manager needs durable state, recovery behavior, and a policy for its own failures. Its commands may use the same capabilities Billing already exposes; participants do not necessarily need a new interface simply because a process manager is calling them. For a single reaction, introducing this machinery may be harder to justify than making Billing’s existing responsibility explicit. DesignMessage crossing the boundaryContract knowledgeAccountability for completionBilling reactsListingSold eventBilling understands the published sale contractBilling can own reaction and reconciliation; this must be agreedListing coordinatesStopListingFee command and an appropriate completion signalListing understands Billing’s exposed capabilityListing tracks the outcome if coordination belongs within its responsibilityProcess manager coordinatesSale event, fee-stop command, and completion or failure signalsThe manager understands the contracts required by the processThe manager tracks progress and escalation; its own recovery must also be designed These alternatives allocate responsibility differently. They do not divide neatly into a coupled design and a decoupled one. Contract knowledge is one form of dependency. Billing understands the sale event in Alternative A. Listing understands Billing’s command contract in Alternative B. The process manager understands the contracts needed to coordinate Alternative C. Runtime availability is a separate concern. A synchronous call may require the responder to be available immediately. Buffered communication can allow the sender to proceed while the receiver is unavailable, provided the system can retain and eventually process the pending work. That changes the timing of the dependency rather than removing it. Release coordination is another concern. Consumers and publishers need compatible contracts, but they do not necessarily need simultaneous deployments. The relevant question is how those contracts can evolve without silently changing the business meaning. Moving the policy changes some of these dependencies. It does not automatically reverse all of them. Nor does its position prescribe a transport: an event does not require Kafka, and a command does not require HTTP. What would favor one option? If Billing independently owns the rule connecting a sale to fee accrual, Alternative A is a strong candidate. If Listing already owns a broader completion process that legitimately includes requesting the fee change, Alternative B may fit. If the interaction needs its own lifecycle, deadlines, and recovery across several participants, Alternative C becomes easier to justify. The business flow is an input to that decision. Ownership, contract obligations, and failure handling determine whether the chosen design can support it. The Missing Question Is Often About Time The sequence ListingSold → StopListingFee leaves an important question unresolved: when should accrual stop? The policy note alone does not specify that. It could be the effective sale time, when the sale is recognized by the business. It could be when Listing records the change, or when Billing processes it. Those moments can differ. If the contract carries timestamps, their meanings need to be explicit. For example, a published event might distinguish the effective time from the time the sale was recorded: JSON { "type": "ListingSold", "eventId": "...", "listingId": "...", "effectiveAt": "...", "recordedAt": "..." } These fields make the distinction visible. They do not establish which timestamp Billing should trust or what it should do with it. If Billing calculates accrual using an authoritative effective time, delayed delivery need not change the amount ultimately owed. But the system may already have issued an invoice, collected payment, or shown an outdated balance. It still needs rules for adjustment and for what the customer sees before the correction completes. A timestamp helps preserve the meaning of the fact. It does not remove the consequences of learning about it late. Suppose the business accepts a delay before the balance reflects a sale, provided the final calculation uses the correct effective date. An asynchronous reaction may satisfy that requirement. Suppose instead that no fee-related charge may be issued after the sale is accepted. That requires a stronger coordination or validation mechanism than receiving a sale event later. Merely replacing an event with a queued command does not provide that guarantee either. Corrections expose further decisions. If the sale was recorded incorrectly, should accrual resume? From which date? If an older sale notification arrives after a correction, how does Billing recognize that it no longer represents the applicable state? These questions concern the business meaning of the update as much as its delivery. This is where business outcomes beyond aggregate boundaries become relevant. A process spanning Listing and Billing does not automatically require one aggregate or one distributed transaction. Billing can protect a local invariant, such as refusing to calculate accrual beyond its recorded stop date, while the broader process still fails to establish that date correctly. The cross-component outcome needs explicit completion and recovery semantics. Compensation is another business operation, with its own rules and potential failures, rather than a database rollback. Eventual consistency does not by itself establish that the relevant rule was interpreted correctly or that unresolved work will be detected and repaired. Durable publication, retries, and idempotent handling support the design. They cannot supply a business rule the participants have not agreed on. A Boundary Still Needs a Reason An event on the board is not automatically an integration event. ListingSold may initially describe a fact inside the Listing model. Exposing that fact to another component creates a contract that needs deliberate semantics and an evolution policy. The published representation may differ from the internal event. Likewise, a cluster of activities does not establish an aggregate merely because the notes appear together. The grouping must be examined against the rules that must remain consistent. An aggregate does not prescribe a microservice, and a bounded context does not prescribe an independent deployment. These distinctions are part of why Domain-Driven Design does not select an architectural style. The ownership alternatives also exist inside a monolith. Listing and Billing could be modules in one application, with Billing reacting to an in-process event or receiving an explicit request through a module interface. The questions about rule ownership and contract knowledge remain. The execution guarantees may be quite different, however. A synchronous handler participating in the same database transaction can fail together with the originating operation. A background handler in the same application may run after that transaction commits and require durable delivery and recovery. Sharing a deployment does not settle those choices, just as separating deployments does not settle ownership. An independent deployment therefore needs its own justification. Billing might have a materially different scaling profile, need isolation from listing traffic, or require a separate security boundary. Its team might need genuine release autonomy. A runtime constraint might be difficult to satisfy within the existing application. These are reasons to evaluate separation, rather than automatic instructions to introduce it. The costs are concrete. An interaction that was previously local may now time out or complete without the caller receiving confirmation. Contracts need to evolve while different versions coexist. Observability must follow a business outcome across processes. Data previously updated within one transaction may require a different consistency strategy. Independent deployment can reduce some coordination costs while introducing others. I would not impose a fixed sequence of aggregates first, contexts second, services last. Domain learning and architectural constraints inform each other. A scaling or compliance requirement can change where responsibilities execute or force us to reconsider a proposed partition. It does not automatically change the meaning of the business model. When a boundary changes, the team should be able to explain which concern required the change. What I Would Record Beside the Wall I would capture a short decision record while discussing each significant policy, rather than leave all the reasoning for someone to reconstruct after the workshop. First, who owns the rule connecting the fact to the response? In this example, does Billing own the rule relating a sale to fee accrual, or does a broader business process require that outcome? Billing owns the fee mutation in either case. The separate question is who initiates the response and tracks its completion. Next, which contract crosses the boundary? Who maintains it, and what changes can its consumers tolerate? A note on the wall becomes an integration commitment only when the participants agree on the meaning and obligations it carries. The record should also identify what must remain consistent, the acceptable delay between steps, and the effective time used when information arrives late. It should say who detects an incomplete outcome, who resolves it, and what counts as evidence of completion. Acceptance scenarios make those decisions harder to leave vague. The official Extract Acceptance Tests pattern connects design sessions with more precise expectations about behavior. For this example, I would include a sale received after an invoice was issued, a correction received before an older sale event, and a fee-stop request whose completion cannot immediately be confirmed. The expected result in each case needs to come from the business rule. The record should remain short enough to maintain. Its purpose is to preserve the reasons behind the model, including decisions deliberately left open. Six months later, another engineer should be able to distinguish an agreed constraint from a convenient assumption. Closing The discussion began with ListingSold, a purple policy, and StopListingFee. Asking who owned the policy led us to contracts, coordination, consistency, and recovery. That was software design taking place through EventStorming. The workshop can help a team make those decisions, and the board can record them. What the event sequence cannot provide on its own is the justification for a particular allocation of responsibilities or a particular deployment boundary. Billing might own the reaction. Listing might coordinate it. A process manager might track a broader outcome. Choosing among them requires knowing the rule, the timing requirements, and who is accountable when the expected response does not happen. The wall can show the flow. Your design must explain who owns each decision and what happens when the flow breaks. References and Further Reading Introducing EventStormingEventstorming PatternsExtract Acceptance TestsWhere Does Software Architect Fit in Domain-Driven Design?Modern Software Architecture (Part 2): Coupling Is Where Architecture Actually HappensDomain-Driven Design Beyond Aggregate BoundariesDomain-Driven Design Is Not an Architecture

By Alireza Rahmani Khalili

CORE

Managing Technical Debt in AI-Generated Code

AI-generated pull requests are moving from experimental workflow to routine software delivery. Coding agents can now plan changes, edit multiple files, run tests, and submit pull requests with limited human intervention. A 2025 study of 567 Claude Code pull requests across 157 open-source projects found that 83.8% were eventually merged, and about 55% of merged changes required no further modification. A broader 2026 study analyzed more than 302,000 verified AI-authored commits from 6,299 repositories. The practical question is therefore no longer whether generated code can enter production. The harder question is what happens after the merge, when readability problems, architectural drift, latent defects, and future maintenance costs become part of the codebase. Faster Generation Changes the Review Equation AI assistance does not automatically imply lower quality code. A GitHub randomized controlled study involving 202 experienced developers found that Copilot-assisted implementations performed better on the study's functional tests and received modestly higher ratings for readability, reliability, maintainability, and conciseness. That evidence matters because technical debt should not be treated as an unavoidable property of machine-generated code. Production repositories, however, create conditions that controlled coding tasks cannot fully reproduce. The 2026 large-scale study of AI-authored commits identified 484,366 introduced issues where 89.3% were code smells, and more than 15% of commits attributed to every assistant studied introduced at least one detectable issue. Most importantly, 22.7% of tracked AI-introduced issues were still present at the repository's latest analyzed revision. The same research also found that AI-authored commits can remove existing smells, showing that generated code can both create and repay debt. That distinction changes the engineering problem. The central risk is not generation itself but generation speed combined with weak validation. DORA's 2025 research describes AI as an amplifier that magnifies existing organizational strengths and weaknesses. Earlier DORA analysis also associated increased AI adoption with larger change batches, slower review, and weaker delivery stability. Faster code production therefore raises the value of disciplined review rather than reducing it. Debt Begins at the Merge Boundary Technical debt becomes an organizational liability when a change is accepted into a maintained branch. The model that produced the code cannot own an incident, negotiate a migration, preserve a service level objective, or explain an architectural compromise six months later. Accountability must therefore follow the repository and the component, not the mechanism that generated the diff. This makes the pull request author field an increasingly weak representation of ownership. A generated change may carry an agent identity, a bot account, or a human account that delegated most implementation work. None of those identities answers the operational question of which engineering group is responsible for the behavior after deployment. Repository ownership rules provide a stronger boundary. GitHub CODEOWNERS can automatically request reviewers for designated paths, and protected branch settings can require approval from a code owner before merge. Required status checks can additionally prevent merging until defined CI checks pass. A policy can remain simple: Plain Text /src/main/java/com/acme/payments/ @payments-platform /src/main/java/com/acme/identity/ @identity-platform The important property is not the syntax. The important property is that an AI-generated payment change still reaches the team accountable for payment correctness, while an identity change still reaches the team accountable for authentication and authorization. Provenance may influence review depth, but provenance should never replace domain ownership. Human Review Becomes More Valuable, Not Less Higher generation throughput can create pressure to automate review with another model. AI review is useful for summarization, repetitive checks, suspicious patterns, and localized suggestions, but available evidence does not support treating it as equivalent to accountable human review in every context. A 2026 study covering 278,790 review conversations across 300 mature open source projects found that human reviewers exchanged 11.8% more review rounds when handling AI-generated code than human-written code. Human reviewer suggestions were adopted at a much higher rate than AI-agent suggestions, 56.5% versus 16.6%, and more than half of unadopted AI suggestions were either incorrect or addressed through alternative fixes. Another study analyzing more than 22,000 comments from AI-based GitHub review actions found substantial variation in effectiveness and reported that concise, contextual, hunk-level feedback was more likely to produce code changes. Current GitHub documentation explicitly states that Copilot code review does not replace human code review. GitHub also supports configurations in which Copilot approvals can count toward merge requirements, making repository policy an important governance decision rather than a tooling default. For critical components, AI review is better positioned as an additional signal before accountable approval, not as the entity carrying final ownership. Quality Gates Must Follow Risk A generated pull request should not receive weaker controls because tests pass. Passing tests establish only the behavior covered by those tests. Technical debt often appears elsewhere as duplicated logic, boundary violations, unnecessary abstractions, missing failure handling, insecure defaults, excessive coupling, or code that satisfies the immediate task while conflicting with local design conventions. CI can convert some of those conventions from reviewer memory into executable policy. An AI-assisted label, for example, can trigger additional checks without requiring a separate delivery pipeline: YAML ai-change-gate: if: contains(github.event.pull_request.labels.*.name, 'ai-assisted') steps: - run: ./gradlew test integrationTest - run: ./gradlew check - run: ./scripts/verify-architecture.sh The architecture check can encode repository-specific constraints such as forbidden dependencies, package boundaries, or service-layer access rules. The AI label is only a routing signal where the underlying controls should be based on change risk. A generated documentation correction does not require the same scrutiny as an authorization change, while a human-written authorization change should receive the same high-risk checks. GitHub rulesets support required checks and required reviews as merge conditions, allowing these policies to be enforced rather than documented as optional guidance. The Useful Metric Is Rework Counting generated lines, completed pull requests, or acceptance rate measures production volume, not maintenance quality. A more meaningful evaluation connects AI-assisted changes to downstream engineering cost. Review rounds, escaped defects, rollbacks, post-merge rework, issue survival, architecture violations, and time spent modifying recently merged code reveal whether higher implementation throughput is creating future work. This framing also avoids a false binary between "AI code" and "human code." The large-scale debt study found both issue introduction and issue removal in AI-authored commits. The relevant objective is therefore not minimizing AI involvement, but minimizing unmanaged debt while preserving useful productivity gains. Ownership should remain attached to the team that accepts and operates the change, while automated gates provide evidence that acceptance criteria were actually satisfied. Conclusion AI-generated pull requests change who produces code, but they do not change who must maintain software after deployment. Technical debt cannot be delegated to a model, hidden behind a bot identity, or treated as the future responsibility of an unspecified reviewer. The durable ownership boundary is the engineering team responsible for the affected component, supported by required human approval, executable quality gates, repository ownership rules, and measurements that expose downstream rework. AI can accelerate implementation and can even remove existing debt, but merge authority must remain tied to accountable engineering ownership. When generation becomes nearly instantaneous, disciplined acceptance becomes the scarce resource, and that acceptance is where responsibility for technical debt belongs.

By Uthej Mopathi

CORE

Designing the Runtime Layer for Stateful AI Agents: Where Agent Substrate Fits

Introduction: The Layer Nobody Designs AI agents are evolving from simple prompt-response systems into stateful, long-running applications that can reason, execute tasks, and interact with external systems. Yet, most agent architectures are driven by models, tools, memory, and orchestration — not runtime infrastructure that can run agents reliably. Unlike traditional applications, stateful AI agents need execution context to run (and recover from failures), concurrent tasks, and the ability to resume workflows without losing progress if they fail, etc. These capabilities cannot be provided by prompts or orchestration frameworks alone — there needs to be a runtime layer. This is where Agent Substrate becomes relevant. It provides a runtime foundation for stateful agent execution so that developers can separate the intelligence of the agent from the infrastructure that runs it (aka runtime). A closer look at stateful AI agents reveals why a dedicated runtime layer is essential, the architectural challenges it must address, and how Agent Substrate fits into the broader agentic AI technology stack. What Makes Agent Workloads Different AI agents might look like traditional workloads, but their execution patterns challenge the fundamental assumptions of application infrastructure. Request-driven services or batch jobs cannot run forever; agents can make random decisions, interact with other systems, and change their own state continuously. Five characteristics that make their runtime requirements fundamentally different. Long-lived, mostly waiting: An agent task can take minutes, hours, or days, and most of that time is spent waiting on a model, a tool, or a human. Holding a process open for the whole duration is wasteful and fragile. Requirement: Execution must be able to pause, release resources, and resume.Non-deterministic: The same input can take a different path each run. You can’t recover by re-running the code and expecting the same result, so what the agent did and decided has to be recorded as it happens. Requirement: Durable records of steps and outcomes, not just final state.Side effects on the real world: Agents send emails, move money, and modify systems. A naive retry after a crash can repeat an action that already happened. Requirement: Know exactly which steps completed, and make retries safe.State is the product: Memory, plans, and open tasks accumulate and are read and rewritten constantly. This isn’t a cache you can throw away; losing it means losing the agent. Requirement: State that is durable, versioned, and structured enough to tell current from stale.Many actors, shared state: Agents act for users, hand work to other agents, and collaborate with humans. Without boundaries, they overwrite each other, and nobody can say who decided what. Requirement: Isolation, access control, and an audit trail. Stateful AI agents need the durability of a database, the recoverability of a workflow engine, and the isolation of a multi-tenant platform, all integrated into a coherent execution environment. While traditional infrastructure offers these capabilities independently, agent workloads demand them together. This is the architectural gap that an agent runtime layer must address. The Lifecycle of a Stateful AI Agent Traditional applications run in predictable request-response cycles, but stateful AI agents run through dynamic long-running workflows. Their execution involves multiple reasoning steps, tool calls from the outside world, human approvals, and periods of inactivity. A typical stateful agent lifecycle involves six stages: Initialize: Receive a task, establish execution context, and create an initial state.Execute: Perform reasoning, make decisions, invoke tools, and update execution state.Persist: Save progress, intermediate results, and relevant context at durable checkpoints.Suspend: Pause execution while waiting for an external event, tool response, or human approval.Resume or Recover: Continue from the last valid checkpoint when the required event arrives or recoverable failures occur, avoiding duplicate side effects.Complete: Finalize the task, record the outcome, and retain relevant execution history for auditability. Importantly, this lifecycle is not strictly linear. An agent may repeatedly execute, persist, suspend, and resume before reaching completion. Failures may also trigger retries, recovery, or escalation. The runtime layer is responsible for managing these transitions reliably, ensuring that agent execution remains durable, observable, and recoverable without requiring the application developer to implement the lifecycle infrastructure from scratch. Why Traditional Infrastructure Is Not Enough Traditional infrastructure provides many of the building blocks needed to run AI agents, but no single piece of it covers the entire lifecycle of a stateful agent. The table below shows where traditional infrastructure falls short. Need Typical tool Why it is not enough for agents Preserve execution progress Stateless service Handles requests, but does not inherently retain agent state or progress across steps Isolated execution Containers Provide process isolation, but not durable execution or recovery Asynchronous processing Queue Moves work reliably, but does not manage full agent context, memory, or state transitions Durable persistence Database Stores data, but does not coordinate execution, retries, or recovery Orchestration and retries Workflow engine Provides strong primitives, but still needs integration with agent memory, policies, and model-driven behavior The real challenge is not the absence of infrastructure primitives. It is the need to combine them into a coherent runtime layer that can preserve state, resume execution, recover from failure, and enforce policy across long-running agent workflows. Designing the Runtime Layer for Stateful AI Agents A stateful AI agent needs more than compute resources and an execution environment. It also needs a runtime that preserves execution state, handles long-running processes, recovers from interruptions, and efficiently allocates resources. Unlike traditional application runtimes, which handle processes and requests only, an agent runtime has workloads that are frequently switching between active execution and waiting for external events. A well-designed agent runtime should address five core capabilities. These capabilities lay the foundation for running stateful agents reliably without requiring every application to build its own execution infrastructure. Runtime capability Responsibility State persistence Preserve execution context, memory, and intermediate progress across interruptions Lifecycle management Manage agent creation, execution, suspension, resumption, and termination Execution isolation Run agents in secure, isolated environments Resource efficiency Release compute resources when agents are idle and restore them when needed Recovery and continuity Restore saved execution state after interruptions and support reliable continuation Agent Substrate is an open-source runtime for running stateful workloads (e.g., AI agents) on Kubernetes. Instead of treating each agent as a container permanently running, it uses an actor-based execution model where instances of agents can be suspended when idle and resumed when needed. Its architecture combines: Actor lifecycle management: Tracks actor instances and manages their suspension and resumption.Checkpointing and restoration: Preserves process memory and writable filesystem state through snapshots.Worker pools: Maintains available compute capacity for scheduling and restoring actors.Execution isolation: Supports sandbox technologies such as gVisor and microVMs.Kubernetes integration: Uses Kubernetes for underlying worker provisioning and infrastructure management. Separating Reasoning, Orchestration, and Runtime Infrastructure A reliable agent architecture separates three responsibilities: what the agent decides; how its work is coordinated; and where its execution is maintained. They are tightly coupled, but they address different problems (though they interact). Layer Responsibility Examples Agent reasoning Determines what actions to take, selects tools, and interprets results LLMs, LangGraph, CrewAI Execution orchestration Coordinates tasks, dependencies, retries, approvals, and workflow transitions Workflow engines, agent orchestrators Runtime infrastructure Manages execution environments, state persistence, isolation, suspension, and recovery Agent Substrate, Kubernetes-backed runtime services The reasoning layer decides what to do next; the orchestration layer coordinates when and how that action is executed. The runtime infrastructure ensures that execution can continue reliably, even when agents become idle, encounter failures, or require additional resources. Where Agent Substrate Fits Traditional infrastructure provides the foundation for running applications, but stateful AI agents need a runtime layer to manage execution continuity, isolation, and resource utilization. Agent Substrate is an open-source Kubernetes-based runtime that addresses these requirements. Instead of treating each agent as a running container, it uses actor-based execution so stateful workloads can be suspended when idle and restored when execution needs to continue. Agent Substrate introduces several capabilities that align with the runtime architecture discussed earlier: Actor-based execution environments: Each agent workload is represented as an actor with its own execution context, enabling independent management of stateful instances.Actor lifecycle management: Coordinates actor creation, placement, execution, suspension, resumption, and termination.Suspension and resumption: Allows inactive actors to be suspended and restored when new requests arrive, reducing the need for continuously allocated compute.Memory and filesystem preservation: Uses snapshots to capture process memory and writable filesystem state, enabling execution to continue across suspension and restoration.Worker pools and resource multiplexing: Maps a large number of stateful actors onto a smaller pool of available workers, improving infrastructure utilization.Isolation and request routing: Uses sandbox technologies such as gVisor and microVMs to isolate execution, with ingress routing and controlled network access to manage communication securely. Mapping Agent Substrate to the Runtime Architecture Runtime capability How Agent Substrate addresses it State persistence Process memory and filesystem snapshots Lifecycle management Actor creation, suspension, resumption, and termination Execution isolation Sandboxed actors using gVisor or microVMs Resource efficiency Worker pooling and actor multiplexing Recovery and continuity Snapshot restoration and resumable execution Request management Actor-aware routing and network mediation Why This Matters Consider an AI agent that initiates a task, invokes external tools, and then waits for human approval. In a conventional deployment, its execution environment may remain allocated throughout the waiting period. With Agent Substrate, the actor can be suspended, its execution state preserved, and its compute resources released. When execution needs to continue, the actor can be restored onto an available worker. This separation allows many stateful workloads to be managed without maintaining a dedicated running environment for every idle agent. Agent Substrate does not replace agent frameworks or workflow orchestration engines. It provides the execution infrastructure; frameworks do reasoning and application logic, while runtime does isolation, state preservation, scheduling, and continuity of execution. Real-World Use Case: Stateful AI Incident Investigation Various use cases of stateful AI agents where Substrate can be useful are – Use case How Agent Substrate helps Implementation complexity AI incident investigation agent Preserves investigation context while waiting for telemetry or human approval Medium AI coding agent Maintains isolated development environments across coding sessions Medium AI customer support agent Preserves session state during long-running interactions Low–Medium AI infrastructure remediation agent Suspends while awaiting approval and resumes with its execution context Medium–High AI research agent Preserves intermediate work across extended research sessions Medium Implementation Details: AI Incident Investigation Consider an AI incident investigation agent that diagnoses infrastructure failures. When an alert fires, the agent collects telemetry, looks at logs, correlates recent deployments, and builds an investigation context. Unlike a request-response application, the investigation may take several minutes or hours. The agent might have to wait for monitoring data, an engineering response, or human approval before continuing. Keeping a dedicated execution environment running during this wait time is unnecessary. Restarting the agent, however, requires reconstructing its investigation context and intermediate results. An agent runtime (e.g., Agent Substrate) solves this problem by providing a stateful execution environment that can be suspended and restored. For example: Initialize: An infrastructure alert starts an investigation actor.Execute: The actor collects logs, metrics, and deployment information.Persist: Investigation context and intermediate findings are retained in the actor's execution state.Suspend: The actor is suspended while awaiting additional information or human approval.Resume: When new information becomes available, the actor is restored and continues using its preserved context.Complete: The investigation produces a diagnostic summary and recommended remediation actions. Step 1: Build a Minimal Incident Investigation Service Create a Python or Go application that exposes: endpointpurpose POST /investigate Start an investigation and save the incident context GET /status Return the current investigation state POST /evidence Add new telemetry or investigation findings POST /complete Finalize and summarize the investigation Step 2: Maintain Investigation State The actor could preserve a state object similar to: JSON { "incident_id": "INC-1042", "service": "payment-api", "status": "waiting_for_approval", "investigation": { "error_rate": "12%", "suspected_cause": "recent deployment", "deployment": "v2.4.1" }, "approval_required": true, "recommended_action": "review rollback" } This represents application-level investigation context. The actor must actually load and update this data for the demo to demonstrate preservation. Step 3: Deploy the Service as an Agent Substrate Actor Package the application as an OCI image and create an Agent Substrate actor template configured with an appropriate sandbox, worker pool, and full-state snapshot policy. The existing counter example provides a useful reference for configuring an actor template. Step 4: Demonstrate Suspension and Resumption Once the actor has stored its investigation context, manually suspend it: Shell kubectl ate suspend actor incident-1042 -a incident-demo When a new request arrives through the configured actor router, the suspended actor can be activated and its state restored. You can also use the explicit resume lifecycle operation. These commands assume you have created an actor named incident-1042 in an atespace named incident-demo; they are not commands for the existing counter demo. Step 5: Verify the Result After resumption, retrieve the investigation status and verify that the agent still has its original findings. Hands-on Tutorial: Running and Resuming a Stateful Workload with Agent Substrate The objective of this tutorial is to execute a stateful workload, suspend it, and resume it without losing its execution state. This experiment uses a local Kubernetes environment with Podman and Kind. Setting Up Agent Substrate First, clone the Agent Substrate repository Shell git clone https://github.com/agent-substrate/substrate.git cd substrate With Podman configured as the container provider for Kind, create the local Kubernetes cluster and install the Agent Substrate runtime: Shell export KIND_EXPERIMENTAL_PROVIDER=podman ./hack/create-kind-cluster.sh ./hack/install-ate-kind.sh \ --deploy-ate-system \ --credential-provider='{"name":"k8s.io"}' Note: The upstream installation scripts assume Docker. Our Podman-based setup requires local compatibility adjustments, so this should be treated as an experimental installation path rather than an officially supported deployment configuration. Once deployed, verify that the runtime components are running: kubectl get pods -n ate-system Deploying the Stateful Counter Agent Substrate provides a counter demo that maintains state across three locations: Process memory: An in-memory counter.Filesystem: A counter stored in a file.Root filesystem: A counter stored in the actor's writable root filesystem. Deploy the demo and create an actor instance: Shell ./hack/install-ate-kind.sh --deploy-demo-counter kubectl ate create actor my-counter-1 \ -a ate-demo-counter \ --template counter Forward the router service to the local machine: Shell kubectl port-forward -n ate-system \ svc/atenet-router 8000:80 From another terminal, invoke the actor: Shell curl -X POST \ -H "ate-target-actor: ate-demo-counter/my-counter-1" \ http://localhost:8000/ The initial execution returned: Each subsequent request increments all three counters, demonstrating stateful execution within the actor. Check the status of the actor: Suspending the Actor After invoking the actor multiple times, suspend it using Agent Substrate's lifecycle controls. During suspension, Agent Substrate checkpoints the actor's execution state and releases its active execution slot. Unlike terminating a conventional container, suspension is designed to preserve the state required to continue execution later. Resuming the Actor Resume the suspended actor and invoke the actor again. Instead of restarting the counters from their initial values, the actor continues incrementing from its previously preserved state. For example, if all counters reached 5 before suspension, the next successful invocation produces: This illustrates the key capability of the runtime: execution can be suspended and restored while preserving both in-memory and filesystem state. Observing the Results The experiment demonstrates the following behavior: Stage Memory Counter File Counter Rootfs Counter Initial execution 1 1 1 Before suspension 5 5 5 During suspension Checkpointed Preserved Preserved After resumption and next request 6 6 6 Illustrative values for five requests before suspension; actual counts depend on the number of requests executed. The important observation is not the counter value itself, but its continuity across the suspension and resumption boundary. For the complete Podman, Kind, and Agent Substrate setup instructions, along with all tutorial execution commands, refer to the GitHub repository. What Happens Behind the Scenes? Agent Substrate coordinates several runtime mechanisms to support this behavior: Actor lifecycle management: Tracks running, suspended, and resumed actors.Checkpointing: Captures execution state, including process memory and filesystem changes, according to the configured snapshot scope.Snapshot storage: Retains checkpoints so the actor can be restored.Worker management: Releases execution resources during suspension and allocates resources when the actor resumes.Request routing: Directs subsequent requests to the appropriate actor execution environment. The experiment shows that there is a fundamental difference between normal process execution and stateful actor runtime: an actor can preserve its logical execution state without continuously occupying an active worker execution slot. This is particularly useful for AI agents that spend large portions of their lifetime waiting for external tools, model responses, or human decisions. However, preserving execution state does not guarantee exactly-once execution of external side effects or transactional consistency. Those concerns require additional application-level controls. Architectural Trade-Offs and Production Considerations Agent Substrate provides capabilities for running stateful AI agents efficiently, but having a runtime layer of its own has architectural trade-offs too. Production environments need to balance execution continuity with resource use, performance, security, and operational complexity. The following considerations are important when designing a production-ready agent runtime. Architectural consideration Trade-off / Production implication State persistence vs. storage overhead Frequent snapshots improve recoverability but increase storage consumption, checkpoint latency, and I/O overhead. Dedicated environments vs. shared worker pools Dedicated environments provide predictable resource availability, while shared pools improve utilization but introduce scheduling and capacity-management challenges. Resource efficiency vs. resume latency Suspending idle agents reduces compute costs, but restoring execution may introduce latency when processing new requests. Isolation vs. operational complexity Sandboxing strengthens workload isolation but introduces additional runtime configuration, monitoring, and debugging requirements. Infrastructure vs. workflow recovery Restoring process state enables execution continuity but does not guarantee successful recovery of application-level workflows or external operations. Security, policy, and observability Production deployments require authentication, authorization, audit logging, secrets management, and end-to-end tracing beyond basic execution management. Beyond Infrastructure-Level Recovery One important point to note is that preserving an agent’s execution state is not the same as guaranteeing correctness of its actions. For example, an agent calls a payment API and fails immediately after the payment succeeds but before recording the response. Restoring the agent from its last checkpoint could cause the payment request to execute again. A runtime can restore execution context, but it cannot tell whether an external operation was done successfully or not. Production systems therefore need additional mechanisms (idempotency keys, durable operation records, transactional outbox patterns and reconciliation workflows) to reduce duplicate side effects and ensure consistency across system boundaries. Similarly, runtime-level checkpointing does not guarantee transactional consistency between agent state, databases, and external services. Security and Operational Readiness Beyond execution management, production deployments require integration with broader platform capabilities, including identity-based access controls, policy enforcement, secrets management, observability, and resource governance. These capabilities help ensure that agents operate within defined security boundaries and that their execution can be monitored, audited, and controlled. The key architectural principle is that an agent runtime provides execution continuity, while application-level orchestration and governance remain responsible for ensuring operational correctness. Agent Substrate addresses important infrastructure challenges, but building production-ready agentic systems still requires coordinated design across the reasoning, orchestration, runtime, and platform layers. Conclusion As AI agents evolve from simple request-response systems to autonomous long-running workloads, the infrastructure supporting them has to evolve too. Reasoning capabilities aren’t enough; agents need reliable execution, persistent state, isolation, and recovery to work in production. Traditional infrastructure provides the building blocks, but integrating them into a coherent runtime remains a challenge. Agent Substrate demonstrates how an actor-based execution model, combined with checkpointing, suspension, resumption, and resource multiplexing, can help bridge this gap. However, a runtime is only one part of the broader agentic architecture. Workflow orchestration, security, policy enforcement, and application-level consistency remain equally important for building trustworthy systems. Ultimately, the next step in agentic AI is not just making agents more intelligent, but making their execution more reliable, resilient, and operationally sustainable. A well-designed runtime foundation is what enables intelligent agents to move from experimentation to production-scale systems.

By Josephine Eskaline Joyce, Ph.D

CORE

AI-Powered CI Test Failure Triage

Last month, a nightly regression came back with 60 failures out of roughly a thousand tests. I spent the first half of the next morning fixing none of them. I sorted them: some were the same failure under different names, some were environment noise, and some were the product actually breaking. By lunch it was three things. A dependent service had returned errors for about a two-hour window, a data seeding job hadn't finished before the suite started, and one genuine defect. Sixty red rows, three problems. That sorting is most of my week. It's also the part nobody automates, because it isn't a fixed sequence of steps. It's judgment applied to evidence scattered around a CI dashboard, a log group, a ticket history, and whatever got merged Friday afternoon. So I built a triage step that does the mechanical sorting for me. Not an agent, not self-healing tests. A post-suite job that clusters failures, pulls surrounding evidence, and hands me a summary I can verify. This is what's in it, what it cost, and the two ways it went wrong before it went right. What Automation Is Good At, and What It Isn't Automation answers one question extremely well: did this pass or did it fail? It answers it faster than I can and the same way every time, which is the whole point. What it doesn't tell me: Why did this fail?Are these eight failures actually one failure?Which of them do I care about before the release call at 11?Have we seen this before? (We usually have.)What should I look at next? Every one of those is a reading problem, not an execution problem. The data already exists. It's just scattered around, and reassembling it by hand is slow and boring. The split I settled on: automation executes, a model reads, and I decide. The third part isn't a formality, and most of the engineering below exists to protect it. Step One: Cluster Without the Model My first mistake was handing raw failures to a model and asking what happened. Sixty stack traces is a lot of tokens; the answer took thirty seconds and was vague in a way that's hard to argue with. The fix was to do the deterministic part deterministically. Most failures in a big suite are duplicates. You can find that out with string normalization and a hash — no inference involved. Python # triage/cluster.py import hashlib import re from collections import defaultdict from xml.etree import ElementTree as ET # Order matters: strip the most specific patterns first. NOISE = [ (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-" r"[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"), (re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}\S*"), "<TS>"), (re.compile(r"0x[0-9a-fA-F]+"), "<ADDR>"), (re.compile(r"https?://[^\s'\"]+"), "<URL>"), (re.compile(r"'[^']{0,80}'"), "<STR>"), (re.compile(r"\b\d+\b"), "<N>"), ] def signature(message: str) -> str: """Collapse a failure message to a stable shape.""" if not message.strip(): return "" first_line = message.strip().splitlines()[0][:300] for pattern, replacement in NOISE: first_line = pattern.sub(replacement, first_line) return first_line def cluster(junit_path: str) -> dict: groups = defaultdict(list) for case in ET.parse(junit_path).getroot().iter("testcase"): problem = case.find("failure") if problem is None: problem = case.find("error") if problem is None: continue shape = signature(problem.get("message", "")) key = hashlib.sha1(shape.encode()).hexdigest()[:10] groups[key].append({ "test": f"{case.get('classname')}.{case.get('name')}", "shape": shape, "time": float(case.get("time", 0)), }) return groups On that 60-failure run, this collapsed to six clusters. Two of them held 30 and 15 tests. That single number — 60 down to 6 — is most of the value, and no AI was involved in producing it. It also changes the economics of everything downstream. The model never sees 60 stack traces. It sees six normalized shapes plus a slice of evidence for each shape, which is roughly 4K tokens instead of 60K. Cheaper, faster, and a much smaller surface for the thing to make something up on. If you build one piece of this, build this piece. Step Two: Attach Evidence, With Hard Limits Each cluster then gets enriched from sources that are already there: the log group for the window the suite ran in, the deploys that landed since the last green run, and any tickets whose text matches the normalized shape. Python # triage/evidence.py import boto3 logs = boto3.client("logs") PLACEHOLDERS = ("<UUID>", "<TS>", "<ADDR>", "<URL>", "<STR>", "<N>") def log_hits(shape: str, start_ms: int, end_ms: int, group: str, limit: int = 5): """Pull a bounded sample of matching log lines for one cluster.""" # The normalized shape has placeholders in it; search on the # longest literal run instead. stripped = shape for token in PLACEHOLDERS: stripped = stripped.replace(token, "|") literal = max(stripped.split("|"), key=len).strip() if len(literal) < 12: return [] # too generic to search on; don't guess response = logs.filter_log_events( logGroupName=group, startTime=start_ms, endTime=end_ms, filterPattern=f'"{literal}"', limit=limit, ) return [ {"ts": e["timestamp"], "stream": e["logStreamName"], "message": e["message"][:400]} for e in response.get("events", []) ] Two constraints in there are deliberate, and both came from things going wrong. The limit=5 is because the second version of this pulled every matching line and blew past the context window on a noisy cluster. Five representative lines with timestamps answer "is this the same error, and when did it start" just as well as five hundred. The len(literal) < 12 guard is because a cluster whose shape normalizes down to something like Expected but was will match half the log group. Returning nothing is correct there. An empty evidence list is a fact, and the prompt below is written so the model has to say so rather than fill the gap. Step Three: Ask for a Reading, Not a Verdict The prompt is the part I rewrote the most. What finally worked was forcing every claim to point at a piece of evidence by ID, and constraining the service field to a list of things that actually exist. Python SYSTEM = """You are triaging failures from one CI run. Rules: - Every claim must cite evidence by its id. No id, no claim. - If the evidence does not support a cause, set cause to null. Do not guess. "Insufficient evidence" is a correct answer. - suspected_service must be one of KNOWN_SERVICES or null. - Do not restate the stack trace back to me. Return only JSON matching this schema: {"clusters":[{"cluster_id":str,"test_count":int, "cause":str|null,"suspected_service":str|null, "confidence":"low"|"medium"|"high", "evidence_ids":[str],"suggested_next_step":str}]} """ Before that enum existed, the model named a service we don't run. It sounded completely reasonable — right naming convention, plausible responsibility, adjacent to the real one. That's the failure mode that actually costs you: not obvious nonsense, but a confident answer that reads as if it came from someone who knows your architecture. An engineer who didn't know the service list would have gone looking for it. KNOWN_SERVICES gets injected from the same dependency manifest the deployment uses, so it can't drift from reality. Step Four: Check Its Work Before a Human Sees It This is the piece I'd argue is non-optional, and it's about forty lines. Python # triage/validate.py def validate(model_output: dict, clusters: dict, evidence: dict, window: tuple) -> dict: start_ms, end_ms = window kept, dropped = [], [] for item in model_output.get("clusters", []): cid = item.get("cluster_id") if cid not in clusters: dropped.append((cid, "unknown cluster id")) continue if item.get("test_count") != len(clusters[cid]): item["test_count"] = len(clusters[cid]) # we own this number cited = set(item.get("evidence_ids", [])) real = {e["id"] for e in evidence.get(cid, [])} if item.get("cause") and not cited: dropped.append((cid, "cause with no evidence")) continue if not cited <= real: dropped.append((cid, f"cited missing evidence: {cited - real}")) continue for e in evidence[cid]: if e["id"] in cited and not (start_ms <= e["ts"] <= end_ms): dropped.append((cid, "evidence outside run window")) break else: kept.append(item) return {"clusters": kept, "dropped": dropped} Counts come from the parser, never from the model — that's the one class of error I refuse to ship, because a wrong test count makes every other number in the summary suspect. Anything citing evidence that doesn't exist gets dropped rather than shown with a caveat, since a caveat still puts the idea in the reader's head. The dropped list is not thrown away. It's the closest thing I have to a quality signal on the triage step itself, and I look at it weekly. When one cluster type keeps landing there, the problem is usually my evidence collection, not the model. What It Actually Produces Before, for one cluster: Plain Text 30 tests failed. After: Plain Text cluster 4a1c9f2e — 30 tests cause: dependent service returned 503 on order lookups service: order-api confidence: high evidence: log lines at 02:14:06, 02:31:44, 03:02:10 (same message, same stream); no matching lines in the prior night's run next: check order-api health for the 02:10–03:20 window The second one I can disprove in about ninety seconds. That's the actual bar. Not "is it right" — "can I check it quickly." Every design decision above serves that. Numbers Triage time on a large regression run: roughly 90 minutes to about 20.Clusters per run: 5–8, from 40–70 raw failures.Model cost: a few cents per run. Not a factor.Wall clock for the triage step: under a minute, running after the suite.Claims dropped by the validator: around 1 in 8 in the first month, closer to 1 in 20 once the service enum and evidence limits were in. That last number is the one I'd watch if you build this. A rising drop rate means something upstream changed. Where I Stopped, On Purpose The obvious next step is letting it act. Reopen the ticket, rerun the cluster, patch the test. I did the first two. I won't do the third. The pitch for self-healing tests goes: the agent notices a failing test, decides the locator is stale, updates it, the test goes green, nobody gets paged. Run that on a test that was failing because the product broke. The agent didn't fix anything. It deleted the evidence, faster than a human would have noticed. The suite is green, the dashboard is calm, the defect ships. Green isn't the goal. Green with the check still intact is the goal, and nothing I've built or used reliably tells the difference between "this assertion is stale" and "this assertion just caught something." So I tier by blast radius: Tier Examples Gate Read-only clustering, summarizing, log search, explaining a trace none — worst case I ignore it Suggest root cause, test selection, defect priority, proposed edits engineer reviews before it moves Change config, closing defects, disabling tests, anything release-related explicit approval, always Most of the anxiety about AI in QA comes from treating it as one permission level. It isn't one. Testing the Thing That Helps You Test There's a part of this the industry hasn't sorted out, and I'm including it because pretending otherwise would be dishonest. The output isn't deterministic, so pass/fail assertions don't map onto it cleanly. What I do instead is keep a fixture set of about twenty past runs where I already know the real root cause, and score new prompt or model versions against them on two things: did it name the right cause, and did it cite evidence that actually supports it. It's regression testing with a rubric instead of an assert, and it's the only reason I noticed when a model version change made the summaries longer and vaguer without making them wrong. That's the other thing to plan for. Behavior shifts underneath you on someone else's release schedule. A prompt-based workflow that's been stable for months can quietly degrade after an upgrade nobody on your team asked for. Pin your model version, and treat a version bump as a change that needs a test run. And you are feeding logs — possibly source, possibly customer data — into a service. Whatever your org's answer is on that, get it in writing before you build this, not after. It's a security conversation more than a QA one, but it lands on our desk anyway. The Point The most valuable thing here isn't generating tests. It's reading everything we already collect and can't get through fast enough. Speed comes from the model. Repeatability comes from automation. Evidence comes from systems we already run. The call stays with me, mostly because I have to answer for the release.

By Anil Reddy Putlacheru

Decoupling a Data Pipeline From the Tables It Was Never Supposed to Depend On

I built a pipeline framework on top of a convenience feature, and for about a year it looked like the best decision in the design. The framework generates analytics pipelines from declarative configuration: a user describes the events, stages, and metrics they care about, and the framework produces the Spark jobs, output tables, and dashboards. To make those pipelines incremental, I required every upstream source to be a table with change data feed enabled. The table format's changelog told us exactly which rows had been inserted, updated, or deleted since the last run, so every generated pipeline processed only deltas, for free, with no bookkeeping code of its own. The fragility hidden in that decision took longer to surface than it should have because it never showed up as a framework bug. It showed up as other people's schema changes. The Coupling Nobody Signed Up For When your only supported source type is a physical table with a change feed, every pipeline you generate is bound directly to that table's physical schema. Not to the concept the table represents, and not to some resilient subset of columns. It is bound to the table itself, exactly as its owning team defined it this week. Upstream teams in a large organization evolve their tables constantly, and they are right to. A column gets renamed to match a naming convention. A type gets widened. A table gets rebuilt as part of a storage migration and picks up a new layout. From the owning team's perspective, these are internal changes to an asset they own. From my framework's perspective, each one was a breaking change delivered without notice to every downstream pipeline reading the table. Because the framework made pipelines cheap to create, there were a lot of them. One upstream rename could break a dozen production pipelines in a single morning, none of which had changed at all. The standard answer to this problem is an isolation layer: put a view between the physical table and its consumers, and absorb schema changes in the view definition. That is exactly the layer my design made impossible. Change data feeds are a property of physical tables. A view does not emit a changelog. By requiring change-data-feed sources, I had not just failed to build an isolation layer. I had structurally forbidden anyone from adding one. The convenience feature and the fragility were the same feature. It is worth being precise about why the usual mitigations do not work here. Asking upstream teams to freeze their schemas does not scale past two or three teams, and should not, because it converts your fragility into their paralysis. Materializing private copies of every upstream table restores isolation but doubles storage and adds a freshness lag you now have to monitor. Contract tests detect the breakage earlier, which is worth something, but detection is not absorption. The pipelines still break; you just get a nicer error message. The only structural fix is to let pipelines depend on an interface rather than an implementation, which meant the framework had to stop requiring physical tables at all. Re-Architecting for Views and Arbitrary SQL Supporting views and arbitrary SQL queries as first-class upstream sources sounds like adding a source type. It is actually a rewrite of the framework's incremental-processing model, because the thing you lose when you give up the change feed is the framework's entire notion of "what's new." The replacement is less elegant and more honest: date-field-based processing. Every view or SQL source declares a date or timestamp field that partitions its data by processing period. In the declarative config, the difference between the old world and the new one is small on the surface: YAML sources: # Old style: physical table, incremental via change data feed orders_cdf: type: table name: prod.orders_v2 # New style: view or SQL, incremental via a declared date field orders_view: type: view name: analytics.orders_stable date_field: order_date sessions_sql: type: sql query: > SELECT s.session_id, s.user_id, s.started_at, d.channel FROM prod.sessions s JOIN prod.session_details d ON s.session_id = d.session_id date_field: started_at Underneath, everything changes. Instead of asking the storage layer which rows changed, the framework tracks which date ranges each pipeline has processed and generates predicates that read exactly the unprocessed ranges: SQL SELECT * FROM analytics.orders_stable WHERE order_date >= '{{ watermark }' AND order_date < '{{ run_end }' This is watermark-based incremental processing, one of the oldest patterns in data engineering, and there is a reason it has survived: it depends only on the logical content of the data, never on any physical property of how the data is stored. A view can answer a date-range predicate. A three-table join can answer it. A changelog can only be asked for by the physical table's direct reader. Moving to date-range semantics forced two other pieces of the re-architecture. The first was backfill handling. Change-feed processing has no natural story for reprocessing history; the feed only moves forward, and most implementations retain limited history anyway. Date-range processing makes backfill a first-class operation. Reprocessing means re-running a set of date ranges, and correctness means each range must be idempotent, overwriting that period's output atomically rather than appending to it. Building that idempotency into every generated pipeline, instead of leaving it to each pipeline author, was most of the work. It also required adding a true batch mode, because a three-year backfill and a daily increment are the same operation at different scales only if the execution engine treats them that way. The second piece was a source-contract layer, and this is the part I would build first if I were starting over. Each pipeline declares the columns it needs from each source, with expected types and nullability, and the framework validates every source against its contract before any run: Python def validate_contract(source_schema, contract): failures = [] for col in contract.columns: actual = source_schema.get(col.name) if actual is None: failures.append(f"missing column: {col.name}") elif not compatible(actual.dtype, col.dtype): failures.append( f"type drift on {col.name}: " f"expected {col.dtype}, found {actual.dtype}" ) elif col.required and actual.nullable: failures.append(f"nullability drift on {col.name}") if failures: raise SourceContractError(source=contract.source_name, failures=failures) When an upstream schema changes now, one of two things happens, and both are enormous improvements over silent breakage. Either the view definition absorbs the change, and the contract still validates, in which case downstream pipelines never know anything happened; or validation fails before the run starts, producing a named, actionable discrepancy (this column, this type, this source) instead of a runtime failure deep inside a generated Spark job at 3 a.m. What Actually Changed The measurable effect was the disappearance of a whole failure category. Schema drift incidents went from a recurring operational tax, each one a cross-team scramble to identify what changed and patch every affected pipeline, to a non-event absorbed at the view layer, usually by the upstream team itself as part of making their change. The less measurable effect was on how teams model their data. Once views were first-class sources, upstream teams could treat their physical tables as private implementation details and publish stable view interfaces on top, which is how table ownership should work anyway. The framework had previously punished that discipline. Now it rewarded it. The lesson I keep returning to is about where convenience features belong in a design. Change data feeds are a genuinely good optimization. My mistake was promoting an optimization into an interface. The framework's contract with its sources became "you are a physical table with a changelog" when it should have been "you can answer a date-range query and honor this column contract." Optimizations belong behind interfaces, where you can keep them for the sources that support them and fall back cleanly for the sources that do not. When an optimization becomes the interface, every one of its physical preconditions becomes a coupling point, and you inherit fragility from parts of the system you never intended to depend on. The tables were never supposed to be the dependency. The data was.

By Rahul Narakula

Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade

When JDK 27 was released on 15 September 2026, most of the attention went to compact object headers and post-quantum TLS. JEP 523 received less coverage, partly because its description sounds like housekeeping: G1 becomes the default garbage collector in all environments. For most applications, that changes nothing. For a specific class of container, it changes the collector silently, with no warning in the logs and no failure at startup. This article covers what the fallback did, which workloads lose it, how to identify affected services, and how to run a comparison that produces usable numbers rather than noise. What the Serial GC Fallback Did G1 has been the nominal default since Java 9. What is less widely known is that HotSpot inspected the machine at startup and overrode that choice in constrained environments. The JVM selected Serial GC when either condition was true: ConditionThresholdAvailable processors1Physical memoryLess than 1,792 MB The reasoning was sound at the time. Serial GC is single-threaded with a very small native memory footprint. On a single-core machine, a concurrent collector spends time competing with application threads for the only available core, and the bookkeeping structures G1 maintains were expensive relative to a small heap. G1 has improved substantially since then. Its native memory footprint has been reduced across several releases, and JDK 26 delivered JEP 522, which removed synchronization overhead from G1's write barriers by introducing a second card table. JEP 523 concludes that work. The fallback is removed, and G1 is now selected in every environment regardless of CPU count or available memory. Serial GC remains fully supported and can still be selected explicitly: Shell java -XX:+UseSerialGC -jar application.jar Applications that already specify a collector, whether G1, ZGC, Parallel, or Shenandoah, are unaffected by this change. Which Workloads Are Affected The change only reaches services that never had an explicit collector flag and that run in a constrained environment. In practice, this describes a predictable set of deployments: Sidecar containers. Log shippers, metrics agents, and service mesh components typically receive a fraction of a CPU and a few hundred megabytes of memory. They are rarely tuned.Batch and queue workers. Short-lived jobs scheduled onto whatever capacity is available.Low-traffic internal services. Admin APIs, configuration services, and internal tooling that are deliberately given small pods.CI runners and development containers. Environments where JVM flags are seldom set because the workload is not production.Pods with fractional CPU limits. This case is easy to miss. A container configured with cpu: 500m may report a single available processor to the JVM, which placed it in Serial GC territory on earlier releases. These are the workloads least likely to appear in a performance test plan, which is precisely why the change deserves deliberate verification. Step 1: Identify Your Current Collector Before changing anything, determine which of your services are actually running Serial GC today. The GarbageCollectorMXBean interface exposes this directly: Java import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; public final class RuntimeFacts { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); System.out.printf("java version : %s%n", Runtime.version()); System.out.printf("available cpus : %d%n", rt.availableProcessors()); System.out.printf("max heap MB : %,d%n", rt.maxMemory() / (1024 * 1024)); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { System.out.printf("collector : %s%n", gc.getName()); } } } The collector names identify which implementation is active: CollectorReported MXBean namesSerial GCCopy, MarkSweepCompactG1G1 Young Generation, G1 Concurrent GCParallelPS Scavenge, PS MarkSweepZGCZGC Cycles, ZGC Pauses For a Spring Boot application, registering this as a startup listener puts the information into your existing log aggregation without any additional infrastructure: Java import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.stereotype.Component; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.util.stream.Collectors; @Component public class GcFactsLogger { @EventListener(ApplicationReadyEvent.class) public void logCollector() { Runtime rt = Runtime.getRuntime(); String collectors = ManagementFactory.getGarbageCollectorMXBeans() .stream() .map(GarbageCollectorMXBean::getName) .collect(Collectors.joining(", ")); System.out.printf( "JVM %s | cpus=%d | heap=%,dMB | gc=%s%n", Runtime.version(), rt.availableProcessors(), rt.maxMemory() / (1024 * 1024), collectors); } } Deploy this across your estate, then filter your logs for entries reporting Copy as a collector name. That result set defines the scope of your testing. Step 2: Understand the Native Memory Difference Discussions about garbage collector selection usually center on pause times. For this particular change, native memory consumption is the more consequential metric. G1 maintains bookkeeping structures that Serial GC does not require, including remembered sets, card tables, and per-region metadata. This memory is allocated outside the Java heap, which means it counts against a container memory limit without appearing in heap metrics. Measurements published after the JDK 27 release place G1's native memory overhead at approximately 43 MB, compared with roughly 1.7 MB for Serial GC. Ankur Mundra measured both collectors on Spring Boot 4.1 within real cgroup limits and reported these figures. The absolute difference is modest. The proportional impact depends entirely on container size: Container memory limitApproximate G1 overhead share1024 MB4%512 MB8%256 MB16% On a tightly sized container, this overhead is consumed before the application allocates its first object. Where memory limits were set with minimal headroom, the change alone can produce OOM kills after an upgrade that altered nothing else. Native Memory Tracking provides the per-category breakdown if you want to confirm the figure in your own environment: Shell java -XX:NativeMemoryTracking=summary \ -XX:+UnlockDiagnosticVMOptions \ -jar application.jar Then, against the running process: Shell jcmd <pid> VM.native_memory summary The GC section of the output reports committed native memory attributable to the collector. Step 3: Run a Controlled Comparison A meaningful comparison requires the smallest container size you actually deploy, representative load, and a run long enough to reach steady state. A thirty-second run measures startup behavior rather than collection behavior. Capture GC logs for both configurations: Shell # Pre-Java 27 behaviour java -XX:+UseSerialGC \ -Xlog:gc*:file=serial.log:time,uptime,level,tags \ -jar application.jar # Java 27 default java -Xlog:gc*:file=g1.log:time,uptime,level,tags \ -jar application.jar Three measurements matter, and the third is the one most often omitted: Total pause time. Sum the pause durations across the run rather than comparing individual collections. Serial GC performs fewer, longer pauses; G1 performs more, shorter ones. Only the aggregate is comparable.Tail latency. G1 runs concurrent phases. On a single-core container, those phases contend with request handling for the only available processor. This effect is invisible in a mean and clearly visible at p99.Resident set size under load. This is where the native memory difference becomes operationally relevant. Sample RSS throughout the run rather than reading it once at startup: Shell while true; do ps -o rss= -p "$PID" | awk '{printf "%s %d MB\n", strftime("%H:%M:%S"), $1/1024}' sleep 5 done If RSS approaches your container limit under G1 where it did not under Serial, you have found the problem before it found you. Step 4: Pin the Collector When G1 Loses If measurement shows that G1 performs worse on a constrained workload, the appropriate response is not to defer the JDK upgrade. It is to state the collector explicitly: Shell java -XX:+UseSerialGC -jar application.jar This is a supported configuration. Serial GC is neither deprecated nor scheduled for removal, and JEP 523 explicitly preserves it. In a Kubernetes deployment, express the choice where it will be reviewed: YAML spec: containers: - name: batch-worker env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UseSerialGC" There is an argument for setting the flag explicitly even where G1 performs well. An explicit collector choice is self-documenting. A future engineer reading the manifest can see that a decision was made and tested, rather than inferring what the JVM selected at runtime. Conclusion JEP 523 is a reasonable change. G1 has genuinely closed the gap that justified the fallback, and removing a hidden conditional from the JVM's startup logic simplifies the mental model for everyone. The operational consideration is narrower: a set of small, rarely tested containers will change collectors on upgrade without any signal that it happened. Identifying those containers takes one log statement. Testing them takes one load run. The cost of skipping both is a latency regression or an OOM kill that appears unrelated to any change your team made. Java 27 is not a long-term support release, and many teams will remain on Java 25. That does not remove the relevance of this work. The same default will arrive with the next LTS, and evaluating it on a short-term release carries considerably less risk than discovering it mid-migration. Further Reading JEP 523: Make G1 the Default Garbage Collector in All EnvironmentsJEP 522: G1 GC: Improve Throughput by Reducing SynchronizationJDK 27 Release Notes

By Garima Agarwal

Delegated Transport, Forked Authorization: Where Chat and AI-Agent Integrations Actually Break

When you move messaging onto a third-party platform, you don't hand off the authorization problem. You split it across two systems. Most incidents live in the gap. A simulation harness, and a documentation audit of QuickBlox as a worked example. The Revoked User Who Kept Reading Say a clinic's app removes a contractor's access to a patient thread at 14:02. The app's database says so, the UI hides the thread, and the audit table has the row. The contractor's phone keeps receiving new messages until 15:00, and then keeps going. Nothing in the application is wrong. The membership removal went to the chat platform's API as a single best-effort call, and the API was having a bad ten minutes. The call failed, and the exception was swallowed. The contractor's session token is still valid, and the contractor's client keeps making requests that keep it valid. This is a hypothetical, but every piece is routine: a removal path with no retry, an admin API that can be unavailable, a session whose lifetime depends on activity. The useful question is how long the window lasts, and it becomes measurable once you stop treating the integration as an API call and start treating it as a consistency boundary. This article builds a small deterministic harness, seamlab.py, to measure that window along with seven other failures at the same boundary, including an AI agent acting after a human has taken over a conversation. It then audits one real platform, QuickBlox, against the same questions using its public documentation. QuickBlox is the worked example because its developer documentation is public and detailed enough to check. The harness measures a class of platforms and nothing about any vendor. The documentation audit reads documentation and tests nothing. I keep those two kinds of evidence apart throughout. The Flawed Assumption The usual framing of "build vs. buy" for chat or calling is about transport: delivery, ordering, presence, media. Delegating those is reasonable. But "who may read this thread, who may post, who may trigger an action" is not a transport concern, and delegating transport forks it: Your application holds the relationships: this clinician treats that patient, this contractor was removed.The platform holds a projection of them: dialog memberships, session tokens, message history. The projection is only as fresh as your last successful write to it, and the platform enforces the projection, not your database. So at least two places answer "is this allowed," and they can disagree. Each failure below is a case of that disagreement, or of one side believing a claim the other never checked. Two quantities make this concrete. Revocation window W: the time between the app committing a revocation and the last successful read by the revoked principal. If the platform checks live membership on every read, W is bounded by the time until the removal call lands. If it authorizes from the session token alone, W is bounded by the token's remaining life, and whether that life is fixed or extended by activity decides everything.Authority provenance: for any action in the system, whose authority is it exercising? For a human action, the answer is trivial. For an AI agent in a multi-party conversation, it usually isn't. Threat Model Six attacker positions, all realistic for a chat integration: #AttackerCapabilityAExternalKnows the webhook URLBAuthenticated low-privilege participantCan post messages into a thread the agent readsCSelf-registered platform userPlatform allows signup, no verified binding to your user modelDRevoked participantHolds a valid token from before revocationEReplayerHas a captured, validly signed event (logs, proxy, compromised sink)FWriter on the audit storeCan modify rows Trust boundaries: client to app, app to platform (admin API), platform to app (event delivery), agent to tools, and the audit store. ClassPreconditionMechanismTestControlForged eventAUnauthenticated event endpoint acceptedforgedSignature verificationIdentity confusionCSender resolved by mutable attribute (email)spoofed identityImmutable-ID binding tableReplayEDedupe TTL shorter than replay horizonreplayTimestamp window plus retention ≥ 2×windowDuplicate side effectNormal at-least-once deliveryNon-atomic check-then-actraceAtomic claim before actingStale accessDToken and membership outlive revocationrevocation sweepShort TTL, outbox retry, token killAgent confused deputyBInjected text drives a privileged toolA1–A4Parameter-bound approvalPost-handoff actionRace with human takeoverAgent call in flight at handoffA5–A6Epoch fencingScope overreachBAgent reads history beyond sessionA7Capability-scoped history Audit tampering (F) is covered separately below. The Harness seamlab.py is about 480 lines of stdlib Python on a logical clock, with no network and no randomness. It contains: A platform simulator: at-least-once event envelopes signed with HMAC, expiring tokens with scopes frozen at mint, optional idle (sliding) expiry, self-service registration, and an admin API that can be switched off. Whether reads check live membership, and whether a server-side token kill exists, are switches.Two app implementations: a naive one and a hardened one.An agent world with a tool proxy configurable per control, so the table shows which control closes which attack.A worst-case agent: it obeys every CALL tool key=value line in its context. That is deliberate. The tests target the authorization layer and say nothing about how robust any model is to injection. If your model resists injection, that's a bonus. These controls assume it doesn't. The duplicate-delivery test enumerates every distinct interleaving of two workers handling the same event, rather than sampling a few. Naive Design, Then the Attacks The naive ingress handler is a shape I'd expect to find in plenty of codebases: no authentication on the endpoint, dedupe with a TTL, check then act then record, sender resolved by email. def handle(self, env): t = self.seen.get(env["id"]); yield # check if t is not None and self.clock.t - t < self.DEDUPE_TTL: # 300s return "dup" uid = self.resolve(env["body"]["sender"]) # email match if uid: self.run_command(uid, env["body"]["text"]) # act yield self.seen[env["id"]] = self.clock.t; yield # record AttackNaiveHardenedForged unsigned eventsucceedsblockedValidly signed event, spoofed identitysucceedsblockedReplay at t+225 sblockedblockedReplay at t+3600 ssucceedsblockedDuplicate delivery: interleavings producing a double effect60 of 660 of 12 The spoofed-identity row is the interesting one. The event carries a correct platform signature. Nothing was forged. The attacker registered a platform account with email [email protected], unverified. The naive app lowercases and matches against its user table, so a perfectly authentic message from the wrong person runs with the clinician's authority. Signature verification can't help, because the signature is telling the truth about who sent it. The control is a binding table keyed on the platform's immutable user ID, written only by your backend when it provisions the platform user for an authenticated app user. The attacker's ID has no binding, so the event is rejected as unbound. The race result is worse than it looks. 60 of 66 distinct schedules double-fire. Only interleavings where the first worker finishes recording before the second starts checking are safe. This is the ordinary case for any retry storm. The fix is one atomic operation (SET NX EX, or an insert against a unique constraint) before any side effect. The hardened handler: def handle(self, env): if not hmac.compare_digest(env.get("sig", ""), sig_of(env)): return "bad_sig" if abs(self.clock.t - env["ts"]) > self.TOL: return "stale" ok = self.claim(env["id"]); yield # atomic set-if-absent, retention 2*TOL if not ok: return "dup" uid = self.bindings.get(env["body"]["sender"]) if uid is None: return "unbound" self.run_command(uid, env["body"]["text"]); yield return "ok" Mutation-Testing the Control The suite should be able to fail the hardened design; otherwise, the green column is decoration. So I mutated one parameter: dedupe retention set to half the timestamp tolerance. The replay at t+225 s then succeeds, because the event is still inside the accepted time window while its dedupe record has expired. The replay at t+3600 s is still blocked by the timestamp check. That yields an invariant worth writing into code review: dedupe retention must be at least twice the accepted timestamp skew (once for events slightly in the future, once for slightly in the past). Timestamp validation and dedupe are two halves of one control, and shortening either silently opens the gap. One caveat: claim-before-act gives at-most-once side effects if the process dies between claim and effect. For effects that must not be lost, the downstream operation needs its own idempotency key derived from the event ID. The harness does not model crash recovery. Revocation Window, Measured The harness drives a revoked user who keeps polling and refreshing, with the token's age at revocation sampled across its lifetime. The scenario that matters: the admin API is down when the revocation is issued and comes back 600 s later. In the first table, the platform checks live membership on every read: ConfigWorst W (s)Mean W (s)Token mints per user per hourNaive, TTL 3600, single attempt360018301Hardened, TTL 303016120Hardened, TTL 60603060Hardened, TTL 30030015212Hardened, TTL 9006014054Hardened, TTL 36006015561 "Hardened" means a transactional outbox retrying the removal with capped backoff, a broker that re-checks the app's authorization on every token mint, and a short TTL. Once TTL exceeds the outage, W is capped by the outage plus up to one backoff interval (601 s in this run). Below the outage length, TTL is the binding constraint. With a healthy admin API, every configuration, naive included, shows W = 0. The naive design only fails when the platform is degraded, which is when nobody is watching. When the Platform Doesn’t Check Membership, and Tokens Slide Two platform properties change the picture, and I only added the second after reading the vendor documentation described below, which is why the harness changed between drafts. If reads are authorized from the token alone, removal calls don't matter, and the TTL is the only built-in lever. If the token's expiry is idle-based, so each request pushes it out, then a revoked client that keeps making requests never lets the token expire, and the TTL stops being a bound at all. The broker's re-check at refresh never runs, because refresh never happens. The last control available is a server-side token kill, if the platform offers one. Results for a token-only platform (hardened TTL 60, naive TTL 3600; "never ends" means access persisted to the end of the simulated horizon): Expiry model, admin APINaiveHardened, no token killHardened + token killFixed TTL, healthy3600 s60 s0 sFixed TTL, 600 s outage3600 s60 s60 sIdle expiry, healthynever endsnever ends0 sIdle expiry, 600 s outagenever endsnever ends601 s Read the third row twice. With idle expiry and no kill, the short TTL and the broker re-check buy nothing against a client that keeps talking. The only controls that end access are live-membership enforcement on the platform or an invalidation you can trigger from the server. So the most valuable hour you can spend before integrating a platform is finding out which of those you have. At TTL 60, each active user generates 60 token mints per hour. Whether that is trivial depends on your user count and the platform's rate limits, which the harness does not model. The AI Participant: Confused Deputy, Not “Authorization for Bots” A multi-party conversation is a prompt authored by several principals with different privileges. The agent reading it holds one set of tool credentials. Any principal who can put text in the thread can attempt to spend authority they don't have. This is the classic confused deputy, with the deputy now parsing natural language. The harness has a dialog with a patient and a clinician, and three tools: get_my_record (low privilege), approve_refill (clinician only), and read_history. The proxy runs in three configurations: naive (the agent's standing credential decides everything), role-check (the call executes with the authority of whoever authored the triggering message), and hardened (role check plus parameter-bound approval, epoch fencing, and history clamping). AttackNaiveRole-check onlyHardenedA1 Patient's message steers agent into approve_refillsucceedsblockedblockedA2 Clinician's benign turn; patient's injected text sits in contextsucceedssucceedsblockedA3 Approval for rx42 reused to approve rx99succeedssucceedsblockedA4 One approval consumed twicesucceedssucceedsblockedA5 In-flight agent call executes after human takeoversucceedssucceedsblockedA6 New agent turn minted after takeoversucceedssucceedsblockedA7 Agent reads history outside session scopesucceedssucceedsblockedB1 Clinician approves refill through explicit approvalworksworksworksB2 Patient reads own record through agentworksworksworks The B rows make sure the hardened column isn't green merely because it denies everything. A2 is the finding that changed my plan. My pre-run design was authority as the intersection of the acting principals' rights. Role-check is that rule, and it stops the obvious attack (A1). It fails A2 completely. The clinician sends "please summarize the thread." The agent's capability is legitimately the clinician's. The patient's earlier message, still in context, asks for a refill approval, and the agent obliges, spending authority that was never granted for that purpose. Intersection scopes who the agent acts for. It says nothing about what that authority was meant to cover. The fix is to stop treating conversation turns as grants for privileged operations. A privileged call needs an approval bound to the exact call, issued out-of-band by an authenticated UI session of a privileged human: def _approval_ok(self, tool, args, ap): b = ap["body"] good = hmac.new(APP_KEY, json.dumps(b, sort_keys=True).encode(), hashlib.sha256).hexdigest() if not hmac.compare_digest(good, ap["mac"]): return False if (b["tool"], b["args"], b["epoch"]) != (tool, args, self.epoch): return False if self.clock.t > b["exp"] or b["nonce"] in self.used: return False if self.users.get(b["principal"]) != "clinician": return False self.used.add(b["nonce"]); return True The approval binds tool, arguments, epoch, expiry, and a single-use nonce. A3 fails because the arguments differ, and A4 because the nonce is spent. Injected text can still request the call. It cannot carry the authority to make it. The system doesn't try to detect injection. It makes sure injected text can't hold a capability. Fenced handoff (A5, A6). When a human takes over, the app increments a per-dialog epoch. Every capability carries the epoch it was minted under, and the tool proxy rejects stale ones. A5 is the race: a turn starts at t=0, the human takes over at t=5, and the agent's call arrives at t=8. Without the fence it executes. This is a fencing token, and it has a fencing token's limit: it only works if the resource performing the effect checks the epoch. If the agent can also post through a path that bypasses your proxy, such as the platform's own API with the agent's own credentials, the fence is decorative. The platform-side complement is removing the agent from the dialog at handoff, which is another admin call, subject to the same outage window measured above, and which not every dialog type allows (see below). History scope (A7). The capability carries the index where the current session begins, and read_history refuses anything earlier. The harness checks the proxy's enforcement. It does not check that the agent's context assembly respects the boundary, which is separate code you would need to test. The audit log, and what a hash chain does not do The hardened path records each agent tool call (allowed or denied) and each handoff in a hash-chained log with actor type, model version, tool, arguments, result, and principal: Clean chain, verified against an externally stored head hash: passes.Row modified without recomputing the chain: detected.Row modified and the chain recomputed from that row forward (an attacker with write access): verifies clean unless you compare against the externally stored head, in which case it is detected. The chain alone defends against accidental corruption and clumsy tampering. It does not defend against someone who can write the store and knows the scheme. What does is anchoring the head hash somewhere that an attacker can't write: a separate account, an append-only object store, a periodic signed checkpoint. If your threat model includes a malicious insider with database access, the anchor is the control, and the chain is the mechanism that makes anchoring cheap. On compliance: in a regulated setting, "who did what, when, under whose authority" is what this log makes answerable. The HIPAA Security Rule's audit-control standard is 45 CFR §164.312(b) (cited from memory; confirm before relying on it). A platform's compliance posture does not discharge your obligations, and a log design is one input to them, not compliance itself. A Documentation Audit: The Seam on Quickblox Everything above is simulation. This section asks the checklist questions of a real platform using its public developer documentation, retrieved on 4 October 2026. Nothing here was tested against a live QuickBlox account. Documentation can be incomplete or out of date, and where I could not find an answer, I say so instead of guessing. Sources are listed at the end. QuestionWhat the documentation saysConsequence for the seam1. Are server-to-app events authenticated?I found no first-party documentation of outbound server-to-server event webhooks. A competitor's comparison page asserts QuickBlox doesn't support webhooks. That is a competitor's claim and I haven't confirmed it.If there is no signed event feed, the forged/replay/race tests apply to whatever you build instead. The rule that survives is to never let your backend act on events reported by clients, because client-reported events are attacker-controlled. If you poll, poll with a server credential and treat the response as the source.2. Does a read check live membership, or only the token?Not stated in the pages I read.Unknown. This decides which row of the revocation tables applies. It is a two-minute sandbox test: remove an occupant, then read with that user's still-valid token.3. Token lifetimeBoth user and application session tokens expire 2 hours after the last REST request, and each REST response carries an expiry header. I found no configuration for it.This is idle expiry, which is the dangerous row of the table above. A client that keeps making REST calls keeps the session alive. I don't know whether live chat-connection activity also extends it, and I haven't verified an absolute cap.4. How is membership removed?Removal goes through the dialog update call. A dialog's owner can remove any occupant (the REST reference says admins can too), and an occupant can remove only themselves. The SDK and dashboard docs show removing other users working for group dialogs, not for private (one-to-one) or public ones. Deleting a user removes them from group dialogs but not from private dialogs' occupant lists.Private dialogs have no documented remove-another-user operation. If revocation matters, a clinician-patient conversation modeled as a private dialog can't be revoked by removal. You'd have to delete the dialog for everyone (which needs the owner and loses history). Model sensitive conversations as group dialogs with an app-controlled owner. This is a design decision to make before launch, not after.5. Immutable IDs, self-registrationUser creation with an application session is documented as acceptable for prototypes. For production the docs recommend creating users from your backend with the API key and disabling application-session user creation. Users carry an external_user_id field intended to link to an outside system, and login, email and external ID must be unique.The identity-confusion attack depends on that production setting. Bind your app user to the platform's own user ID, written only by your backend, and don't resolve senders by login or email. external_user_id is a convenient mapping field, but the binding should still be enforced on your side.6. Can the agent be a distinct principal?Dialogs have owner, admin, and occupant roles.Make the agent a separate user and an occupant, never an admin or owner. To fence it at handoff by removing it, the dialog must be a group dialog, per row 4. I did not verify whether the agent's session can be invalidated server-side.7. Can you reconcile after a missed event?Dialog and message REST endpoints exist, and the update and delete references show them taking an API key. I did not verify listing and filter semantics for drift detection.Plan the reconciler as a poll of dialogs and occupants compared with your database. Confirm pagination and filter support in a sandbox first. Three observations from the table, none of them about vendor quality. Every platform has choices like these, and every one is a place the seam can fail: The documented 2-hour idle expiry changes the revocation answer. In the harness's token-only, idle-expiry rows, short TTLs and broker re-checks buy nothing, and the only thing that ends access is membership enforcement or a server-side kill. Whether QuickBlox enforces membership at read time is the first thing to test, and whether you can invalidate a session from the server is the second.The private-dialog limitation is the most actionable finding. It is a design constraint you can only avoid by choosing group dialogs up front.The absence of a documented signed event feed moves work to you. The ingress hardening in this article becomes a design for whatever sync mechanism replaces it. On the vendor's compliance statements: QuickBlox's own pages state that it offers HIPAA-ready hosting with a signed BAA, and that it is SOC 2 certified. I haven't verified those claims, and a BAA does not make your application compliant. The seam problems in this article are exactly the part a BAA doesn't cover. Cost and trade-offs Ingress overhead. Hardened ingress costs about 8.8 µs per event against 2.0 µs for the naive handler, in-process, on one machine (Python 3.12). Indicative only. In production, the dominant cost is one atomic store write per event.Token churn. Short TTLs convert revocation latency into mint load: 60 mints per user-hour at TTL 60, 120 at TTL 30. And if expiry is idle-based, TTL tuning may not be available to you at all.Dependence on the admin API. Forced removal and token kill are only as available as that API. The outbox hides outages from the revoking caller. It doesn't shorten them.Approval friction. Every privileged action needs a human click bound to specific arguments. The agent can't autonomously perform privileged operations. That is the intended outcome, and it will frustrate some product teams. Mitigate it with a good approval surface: show the exact arguments, make approval one tap, and expire quickly.Complexity. The hardened design adds a broker, an outbox, an approval service, a per-dialog epoch, and an anchored log. For an internal tool with a few trusted users, it is probably more than you need. Limitations of this work The simulator is simple: one dialog, instantaneous admin success when up, no clock skew, no partial failures inside the platform, a single-node claim store.The worst-case agent models the threat, not any LLM.I did not implement or test a membership reconciler, although I recommend one.The QuickBlox audit is documentation-only. Docs may be incomplete, inconsistent (the owner-versus-admin removal rule is stated differently in different pages), or out of date. Every "unknown" in the table is a sandbox test I haven't run.Remaining exposure none of this addresses: injected text can still influence unprivileged tools (what the agent says, what it summarizes), exfiltration through in-scope reads, and approval fatigue.Nothing here shows that any real platform is vulnerable to any attack in the harness. Conclusion The integration boundary is a consistency boundary, and the interesting failures are state disagreements and authority crossing it unchecked. The revocation window gives you a number to argue about, instead of a feeling, and the number depends on three platform properties you can look up before writing code: whether reads check live membership, whether token expiry is fixed or idle-based, and whether you can remove a member or kill a session from the server. For the AI participant, the real question is whose authority a tool call exercises. Intersecting principals' rights is necessary but not sufficient. A privileged call needs an approval bound to its exact arguments, or conversation text will eventually carry authority it shouldn't. Sources (QuickBlox documentation, retrieved 4 October 2026): Key Concepts: https://docs.quickblox.com/docs/key-conceptsAuthentication (JS): https://docs.quickblox.com/sdks/js-authenticationUpdate Dialog: https://docs.quickblox.com/reference/update-dialogiOS Dialogs: https://docs.quickblox.com/sdks/ios-chat-dialogsDashboard, Chat: https://docs.quickblox.com/docs/chat-dashboardDashboard, Users: https://docs.quickblox.com/docs/users-dashboardCreate User: https://docs.quickblox.com/reference/create-userDelete Dialog: https://docs.quickblox.com/reference/delete-dialog

By Igboanugo David Ugochukwu

CORE

A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

By Ahmed Tariq

Deploying Small Language Models on Sovereign Cloud

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

By Anuj Ashok Potdar