AI/ML

Every few weeks, a new name shows up in security news. Project Glasswing. Daybreak. Lightwell. CodeMender. Big Sleep. Each one is presented as the next answer to AI-driven cyberattacks. It is easy to read this as a race between AI labs, cloud providers, and security vendors, and to ask which one will win. That is the wrong question. Each of these programs strengthens a different part of the same system. Some improve discovery. Some help engineers understand which bugs actually matter. Some generate patches. Some govern who is allowed to use powerful cyber tools at all. None of them, on its own, is a complete answer. The useful question for a security team is not which vendor we should bet on. It is which part of our own system is weakest, and which of these pieces fills that gap. This article lays out the evidence for why the pace of the problem has changed, walks through the lifecycle every vulnerability goes through, maps the current tools, programs, and rulebooks onto that lifecycle, and then treats AI agents themselves as a new class of identity that needs the same governance as any other. Along the way, I point to my own earlier articles where they cover the underlying engineering discipline in more depth. Why the Old Patch Cycle No Longer Works The pressure is not hypothetical. A few figures, each traceable to its source, show the shape of the problem. SignalWhat it showsSourceA record 48,185 CVEs were published in 2025, a 263 percent rise from 2020The volume of vulnerabilities to sort through has grown far faster than security teams haveCloud Security Alliance, "The Collapsing Exploit Window"Mandiant's M-Trends 2026 puts mean time to exploit at roughly negative seven days for high-value targetsAttacks now often begin before a fix is even publicMandiant M-Trends 2026, Google CloudCISA's Binding Operational Directive 26-04 requires a three-day fix for vulnerabilities that meet four risk criteriaThe US federal government has formally shortened its own patch deadlines because of AI-accelerated exploitationCISA, Binding Operational Directive 26-04The average KEV catalog patch deadline in 2026 is 14.4 days, down from 19.7 the year beforeDeadlines across the board are compressing, not just for the most severe bugsFederal News Network, May 2026AI agents found and helped fix 1,072 Chrome security bugs across two releases in June 2026, more than the previous 23 releases combinedDiscovery tooling has genuinely gotten faster, not just louder in marketingGoogle, "Stronger with every update" None of this changes how many hours are in your team's week. The number that actually matters is the one CISA is trying to compress by regulation: the time between a fix exists and the fix is live in production. Every program in this article is, in one way or another, trying to shrink that number. The Lifecycle Every Vulnerability Goes Through Before comparing tools, it helps to agree on the steps a single weakness goes through, from discovery to a proven fix. Lifecycle of a vulnerability or a bug Discovery has become cheap. Deciding, fixing safely, and rolling out remain slow, mostly because they need judgment, testing, and coordination across teams. That is where most vulnerability management programs still struggle, and it is where the new wave of AI security tools is trying to help. This lifecycle is also a loop, not a straight line. A better way to see the whole program is as a continuous cycle that feeds back into itself. Vulnerability management Discovery finds what might be wrong. Context asks whether the vulnerable component is actually reachable, deployed, and exposed. Prioritization ranks it against everything else competing for the same engineering hours. Remediation turns a finding into an engineering change: a dependency bump, a code fix, a rotated credential, a corrected IAM policy. Governance makes sure the whole process is reviewable, approved where it needs to be, and evidenced. Learning feeds what was found back into better defaults so the same class of issue shows up less often. I covered the review and testing discipline that sits behind the remediation and governance stages in Code Review Core Practices, which stays relevant even when the patch is drafted by an AI agent rather than a person. Three Programs, Three Different Jobs Three named programs currently dominate the conversation. Each one is strongest at a different point in the lifecycle above. Project Glasswing: Discovery at the Infrastructure Layer Anthropic launched Project Glasswing in April 2026. Launch partners include Amazon Web Services, Apple, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks, working with Claude Mythos Preview on defensive tasks. Anthropic committed up to 100 million dollars in usage credits and 4 million dollars in donations to open source security groups, and later extended access to roughly 150 more organizations across more than fifteen countries (Anthropic, Project Glasswing). The program targets critical infrastructure and widely used open source software, giving trusted defenders early access to a strong model before similar capability becomes broadly available. Mozilla has reported that the underlying model helped it find and patch 271 vulnerabilities in a recent Firefox release. OpenAI Daybreak: Tiered Access for Defenders OpenAI announced Daybreak on May 11, 2026, describing it as bringing together its frontier models, Codex Security, and a network of security partners to accelerate cyber defense and build security into software from the design stage rather than only patching after the fact (OpenAI, Daybreak). Access is split into two tiers that require separate approval. Daybreak's tiered access In September 2026, OpenAI added a subsidized access tier for organizations that keep essential services running, describing it as billions of dollars in access over roughly six months. Project Lightwell: Fixing Open Source Without Breaking Production IBM and Red Hat announced Project Lightwell on May 28, 2026, as a five-billion-dollar commitment backed by a global force of more than 20,000 engineers, and moved it to commercial launch on July 8 (IBM Newsroom, May 28, 2026; IBM Newsroom, July 8, 2026). It targets a specific and common problem. A library has a known bug. The official fix is a newer version, but the newer version breaks a certified production setup. Lightwell Network launched with a catalog of more than 6,500 remediated, signed, and certified dependencies across ecosystems such as Java and Python, giving teams a patched version of the release they already run instead of forcing a risky upgrade. Lightwell Clearinghouse Premier, in limited availability, adds coordinated patch embargo handling. In August 2026, IBM and Red Hat extended free access to more than 185 universities and 100 NGOs and think tanks that lack the engineering capacity to keep pace on their own (IBM Newsroom, August 4, 2026). Comparing the Three GlasswingDaybreakLightwellLed byAnthropic and partnersOpenAI and partnersIBM and Red HatCore questionWhere are the bugs in critical software?How can defenders use strong AI safely?How do we fix open source bugs without breaking production?Strongest lifecycle stepsFind, patchFind, check, proveFix, test, roll outAccess modelInvited partners, expandingApproved defenders, two tiersPaid subscription, with free access for universities and NGOsPrimary scopeCritical and open source infrastructureAn organization's own code and systemsOpen source dependencies in productionMain limitationMostly invite-basedSeparate approval for each tierCovers dependencies, not custom code Tools That Find and Patch Code Directly Alongside the three named programs, a set of tools now work directly inside the discovery and fix stages. ToolOwnerFunctionLifecycle stepSourceBig SleepGoogle DeepMind and Project ZeroSearches for memory safety bugs in widely used codeFindGoogle, Chrome security blogCodeMenderGoogle DeepMindPatches new bugs and rewrites existing code to remove whole classes of bugsFind, fixDeepMind, "Introducing CodeMender"Codex SecurityOpenAIBuilds an editable threat model from a repository and generates and tests patchesFind, checkOpenAI, DaybreakClaude SecurityAnthropicPublic beta for organizations outside GlasswingFind, checkAnthropic, Project Glasswing DeepMind is explicit that every CodeMender patch is reviewed by a person before it ships. That single fact matters more than the raw bug counts. It matches a habit I described in Ansible Code Scanning and Quality Checks With SonarQube: automated scanning finds candidates, but a human still signs off before anything reaches production. There is also support work happening around the big three programs, aimed at a second-order problem: faster discovery means more reports than open source maintainers can review. Anthropic has partnered with the Open Source Security Foundation's Alpha-Omega project to help maintainers triage incoming reports, and OpenAI's Daybreak commitment includes direct funding for groups such as Trail of Bits. Palo Alto Networks has also worked with IBM and Red Hat so its network-level protections can block attacks while a Lightwell fix is tested and rolled out, a practice known as virtual patching. It buys time. It does not replace the fix. What This Sets Up Discovery is now genuinely fast. Three well-funded programs and a handful of dedicated tools are throwing real capability at the find-the-bug and build-the-fix steps of the lifecycle. But finding more bugs faster does not, by itself, get you a faster or safer program. It just moves the bottleneck. The next problem is knowing which of those findings actually matter, making sure the AI agents doing this work are governed like the privileged systems they are, and having a plan that does not depend on an invitation to any of these programs. That is where Part 2 picks up. Further Reading Code Review Core PracticesAnsible Code Scanning and Quality Checks With SonarQube

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

An LLM application can post a higher evaluation score and still become less useful in production. The contradiction is only apparent. An evaluation score is not quality itself, but it is an estimate produced by a particular dataset, metric, judge, sampling strategy, and aggregation rule. If any of those differ from production, optimization can move the score upward while moving actual system behavior in the wrong direction. Current evaluation guidance explicitly recommends task-specific tests that reflect real-world distributions, production data, continuous evaluation, and calibration against human judgment rather than treating a single numerical score as sufficient. A Better Score Can Be a Worse Signal The cleanest way to frame the problem is distributional. Let an offline evaluation approximate expected utility as E[x ~ P_eval][U(model, x)], while production quality depends on E[x ~ P_prod][U(system, x)]. The two quantities are comparable only when the evaluation distribution and utility function reasonably represent production. Static test sets often underrepresent ambiguous requests, multilingual traffic, long conversations, malformed inputs, retrieval failures, tool errors, and domain-specific edge cases. OpenAI's evaluation guidance specifically warns against datasets that do not reproduce production traffic patterns and recommends mixing production, historical, domain-specific, and human-curated data. Aggregate scores create a second failure mode. A candidate can improve common easy cases enough to raise the mean while regressing on a low-volume but high-value slice such as payment disputes, medical terminology, code generation, or long-context retrieval. The release decision therefore needs both aggregate improvement and slice-level floors. Python def release_allowed(baseline, candidate): critical = ["long_context", "tool_use", "high_risk", "multilingual"] if candidate["overall"] <= baseline["overall"]: return False for segment in critical: if candidate[segment] < baseline[segment] - 0.01: return False return True This gate encodes a simple principle where average quality may improve, but important capabilities are not allowed to silently collapse. The threshold itself should come from application risk and measurement uncertainty rather than from an arbitrary universal value. A third problem is uncertainty. A 0.7-point gain on a small test set is not automatically a real improvement, as it can be sampling noise, judge variability, or sensitivity to the selected prompts. Release comparisons should therefore carry confidence intervals or bootstrap estimates alongside point scores. Chatbot Arena's ranking methodology explicitly models uncertainty and uses confidence intervals rather than treating estimated scores as exact. Benchmark contamination can make the gap wider. LiveBench was designed around frequently refreshed questions and objective ground truth scoring specifically because static benchmark data can enter later training corpora and because human or LLM judging can introduce additional bias. Its design highlights an important distinction where a benchmark can measure capability under controlled conditions without necessarily estimating production utility. The Evaluator Can Become the Optimization Target LLM-as-a-judge makes large-scale evaluation practical, but the judge is another model with systematic preferences. The MT-Bench and Chatbot Arena research identified position bias, verbosity bias, self-enhancement effects, and reasoning limitations in model-based judging. A separate study focused on verbosity bias found that GPT-4 preferred longer responses more strongly than human evaluators in the tested setting. That creates a classic proxy optimization problem. If a judge rewards detailed prose, prompt tuning may produce longer answers that score better even when production users prefer concise answers. If a pairwise judge favors the first position, repeated experiments can select changes partly because of ordering rather than capability. Automated judges therefore need calibration and invariance checks. Python def stable_preference(prompt, candidate, baseline, judge): forward = judge(prompt, candidate, baseline) reverse = judge(prompt, baseline, candidate) if forward == "A" and reverse == "B": return "candidate" if forward == "B" and reverse == "A": return "baseline" return "unstable" Running both orders does not remove every judge defect, but disagreement exposes position sensitivity instead of converting it into a false quality signal. OpenAI's current evaluation guidance similarly recommends pairwise or pass/fail grading, length controls, detailed rubrics, and validation of model graders against human labels before scaling them. Human preference data also changes the picture. Chatbot Arena was built around live, anonymous pairwise comparisons on crowdsourced prompts and reported more than 240,000 votes in its 2024 paper. The study found that crowd questions were diverse and discriminative and that crowd preferences showed good agreement with expert raters. The result does not make live preference a universal metric, but it demonstrates why static benchmark performance and real interaction preferences should not be assumed to be interchangeable. Production Quality Requires System-Level Measurement A deployed LLM rarely operates alone. Retrieval, prompt templates, tool routing, conversation memory, safety filters, output parsers, model versions, and latency budgets all influence the final result. A model upgrade can improve answer quality in isolation while increasing tool-call errors, changing structured-output behavior, raising latency, or interacting poorly with retrieval context. Evaluation therefore has to version and measure the system, not only the base model. A production trace should carry enough provenance to reconstruct the behavior being scored. Python trace = { "model_version": model_version, "prompt_version": prompt_version, "retriever_version": retriever_version, "tool_schema_version": tool_schema_version, "latency_ms": latency_ms, "task_success": task_success, "feedback": feedback } Without this metadata, a quality decline can be visible without being attributable. With it, offline regression sets can be regenerated from real failures and segmented by the component versions involved. OpenAI's guidance recommends logging during development, mining logs for evaluation cases, incorporating production data, and continuously expanding eval sets as new nondeterministic failures appear. Another source of divergence appears when evaluation cases preserve the expected input but remove the production context surrounding it. Real requests arrive with conversation history, retrieved documents, tool results, authorization state, latency constraints, and occasionally malformed upstream data. Each dependency introduces failure modes that a model-only benchmark cannot observe. A response may score highly when evaluated against a clean prompt while failing after retrieval returns stale evidence or a tool produces an incomplete payload. Production evaluation therefore needs to preserve execution context alongside the input and output. Capturing traces, dependency results, and failure labels makes regressions attributable to the system component that actually caused them. Production measurement should also separate model quality from product outcomes. A support assistant can be fluent and factually grounded while still increasing escalation rate because it asks unnecessary follow-up questions. A coding assistant can receive strong judge scores while generating changes that fail compilation or tests. A RAG system can produce polished answers while retrieval recall falls. These cases require executable checks, task-success signals, retrieval metrics, human preference, latency, and cost to coexist rather than being compressed prematurely into one score. OpenAI's evaluator taxonomy makes the same distinction between metric-based checks, human evaluations, and model graders, noting that each has different strengths and failure modes. Production Quality Is the Real Release Gate The strongest evaluation pipeline is therefore not the one with the most benchmarks. It is the one that preserves the relationship between offline measurements and production outcomes. Fresh production samples should feed offline regression suites; critical slices should have explicit guardrails; automated judges should be tested against human labels and judge prompts; versions should be tracked like model versions; and release decisions should consider uncertainty rather than tiny score differences. Live benchmarks can reduce contamination risk, and human preference systems can reveal behavior missed by static tests, but neither replaces application-specific evidence. An improved LLM evaluation score matters only when the metric remains aligned with the behavior that production actually needs. Once the score becomes the target rather than the measurement, optimization begins rewarding the evaluator instead of the application. Reliable AI engineering therefore treats evaluation as a continuously calibrated measurement system connected to real traffic, real failures, and real outcomes. The objective is not to make the dashboard greener, but to make deployed behavior measurably better.

By Uthej Mopathi CORE

The Real Question Is Not: Which Grader Is Smartest? When teams begin building AI evaluations, it is tempting to reach for an LLM-as-a-judge because it can read almost anything and return a score. That flexibility is useful, but it can also hide a simpler option. Suppose an agent is asked to process a refund. The final response says, "Your refund has been completed." You could ask another LLM whether that answer sounds correct. A better grader is to check the transaction state in the database or test environment. The database is closer to ground truth than the prose. That leads to a practical rule: Grade the strongest evidence the task exposes. The more objective the evidence, the less judgment the grader needs. A unit test should beat an opinion. A database state check should beat a natural-language claim. A schema validator should beat an LLM deciding whether JSON "looks valid". The four grader families below form a useful spectrum. They are not competitors. They solve different classes of evaluation problems. This same layered pattern appears in modern evaluation guidance and tooling, which combine code-based, reference-based, model-based, and human grading rather than relying on one universal judge. Figure 1. The grader spectrum: move toward judgment only when objective evidence is insufficient. Grader Comparison at a Glance Grader Best For Speed Cost Main Risk Code / rules Schemas, calculations, unit tests, system state Fast Low Brittle if checks are too narrow Reference matching Known answers, labels, expected values Fast Low Rejects valid variations LLM-as-a-judge Relevance, groundedness, quality, style Medium Medium Bias and non-determinism Human review Ambiguous, expert, or high-risk decisions Slow High Cost, fatigue, disagreement 1. Rule-Based Graders: When Correctness Can Be Executed Rule-based graders use code to verify a condition. They include exact assertions, regular expressions, schema validation, unit tests, static analysis, calculations, tool-call checks, and environment-state verification. These graders should be the default whenever correctness is machine-verifiable. They are fast, inexpensive, reproducible, and easy to debug. If the same input and output are evaluated twice, the result should normally be the same. Consider an extraction task where the model must return JavaScript Object Notation (JSON) with an order_id and confidence field. The most important questions are not subjective: Does the output parse? Are the required fields present? Is confidence between 0 and 1? A validator can answer those questions exactly. Python import json def grade(output: str) -> bool: try: data = json.loads(output) return (isinstance(data.get("order_id"), str) and isinstance(data.get("confidence"), (int, float)) and 0 <= data["confidence"] <= 1) except (json.JSONDecodeError, TypeError): return False This check is intentionally boring. That is a feature: the grader does not need to understand the prose, only the contract. Rule-Based Graders Are Strong, But They Can Be Too Rigid Deterministic does not automatically mean well-designed. A grader can still be too rigid if it checks a specific implementation detail instead of the required system behavior. For example, suppose an AI coding agent is asked to fix an API endpoint so that it returns HTTP 404 when a requested record does not exist. The agent implements the behavior correctly, but uses a different code path than the reference solution. A grader that compares the submitted code with the reference patch may incorrectly mark the solution as wrong simply because the implementation is different. A stronger deterministic grader would call the API with a missing record and verify that the response is actually 404. The principle is simple: validate the required behavior, not whether the implementation matches one expected solution. 2. Reference Matching: When a Known Answer Exists Reference-based graders compare an AI output with an expected answer. This works well for questions with a stable target: classification labels, extracted identifiers, translations with controlled terminology, benchmark answers, known numerical results, or expected API fields. The simplest version is exact match. In practice, normalization often makes it more useful: trim whitespace, ignore capitalization where appropriate, normalize date formats, or compare structured fields instead of raw strings. Reference matching becomes weaker as the answer space becomes more open-ended. If a question asks, "Why did the deployment fail?" there may be several correct explanations with different wording. Exact match can then punish correct answers. Fuzzy or semantic similarity can help, but as the comparison becomes more interpretive, you are moving closer to model-based judgment. A good rule is to use references for facts, labels, and values, not as a universal definition of good writing. 3. LLM-as-a-Judge: When Quality Requires Interpretation Some AI outputs cannot be graded with a simple pass/fail rule. For example, you may need to judge whether an answer is relevant, clear, complete, or supported by the provided context. These are good use cases for an LLM-as-a-judge. An LLM judge usually receives the original task, the AI-generated response, and a grading rubric. It then returns a score, label, or short explanation. This makes it useful when you need to evaluate many responses that require human-like judgment. The rubric should be specific. Instead of asking: “Is this a good answer?” Grade separate qualities such as: RelevanceFactual supportInstruction followingCompletenessClarity The judge should also receive any source material it needs to verify the response. If there is not enough evidence, it should be allowed to return something like “insufficient evidence” instead of guessing. LLM judges are useful, but they are not perfectly objective. Their scores can be influenced by things such as answer order, response length, or writing style. For that reason, the judge itself should also be evaluated. A common approach is to compare its scores with a smaller set of human-reviewed examples and check whether the two generally agree. How to Make an LLM Judge More Trustworthy Start with a clear rubric and narrow the question being graded. A judge that evaluates five unrelated dimensions in one prompt is harder to debug than several focused checks. Next, calibrate it against human-labeled examples. If experts say 40 responses pass, and the judge disagrees on 15 of them, the problem is not solved by running the judge on 10,000 more examples. Improve the rubric, examples, model choice, or grading method first. For preference tasks, pairwise comparison can sometimes be easier than assigning an absolute 1-to-10 score: show two answers and ask which better satisfies a specific rubric. But pairwise graders should also be tested for position effects by swapping answer order on a sample of cases. Finally, do not ask an LLM to judge a fact that your system can verify directly. A judge can assess whether a support response explains a refund clearly. It should not be the authority on whether the refund actually exists in the transaction system. 4. Human Review: When Judgment Is the Ground Truth Human review remains important when success is ambiguous, domain expertise is required, the cost of a wrong decision is high, or you need a trusted reference for calibrating automated graders. For example, OpenAI's GDPval uses expert graders as the primary standard and treats an automated grader as an approximation rather than a replacement. Humans are especially useful for creating the first high-quality labels, reviewing disagreements between graders, evaluating unusual edge cases, and periodically checking whether an LLM judge has drifted away from expert expectations. Human review is not perfectly deterministic either. Two experts can disagree about tone, completeness, or acceptable risk. Structured rubrics, reviewer training, and inter-rater agreement help turn human judgment into a more consistent signal. The goal is therefore not to put humans in every evaluation loop. It is to spend human judgment where it adds information that automated checks cannot provide. A Practical Decision Framework A grader-selection decision can usually be made by walking from objective evidence toward subjective judgment. First ask whether success can be verified with code. If yes, stop there. If not, ask whether there is a stable reference answer. If there is, use reference matching. If the task is open-ended but has a clear rubric, use an LLM judge and calibrate it with humans. If experts themselves must interpret context, or the decision carries significant risk, use human review directly or as the final escalation layer. Figure 2. A simple decision tree for selecting a grader. One AI Task Can Need More Than One Grader Real AI systems rarely have only one definition of success. A customer-support agent, for example, may need to take the correct action and communicate it well. Those two requirements should not be collapsed into one subjective score. Imagine an agent that handles a refund request. A robust evaluation could check four different things: What to Grade Best Grader Example Check Refund actually processed Rule-based Check transaction/database state Refund amount is correct Reference-based Compare with expected amount Response is clear and helpful LLM-as-a-judge Rubric for clarity and relevance Novel policy edge case Human review Subject-matter expert decision This layered design makes failures easier to diagnose. If the database check fails, you have an execution problem. If the action succeeds but the LLM judge flags the explanation, you have a communication-quality problem. A single overall score would hide that distinction. It also creates a natural escalation path. Most evaluation cases can stop after cheap automated checks. Only the cases that remain uncertain, disagree across graders, or carry high risk need human attention. Common Grading Mistakes Using an LLM judge for deterministic facts. Arithmetic, JSON validity, tool success, API status, database state, and unit-test results should normally be checked directly.Using exact match for open-ended answers. Correct responses can use different wording, ordering, or reasoning. Match the grader to the size of the valid answer space.Grading what the model says happened instead of what happened. For agents, inspect the environment or system of record whenever possible.Writing vague LLM rubrics. "Good, helpful, and accurate" sounds reasonable but leaves too much room for interpretation. Define dimensions and observable criteria.Never calibrating the judge. An automated grader is another model or program in your system. It can be wrong, biased, or brittle. Test it against trusted examples.Treating human review as infallible. Humans also disagree and fatigue. Use structured criteria and measure disagreement on important evaluations. A Better Default: Build a Grading Stack Instead of choosing one universal grader, route each quality requirement to the least subjective evaluator that can answer it reliably. A useful production stack is: hard checks first, reference checks second, LLM judgment third, and human escalation last. This ordering keeps evaluation fast and explainable while preserving a path for nuance. The key is not to maximize automation at all costs. It is to preserve strong evidence. If an LLM score conflicts with a unit test, the unit test should usually win. If a rubric-based judge repeatedly conflicts with expert reviewers, the judge needs recalibration. The grading hierarchy should reflect how close each signal is to the actual definition of success. Figure 3. A layered grading stack keeps objective checks cheap while preserving human escalation. Conclusion There is no single best grader for AI systems because AI tasks expose different kinds of ground truth. Rule-based graders are best when correctness can be executed.Reference matching works when a stable expected answer exists.LLM-as-a-judge is valuable for scalable evaluation of open-ended qualities, but it needs a clear rubric and human calibration.Human review remains essential for ambiguity, expertise, calibration, and high-risk decisions. The simplest rule is still the most useful: use the most deterministic grader the task allows. Do not ask an LLM to guess what a test, parser, database, or reference answer can prove. Save judgment for the parts of the system that actually require judgment.

CI failures often expose a narrow defect through a broad stream of evidence such as compiler output, test reports, stack traces, changed files, dependency metadata, and build logs. Large language models can turn that evidence into candidate repairs, and recent industrial research has shown that LLM-assisted CI repair can resolve a meaningful share of compilation failures when the model receives carefully selected context. The difficult part is not generating a patch. It is ensuring that automated repair changes only code justified by the failure, leaves protected files untouched, and proves that the original failure is actually fixed. Automated program repair research has repeatedly identified patch overfitting as a core risk: a patch can satisfy available tests without being semantically correct. Capture Failure Evidence Before Asking for a Fix A repair agent should start from structured CI evidence rather than a raw prompt containing an entire build log. GitHub Actions artifacts can persist test output, logs, and other files after a job completes, making them a practical handoff format between the failing job and a later repair job. For a Maven project, the failing job can retain both console output and Surefire reports: YAML - name: Run tests run: ./mvnw test 2>&1 | tee ci.log - name: Upload failure evidence if: failure() uses: actions/upload-artifact@v4 with: name: failure-evidence path: | ci.log target/surefire-reports/ The repair process should normalize this data into a compact incident object containing the failing test, exception type, stack trace, failing command, commit SHA, and original pull-request diff. Current surveys of LLM-based software repair describe retrieval and analysis augmentation as important layers for repository-level repair, reinforcing the value of selecting failure-relevant context before generation. Before patch generation, the failure should be reproduced in an isolated checkout. Git worktrees support linked working directories and can create a detached worktree for experimental changes without disturbing the primary checkout. A repair job can create a disposable workspace and rerun only the failing test: Shell git worktree add --detach ../repair "$HEAD_SHA" cd ../repair ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test Maven Surefire supports selecting a specific test class and, for supported providers, individual test methods. If the failure cannot be reproduced, automatic patching should stop. A passing rerun provides no reliable oracle for judging a generated fix. Turn Failure Localization Into an Edit Boundary The central control is an explicit distinction between readable and editable files. A failing test, configuration file, or build script may be necessary for diagnosis without being safe for autonomous modification. A scope document can represent that boundary: YAML failure: PaymentServiceTest.shouldRetryTimeout readable: [src/test/java/com/acme/payment/**, src/main/java/com/acme/payment/**, pom.xml] editable: [src/main/java/com/acme/payment/**] protected: [src/test/**, pom.xml, .github/**, db/migration/**] The editable set can be derived from the original PR diff, stack-trace locations, and a small neighborhood of related production symbols. Git provides machine-readable changed-path output through git diff --name-only, while pathspecs can limit comparisons to selected repository regions. The model can inspect broader context, but the orchestrator should reject any patch that escapes the approved surface. A validator needs a repair baseline captured immediately before the model edits the worktree: Shell REPAIR_BASE=$(git rev-parse HEAD) git diff --name-only "$REPAIR_BASE" > repair-files.txt while read -r path; do case "$path" in src/main/java/com/acme/payment/*) ;; *) echo "Out-of-scope change: $path"; exit 1 ;; esac done < repair-files.txt This check matters because prompt instructions such as "modify only relevant code" are advisory, while a diff gate is enforceable. Git also provides exit-status behavior through git diff --exit-code, which is useful for automated policy checks. Generate a Patch, Not an Open-Ended Coding Session The repair model should receive a bounded request like normalized failure data, relevant source fragments, the original PR diff, editable paths, protected paths, and the exact verification command. The response should be structured around a patch rather than unrestricted shell execution. Java RepairRequest request = new RepairRequest( failure, relevantSource, originalDiff, editablePaths, protectedPaths, verificationCommand ); Patch patch = repairModel.generate(request); patchValidator.validate(patch); patchApplier.apply(patch); The model proposes the smallest plausible correction where the orchestrator remains responsible for permissions and validation. This separation reflects a broader trend in LLM-based program repair toward procedural and agentic workflows that combine generation with retrieval, analysis, execution feedback, and external control. For an initial implementation, tests and CI configuration should remain read-only. A failure that genuinely requires changing a test, dependency declaration, migration, or workflow can be escalated for manual review rather than allowing the agent to broaden its own permissions. Validate More Than a Green Build A candidate patch should first rerun the exact failing test. Only after that test passes should validation expand to the affected module and then the normal CI suite. Shell ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test ./mvnw -pl payment test ./mvnw verify Passing tests alone are not proof of correctness. Automated program repair research documents the overfitting problem, where generated patches satisfy the available test suite without correctly repairing the underlying behavior. Scope validation therefore acts as an additional safeguard. A successful repair should preserve protected files, avoid dependency changes, keep the patch reviewable, and retain the original failing test unchanged. A final guard can verify that tests were not modified during repair: Shell git diff --quiet "$REPAIR_BASE" -- src/test || { echo "Repair modified protected tests" exit 1 } The resulting output should be an evidence package rather than an opaque "fixed" status, but contain the root-cause summary, exact patch, changed-file list, focused-test result, module-test result, and full-CI result. That evidence makes the automated repair reviewable. To ensure a patch does not escape its intended scope, the workflow can run the agent in an isolated Git worktree and enforce explicit path-based allowlists. For example, a CI script might use git worktree add -d /tmp/agent main so the agent’s changes occur in a separate directory. After the agent generates a commit, the pipeline can run git diff --name-only origin/main...HEAD and check that all file paths match the allowlist (for instance, grep -qvE '^(src/payment/|config/)' rejects unauthorized modifications). Protected paths (e.g. .git/) would instantly be blocked by the allowlist. The pipeline then reruns the failing test class in isolation (mvn -Dtest=PaymentServiceTest test) to confirm the issue is fixed, and executes the full test suite. A quick diff or mutation-based check can detect if any test assertions were weakened or disabled. Finally, a canary merge or smoke-test job automatically undoes the change on failure. For instance, a GitHub Actions step might include the following so that any failing health check immediately triggers an automated rollback: SQL - name: Rollback if smoke tests fail if: steps.smoke.outcome == 'failure' run: ./rollback.sh These combined safeguards (worktree isolation, allowlist enforcement, retests, and rollback triggers) help guarantee the AI-generated patch is minimal, correct, and reversible. Keep the Repair Job Less Privileged Than the Code It Examines An autonomous repair workflow processes repository content and may execute code originating from a pull request, so CI permissions need careful separation. GitHub warns that privileged workflow_run and pull_request_target workflows must not blindly check out untrusted pull-request code, and recommends granting GITHUB_TOKEN only the minimum required permissions. A safer pattern is to perform diagnosis and patch generation with read-oriented credentials, store the candidate patch as an artifact, and use a separate trusted step to create a branch or pull request. Workflow artifacts should also be treated cautiously when produced by untrusted jobs. This prevents the repair agent from turning a build failure into a path toward repository write access. Conclusion A useful CI repair agent is not an LLM attached directly to a failing build. It is a constrained repair pipeline that captures structured failure evidence, reproduces the defect, narrows the editable surface, generates a minimal candidate patch, rejects out-of-scope changes, and validates the result from the failing test through the full CI suite. Industrial evidence shows that LLM-assisted repair can be effective in CI settings, but automated program repair research also shows why a green build cannot be treated as sufficient proof. Enforced edit boundaries, isolated execution, layered testing, and least-privilege automation convert model-generated code from an uncontrolled suggestion into a reviewable, evidence-backed repair mechanism.

By Uthej Mopathi CORE

TL;DR: A 2×2 Matrix on AI for Agile Practitioners You probably think making sense of AI means keeping up with every new tool, agent, and pricing tier. Steve Jobs faced a similar mess at Apple in 1997, with a dozen versions of the Macintosh, and he fixed it with a two-by-two grid on a whiteboard. Almost 30 years later, the “Jobs Matrix for AI” sorts the AI tool market and, more usefully, the AI use cases of Scrum Masters, Product Owners, Agile Coaches, and anyone else in agile product development. Replace Consumer and Pro with Team and Organization, and Desktop and Portable with Chatbot and Agent, and you get four boxes that show where a use case belongs, what it takes to move it, and which box should stay empty. Thesis: This article applies Steve Jobs’ 1997 Consumer/Pro, Desktop/Portable matrix first to the 2026 AI tool market, and then to agile practices, producing a 2×2 matrix that sorts AI use cases by required decision rights and by delegated authority. Disclaimer: I read Charniak/McDermott’s book on “Artificial Intelligence” decades ago; of course, I use AI for research, translations, proofreading, challenging story arcs and article structures, and summarization. It is a production tool, not a substitute for thinking. Kudos to Claude Design for creating the graphics for this post. Four Boxes Saved Apple From Its Own Product Catalog In 1997, after weeks of product reviews, Jobs had seen enough. According to Walter Isaacson’s account in the Harvard Business Review, he stopped the meeting, walked barefoot to a whiteboard, and drew a two-by-two grid with “Consumer” and “Pro” above the columns and “Desktop” and “Portable” beside the rows: four great products, one per box, and everything else canceled. “Deciding what not to do is as important as deciding what to do,” he later told Isaacson. (Remember Simplicity–the art of maximizing the amount of work not done–is essential?) Jobs presented the matrix publicly in his iMac introduction on May 6, 1998, and at Seybold in September 1998, he added the reason that matters to anyone running a portfolio: with only four platforms, Apple could put its A team on every one of them instead of a B or C team on any. The same keynote showed the matrix’s most useful feature, the empty box: Apple had no consumer portable yet, and Jobs could only promise one for the following year, so everyone, the competition included, could see what was missing until the iBook filled that box in July 1999. The AI Tool Market Fits the Same Matrix, for Now Keep Consumer and Pro, and replace Desktop and Portable with Chatbot and Agent. A chatbot returns material that a person then uses, whereas an agent receives the authority to carry out a bounded task through several steps or systems. ChatGPT defines the Consumer Chatbot; enterprise chat, such as Copilot Chat in Microsoft 365, holds the Pro Chatbot box; and the Pro Agent box holds coding agents like Claude Code and Codex, plus Microsoft’s Copilot Cowork, which, according to Microsoft, runs defined work end to end, returns a finished result, and stays off until an administrator enables it. The Consumer Agent box was the thin one until Meta launched Muse on September 8, 2026; by September 24, it had passed 3.4 million downloads, according to Sensor Tower estimates reported by TechCrunch. Downloads measure interest rather than sustained use, which makes Muse the first candidate for the “iBook position,” with its staying power still unproven. The boxes leak, too: Muse lives in a chat interface, and Microsoft 365 Copilot offers a toggle from chat into Cowork, so I would treat the tool matrix as a snapshot (my guess: it will not survive the next 6 months unchanged). Your Jobs Matrix for AI: Team or Organization, Chatbot or Agent In a professional environment, nobody is a consumer, and the organization buys the tools anyway, so the columns describe the context and decision rights that the work requires. Two questions place any use case in a box: What authority does the AI have after the request? If it returns material for a person to use, it is a chatbot use case; if it carries out a bounded task through several steps or systems, with defined review and approval points, it is an agent use case. What starts it does not matter: a Developer can ask an agent to implement a change, and a schedule can trigger a plain summary.Who must contribute, authorize, or respond? If one product team, its Product Person included, can understand and act on the work, it is a Team use case; if several teams or organizational functions must get involved, it is an Organization use case. Typical use cases that come to mind are: Team Chatbot Summarizing the notes from Sprint Planning, the Sprint Review, or the Retrospective, including a scheduled summary that a person reads before anyone acts on it.Challenging a Product Backlog item for ambiguities and missing acceptance criteria before refinement.Drafting test cases from acceptance criteria.Designing a Retrospective around a specific team problem.Explaining legacy code or domain terms to a new team member. Requires: A practitioner with judgment who makes AI use visible to the team. Where it fails: Private, invisible use, for example, a Product Owner who generates Product Backlog items instead of talking to stakeholders. Team Agent After every event, extracting the agreed actions, checking each one with its owner, and posting only the approved items where the team works.Implementing a well-specified Product Backlog item and opening a pull request that the Developers review against the Definition of Done and development standards.Checking the Product Backlog weekly for stale, duplicate, or unrefined items and proposing changes to the Product Owner.Scanning the team’s codebase nightly for vulnerable dependencies and opening draft fixes.Publishing release notes from the shipped changes once the Product Owner has approved the draft. Requires: A named review standard for every output and enough review capacity to apply it. (Capacity refers to humans; an LLM-as-a-judge-only approach is a tricky proposition.) Where it fails: Output grows faster than the team’s capacity to review it, and approval shrinks to a click. Organization Chatbot Asking “Who decided X, when, and why?” across the organization’s documents.Asking which teams work on anything related to a given company objective.Checking governance and compliance questions, for example, whether customer data may be used in an experiment.Synthesizing past quarterly reviews before a strategy session.Querying the organization’s own user and market research. Requires: Current, well-kept documentation and clear permissions. Where it fails: It retrieves what the organization wrote down, including every stale and contradictory page. Organization Agent Producing a daily digest of the past 24 hours of customer feedback and support issues, compared with the product roadmap, listing gaps and emerging patterns for a product leadership report.Scanning all teams’ plans for dependencies before they turn into blockers.Compiling a weekly portfolio report from the actual work instead of from status meetings.Routing product usage anomalies to the owning team.Tracking competitor releases and comparing them with the roadmap. Requires: Current data from several sources, clear decision rights, and governance over what the agent may do unattended. Where it fails: Automated reporting replaces the conversation, and the agents’ lists of gaps quietly become the strategy. Deciding Which Boxes of the Jobs Matrix for AI Stay Empty Jobs used his matrix to cancel products, and an agile practitioner’s matrix should do the same. First, ask whether AI should touch the task at all, since a script, a checklist, or a ten-minute conversation often does the job better (the first question of the A3 Delegation System; see below). Second, before a use case moves into an agent box, name four things: Standard: What counts as good output.Owner: Who answers for the result.Permissions: What the agent may read, change, and send.Review burden: Who checks the output, how often, and at what cost. If you cannot name all four, do not delegate the task to an agent; keep a person in charge of the work, or leave AI out of it. That test also exposes a costly shortcut: leadership that starts with the Organization Agent box because it wants visibility first, dashboards are in high demand, and it seems likely to repeat the Agile transformations that began with tooling and reporting. The standard differs by output, which is why the Definition of Done and development practices cover code but not every piece of agent work: Code implementing a Product Backlog item: Review standard: The agreed-upon development practices and, likely, the Definition of Done.Who approves: The Developers.Action items from an event: Review standard: Each owner confirms their item.Who approves: The item’s owner.Release notes: Review standard: Accuracy against the shipped changes and a tone that fits the audience.Who approves: The Product Owner.Feedback digest and gap list: Review standard: Named sources and flagged unsupported claims.Who approves: The owner of the digest.Portfolio report: Review standard: Figures traceable to the work.Who approves: Whoever sends it upward. One Use Case, Two Boxes For example, take the customer feedback digest and keep it in the Organization column. As a chatbot use case, someone in product leadership asks Claude to examine last week’s support tickets and survey comments against the roadmap and decides which findings deserve a conversation with the teams. As an agent use case, an agent gathers feedback from the same sources every morning, compares it with the roadmap, and delivers a recurring list of gaps. Sources and columns stay the same, yet the move requires new authority, ownership, and review: someone must own the sources, define what counts as a gap, check claims the data does not support, and decide when to pause the digest. The agent may gather and compare, whereas deciding whether a gap deserves attention remains product strategy, a decision whose accountability stays with people. That is the rule across all four boxes: AI may prepare or carry out bounded work, and people keep the decisions and accountability defined for that work. If nobody can take on those jobs, leave the agent box empty. Jobs Matrix for AI – Conclusion: Find Your Empty Box Take last week’s AI use in your team, and put every instance into one of the four boxes. If you end up with an empty box, it reflects a decision about capacity and authority, so decide whether it is empty because nobody has got to it yet or because nobody could own it if you filled it. Key Questions This Article Answers What Was Steve Jobs’ 2×2 Product Matrix at Apple? Steve Jobs’ product matrix was a two-by-two grid he drew in 1997 to cut Apple’s lineup to four computers, with Consumer and Pro across the top and Desktop and Portable down the side. According to Walter Isaacson, Apple canceled everything outside the four boxes. By 1998, the iMac, Power Macintosh G3, and PowerBook G3 filled three boxes, and the iBook filled the consumer portable box in July 1999. What Is the Difference Between an AI Chatbot and an AI Agent? The difference is the authority the AI has after the request that starts it. A chatbot returns material for a person to read and decide how to use. An agent carries out a bounded task through several steps or systems, with defined review and approval points. A Developer can ask an agent to implement a change, and a schedule can trigger a plain chatbot summary. How Can Agile Teams Categorize Their AI Use Cases? Agile teams can sort AI use cases into a 2×2 matrix modeled on Steve Jobs’ 1997 product grid. The columns ask who must contribute, authorize, or respond: one Scrum Team, the Product Owner or Product Manager, or several teams and functions. The rows ask what authority the AI has: returning material (chatbot) or carrying out a bounded task (agent). A Retrospective summary is Team Chatbot; a daily customer-feedback digest compared with the roadmap is Organization Agent. When Should a Team Delegate a Task to an AI Agent? A product team should delegate a task to an AI agent only when it can name four things: the standard for good output, the owner who answers for the result, the permissions defining what the agent may read, change, and send, and the review burden. If any of the four is missing, a person stays in charge of the work, or AI stays out of it. Should Every Box of an AI Use Case Matrix Be Filled? Not necessarily: an empty box can reflect a sound decision about capacity and authority. Apple’s consumer portable box stayed empty until the iBook arrived in July 1999, and everyone could see what was missing. In an Agile organization, the Organization Agent box should stay empty when nobody can own the sources, define what counts as a gap, check unsupported claims, and decide when to pause the agent.

By Stefan Wolpers CORE

An AI inference service makes a model available to an application through an API. When you ask an inference API a question, the application sends your prompt to the service. The service processes it and generates a response, often streaming the answer back in batches. Producing a correct answer is only part of the job. The service also needs to respond quickly, handle many requests, and use GPU compute resources efficiently. Measuring each incoming request helps services understand what users experience and investigate problems when performance bottlenecks occur. AI Inference Metrics Models process input in units called tokens. Hence, common metrics include time to first token (TTFT), time between tokens (TBT), and time per output token (TPOT). Other measurements could be end-to-end latency, input and output token counts, request rate, error rate, etc. Also, TTFT and TBT describe different analyses. A response might start almost immediately but stream slowly, or take several seconds to start and then finish. Measuring only total response time hides that distinction. Averages can also hide problems. Services often examine P95 latency. This helps reveal slower experiences that an average can mask. To have a deeper understanding of such service behavior, a service can emit a simplified log record like the one below when a request finishes. One record explains one request. Millions of such records may be used to analyze the service and answer questions such as “Why did service-side latency increase during a particular time window?” “Did service errors increase with incoming traffic?” etc. This article explores using Parquet, Vortex, and DuckDB for such analytical queries. JSON { "event_time": "2026-09-01T12:30:00", "model": "llama", "input_tokens": 100, "output_tokens": 200, "ttft_ms": 100 } Columnar Formats Most telemetry queries need only a handful of columns; for example, a high-latency investigation may need only one hour of records. Columnar storage lets query engines access selected columns, and data layouts can help them skip irrelevant data. One such columnar format, Parquet, organizes data into row groups, column chunks, and pages. Another format, Vortex, takes a compressed array approach that supports some operations without fully decoding the data. Neither design guarantees faster queries; the outcome depends on the data, query implementation, and work being performed. This article uses DuckDB as the query engine because it can run SQL queries against both Parquet directly and Vortex through its extensions without setting up a database server. Scenario: Investigating High Latency When an AI service feels slow, a common starting point is to examine requests from the affected time window. For example, “How many requests were handled?” “What was the P95 TTFT?” etc. This experiment explores how file format and row ordering affect the time it takes to query one hour of inference latency data. For this purpose, one million inference records are synthetically generated using the structure shown earlier. Each record contains an event timestamp, model name, input and output token counts, and time to first token (TTFT) in milliseconds. All timestamps are interpreted as UTC. A given record is written in two formats, i.e., Parquet with Zstandard compression and Vortex with default writer settings. For each format, time-sorted and shuffled versions are created. This produces the four files below. Plain Text sorted.parquet - Time sorted records, Parquet with Zstandard compression. shuffled.parquet - Shuffled records, Parquet with Zstandard compression. sorted.vortex - Time sorted records, Vortex with default writer settings. shuffled.vortex - Shuffled records, Vortex with default writer settings. The following query calculates request count, average TTFT, and P95 TTFT for a populated one-hour window. The query is then repeated against the shuffled files. Request counts and latency statistics are checked across all four configurations before timing results are compared. SQL SELECT COUNT(*) AS request_count, AVG(ttft_ms) AS avg_ttft_ms, QUANTILE_CONT(ttft_ms, 0.95) AS p95_ttft_ms FROM read_parquet('data/sorted.parquet') WHERE event_time >= TIMESTAMP '2026-09-01 12:00:00' AND event_time < TIMESTAMP '2026-09-01 13:00:00'; For Vortex, only the file reader and path are changed: SQL FROM read_vortex('data/sorted.vortex') Step 1: Generate the Data Files Python from pathlib import Path import duckdb # Refuse to overwrite an existing dataset. Path("data").mkdir(exist_ok=False) con = duckdb.connect() con.execute("SET threads = 1") con.execute("SET preserve_insertion_order = true") con.execute("INSTALL vortex") con.execute("LOAD vortex") print("Writer DuckDB:", duckdb.__version__) con.execute(""" CREATE TABLE logs AS SELECT -- One request per second; timestamps are interpreted as UTC. TIMESTAMP '2026-09-01' + i * INTERVAL '1 second' AS event_time, -- Synthetic model mix: 70%, 20%, and 10%. CASE WHEN i % 10 < 7 THEN 'llama' WHEN i % 10 < 9 THEN 'gemma' ELSE 'qwen' END AS model, -- Different hash constants give each field a distinct sequence. -- Input tokens: 100–8099; output tokens: 100–1099. CAST(100 + hash(i, 11) % 8000 AS INTEGER) AS input_tokens, CAST(100 + hash(i, 22) % 1000 AS INTEGER) AS output_tokens, -- Baseline TTFT: 100–499 ms; add 800 ms during the slow hour. CAST(100 + hash(i, 33) % 400 AS INTEGER) + CASE WHEN event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' THEN 800 ELSE 0 END AS ttft_ms FROM range(1000000) AS t(i) """) formats = { "parquet": "FORMAT PARQUET, COMPRESSION ZSTD", "vortex": "FORMAT VORTEX", } layouts = { "sorted": "event_time", # Timestamp breaks any hash ties, giving a deterministic order. "shuffled": "hash(event_time), event_time", } for layout, order in layouts.items(): for fmt, options in formats.items(): path = f"data/{layout}.{fmt}" con.execute(f""" COPY ( SELECT * FROM logs ORDER BY {order} ) TO '{path}' ({options}) """) print("Created", path) con.close() Shell Writer DuckDB: 1.5.5 Created data/sorted.parquet Created data/sorted.vortex Created data/shuffled.parquet Created data/shuffled.vortex Step 2: Run the Experiment Each configuration receives one warmup, followed by five measured executions. Execution order is randomized across configurations, and DuckDB is configured to use one query worker. File generation, sorting, and extension loading are excluded from query timings. Median query latency is reported alongside the observed minimum and maximum. File size is recorded separately. Two different kinds of latency are involved: TTFT describes the synthetic AI requests, while query latency measures how long DuckDB takes to analyze those records. The experiment measures the latter, including filtering, aggregation, and fetching the small result. Python import csv import math import random import statistics import time from pathlib import Path import duckdb con = duckdb.connect() con.execute("SET threads = 1") con.execute("LOAD vortex") print("Reader DuckDB:", duckdb.__version__) print("Vortex:", con.execute(""" SELECT extension_version FROM duckdb_extensions() WHERE extension_name = 'vortex' """).fetchone()[0]) query = """ SELECT count(*), avg(ttft_ms), quantile_cont(ttft_ms, 0.95) FROM {source} WHERE event_time >= TIMESTAMP '2026-09-06 12:00:00' AND event_time < TIMESTAMP '2026-09-06 13:00:00' """ jobs = [ (layout, fmt) for layout in ("sorted", "shuffled") for fmt in ("parquet", "vortex") ] times = {job: [] for job in jobs} rng = random.Random(42) reference = None # Round 0 is warmup; rounds 1–5 are measured. for repeat in range(6): rng.shuffle(jobs) for layout, fmt in jobs: sql = query.format( source=f"read_{fmt}('data/{layout}.{fmt}')" ) start = time.perf_counter() result = con.execute(sql).fetchone() elapsed = (time.perf_counter() - start) * 1000 if reference is None: reference = result assert result[0] == 3600 assert all( math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-6) for a, b in zip(result[1:], reference[1:]) ), "Results differ across configurations" if repeat: times[(layout, fmt)].append(elapsed) rows = [] for (layout, fmt), values in sorted(times.items()): rows.append({ "format": fmt, "layout": layout, "size_mib": Path(f"data/{layout}.{fmt}").stat().st_size / 2**20, "median_ms": statistics.median(values), "min_ms": min(values), "max_ms": max(values), }) with open("results.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=list(rows[0])) writer.writeheader() writer.writerows(rows) print("\nRequests, average TTFT, p95 TTFT:", reference) print("\nFormat Layout MiB Median ms Min–max ms") for r in rows: print( f"{r['format']:9} {r['layout']:9} {r['size_mib']:7.2f}" f" {r['median_ms']:12.2f}" f" {r['min_ms']:.2f}–{r['max_ms']:.2f}" ) con.close() Shell Reader DuckDB: 1.5.5 Vortex: 9a8eb67 Requests, average TTFT, p95 TTFT: (3600, 1100.8336111111112, 1281.0) Format Layout MiB Median ms Min–max ms parquet shuffled 9.40 11.91 11.32–12.49 vortex shuffled 6.60 7.14 6.99–9.14 parquet sorted 6.79 2.75 2.30–2.86 vortex sorted 4.10 1.68 1.38–4.12 Takeaway: File Format vs. Row Ordering The selected hour contained 3600 requests, with an average TTFT of 1100.83 ms and a P95 TTFT of 1281 ms. These values describe the synthetic inference records, and the query times measure how long DuckDB took to calculate that summary. Time-sorted Vortex had the lowest median query time and smallest file size. However, ordering improved query performance substantially for both formats. Note that these findings apply to this synthetic dataset and hour window query after warmup. Timings varied across runs, and the results do not establish a universal format ranking or directly measure how much data was skipped. QuestionRESULTSTakeawayDid ordering matter more than format?Sorting improved median query time by 4.33x for Parquet and 4.25x for Vortex. Switching from Parquet to Vortex improved it by 1.64–1.67x within the same layout.Yes, for this workload. Time ordering had the larger effect.Did the smaller file produce the faster query?Shuffled Vortex was slightly smaller than sorted Parquet (6.60 vs. 6.79 MiB), but its median query time was about 2.6x longer.Not necessarily. File size alone did not predict query speed.Which configuration performed best?Sorted Vortex had the smallest file (4.10 MiB) and lowest median query time (1.68 ms). Sorted Vortex used the least storage and had the fastest median query time in this experiment. Conclusion AI inference services generate telemetry that helps explain token usage, identify latency changes, and investigate high latency issues. Efficient storage and querying can make that data less expensive to retain and faster to analyze. In this synthetic DuckDB experiment, time ordering improved median query performance. Sorted Vortex produced the smallest file and fastest median query, but sorted Parquet outperformed shuffled Vortex, highlighting the importance of organizing telemetry around its access patterns. These local warm-up-based results do not establish a universal format winner or demonstrate faster inference. Instead, they illustrate how format choice and data layout can support more efficient AI telemetry analysis, making it easier to investigate performance problems before deciding how to address them. References Build a DIY AI Model Hosting Platform With vLLMUnderstanding Inference Time ComputeFile Format, Parquet Vortex DocumentationDuckDB Documentation, VortexReading and Writing Parquet Files, DuckDB Docs

By Bhala Ranganathan CORE

This is Part 3 of a series on running sentiment analysis inside Neo4j. Part 1, Running Sentiment Analysis Inside Neo4j With a Java Plugin, used the VADER lexicon via a pure Java UDF – rule-based, fully local, high throughput.Part 2, Wasm Inside Neo4j: Building the Example That Didn't Exist, embedded a Wasm runtime inside a Java UDF using wasmtime-java — sandboxed, still local, lower throughput. Part 3 takes a different approach. Instead of bundling a scoring model inside the plugin JAR, we call TypeSafe AI's Jev from inside a Java UDF. Jev is a System One model – it doesn't generate text. It takes a state (the text to evaluate) and a set of typed questions and returns structured decisions with explicit probability and confidence information. No prompt engineering, no JSON schema coercion, no parsing. The result is callable directly from Cypher: Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; The full source code is available on GitHub. What Jev Gives You That VADER Doesn't VADER gives you a compound score. For "The movie had some good moments but ultimately disappointed," it returns something around -0.3. That's a number, but it doesn't tell you how much to trust it. Jev separates the decision from the uncertainty. For the same sentence, it returns: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score — which is the result of the separate confidence question we asked Jev, rated on a scale from 0 to 2 — is 1.55, landing between "Somewhat confident" and "Very confident." The separate confidence score provides an additional signal about the classification and that information can be actionable in a way that a raw sentiment score isn't. For unambiguous text, Jev is equally unambiguous: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, ... } This structured uncertainty output is the key differentiator of Part 3. If you need to act differently on low-confidence classifications — routing them to human review, flagging them for follow-up, or simply surfacing the uncertainty in a UI — VADER's scores don't provide a model-derived estimate of uncertainty about the classification. Jev returns an explicit confidence assessment alongside the decision. Confidence information should not be confused with accuracy: a model can be highly confident and still be wrong. What We're Building A Neo4j Java UDF that calls the Jev API over HTTPS, passing the input text as the evaluation state and asking two typed questions: a sentiment classification (positive, negative, or neutral) and a confidence score. For non-blank input, the UDF returns a Map<String, Object> with six keys: choice, confidence, score, positive, negative and neutral. For blank input, it returns only choice and confidence as a guard response without calling the API. No Wasm runtime, no bundled lexicon, no native libraries. The plugin has no Wasm runtime, native library, or bundled sentiment lexicon. Its only runtime dependency beyond Neo4j itself is Jackson, which is relocated into a private package namespace to avoid classpath conflicts with Neo4j's own Jackson dependencies. Prerequisites Java 21. OpenJDK 21 or later. Confirm with java -version.Maven 3.9.6. Install manually rather than via Homebrew on Apple Silicon. Confirm with mvn -version.Neo4j Desktop. Download and install Neo4j for Desktop. We're using version 2026.07.0. Update the neo4j.version property in pom.xml to match your installation.A Jev API key. Sign up for access at TypeSafe AI. The key is passed to Neo4j via a JVM property in neo4j.conf — more on this in the deployment section. Getting the Code The full source is on GitHub. Clone the repo and navigate to the neo4j-jev-udf directory: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set jev-udf mv jev-udf ../jev-udf cd ../jev-udf/neo4j-jev-udf The Maven Project The pom.xml has two things worth noting. First, org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime. Second, we use maven-shade-plugin to relocate Jackson into a private package namespace. Neo4j ships its own version of Jackson internally, and without relocation, the two can conflict at runtime in ways that produce silent failures. XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>jev-neo4j-poc</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.17.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <relocations> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>com.example.jev.shaded.jackson</shadedPattern> </relocation> </relocations> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Update the neo4j.version property to match your own Neo4j Desktop installation. The Java UDF The UDF uses Java's built-in HttpClient — no third-party HTTP library needed. The Jev API key is read first from the environment variable TYPESAFE_API_KEY, then from a JVM system property of the same name as a fallback. The fallback matters because Neo4j Desktop runs as a separate process and doesn't inherit your shell's environment variables; we'll set the JVM property in neo4j.conf in the deployment section. Java package com.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.databind.node.ObjectNode; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.util.HashMap; import java.util.Map; public class JevUDF { private static final String API_URL = "https://api.typesafe.ai/v1/systemone"; private static final ObjectMapper mapper = new ObjectMapper(); private static final HttpClient client = HttpClient.newHttpClient(); private static String getApiKey() { String key = System.getenv("TYPESAFE_API_KEY"); if (key == null || key.isBlank()) { key = System.getProperty("TYPESAFE_API_KEY"); } if (key == null || key.isBlank()) { throw new RuntimeException("TYPESAFE_API_KEY not set as environment variable or JVM property"); } return key; } @UserFunction("com.example.jev.sentiment") @Description("Classifies text sentiment using Jev (TypeSafe AI). Returns choice, probabilities and confidence.") public Map<String, Object> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { Map<String, Object> empty = new HashMap<>(); empty.put("choice", "neutral"); empty.put("confidence", 0.0); return empty; } String apiKey = getApiKey(); ObjectNode body = mapper.createObjectNode(); body.put("model", "jev-latest"); body.put("state", text); ObjectNode questions = mapper.createObjectNode(); ObjectNode sentiment = mapper.createObjectNode(); sentiment.put("type", "choice"); sentiment.put("instructions", "What is the overall sentiment of this text?"); ObjectNode criteria = mapper.createObjectNode(); criteria.put("positive", "Text expresses positive sentiment"); criteria.put("negative", "Text expresses negative sentiment"); criteria.put("neutral", "Text is neutral or factual"); sentiment.set("criteria", criteria); questions.set("sentiment", sentiment); ObjectNode confidence = mapper.createObjectNode(); confidence.put("type", "score"); confidence.put("instructions", "How confident are you in the sentiment classification?"); ArrayNode levels = mapper.createArrayNode(); levels.add("Not confident"); levels.add("Somewhat confident"); levels.add("Very confident"); confidence.set("criteria", levels); questions.set("confidence", confidence); body.set("questions", questions); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Content-Type", "application/json") .header("Authorization", "Bearer " + apiKey) .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(body))) .build(); HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new RuntimeException("Jev API error " + response.statusCode() + ": " + response.body()); } JsonNode result = mapper.readTree(response.body()); JsonNode answers = result.get("answers"); JsonNode sentimentAnswer = answers.get("sentiment"); JsonNode confidenceAnswer = answers.get("confidence"); Map<String, Object> output = new HashMap<>(); output.put("choice", sentimentAnswer.get("choice").asText()); output.put("confidence", sentimentAnswer.get("confidence").asDouble()); output.put("score", confidenceAnswer.get("score").asDouble()); output.put("positive", sentimentAnswer.get("probabilities").get("positive").asDouble()); output.put("negative", sentimentAnswer.get("probabilities").get("negative").asDouble()); output.put("neutral", sentimentAnswer.get("probabilities").get("neutral").asDouble()); return output; } } A few things worth noting in the Jev request body. The choice question uses a criteria object — a map of option names to descriptions. The score question uses a criteria array — an ordered list of level descriptions. These are different shapes, and mixing them up produces a 422 error from the API. Both questions are answered in a single API call; Jev evaluates them in parallel against the same state. Deploying to Neo4j Build the JAR: Shell mvn -q clean package Stop Neo4j in Desktop, then copy the JAR to your plugins folder: Shell cp target/jev-neo4j-poc-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ On Linux, the plugins folder is typically under $NEO4J_HOME/plugins/. On Windows, the plugins folder is typically under %NEO4J_HOME%\plugins\. Edit neo4j.conf in the conf/ folder of the same dbms directory and add three lines: Plain Text dbms.security.procedures.unrestricted=com.example.jev.* dbms.security.procedures.allowlist=com.example.jev.* server.jvm.additional=-DTYPESAFE_API_KEY=<your-api-key> Two lines are shown in the security configuration. allowlist controls which extension functions are loaded and makes the function callable from Cypher. unrestricted gives matching extensions full database access, bypassing Neo4j's security restrictions and should be used only where necessary. For this example, both settings are configured for the com.example.jev.* namespace; use the narrowest configuration that your Neo4j version and deployment require. The server.jvm.additional line passes the API key to Neo4j's JVM as a system property. For this local Neo4j Desktop example, that's a practical way to make the key available without modifying the UDF code. The Neo4j Desktop process does not automatically receive environment variables exported in your terminal session, so the JVM property is the reliable alternative. Do not commit this value to source control. For production, use your deployment environment's secret-management mechanism and inject the value into the Neo4j process without storing the secret in a tracked configuration file. Restart Neo4j in Desktop, then select Query from the left-hand navigation pane, connect to your instance, and confirm the function is registered: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'com.example' RETURN name; Expected: "com.example.jev.sentiment" Results Positive Sentence Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; Expected: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, "positive": 1.0, "negative": 0.0, "neutral": 0.0 } Negative Sentence Cypher RETURN com.example.jev.sentiment('The movie was absolutely terrible.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 2.0, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } Mixed Sentiment Cypher RETURN com.example.jev.sentiment('The movie had some good moments but ultimately disappointed.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score drops to 1.55 out of 2 – somewhere between "Somewhat confident" and "Very confident." This is structured uncertainty output, and it's what distinguishes Jev from a rule-based scorer. There are two different quantities here. confidence belongs to the sentiment choice itself: it describes the probability concentration of the classification. score comes from a separate question that explicitly asks Jev how confident it is in that classification. They answer different questions and should not be interpreted as two versions of the same confidence value. Blank-Input Guard Cypher RETURN com.example.jev.sentiment('') AS result; Expected: JSON { "choice": "neutral", "confidence": 0.0 } The blank-input guard returns immediately without making an API call. Here neutral is a guard value rather than a Jev classification: no sentiment analysis is performed for blank or whitespace-only input. Scalability and Limitations Every call to com.example.jev.sentiment() makes an outbound HTTPS request to the Jev API. That shapes where this approach fits and where it doesn't. Latency. Jev responds in 70-500ms per call. For an occasional interactive request — a customer submitting a review, a user looking up a specific node — that latency may be acceptable. For a batch pipeline, if 1,000 calls are made sequentially at a 200ms average, that's over three minutes of API time; concurrent execution changes the wall-clock time but introduces its own rate-limit and resource considerations. Cost. At the pricing available when this article was written, input tokens cost approximately $0.042 per million and output tokens were free. The test requests in this article averaged approximately 381 input tokens per call, giving a per-call cost of around $0.000016 (381 ÷ 1,000,000 × $0.042). Pricing is subject to change; check TypeSafe AI's current pricing before estimating production cost. At this workload size, cost is unlikely to be the primary constraint; latency, throughput, network dependency, and data residency are more significant considerations. At that rate, 1,000 calls would cost roughly $0.016, while one million calls would cost roughly $16, assuming the same average input size. Network dependency. Parts 1 and 2 of this article series are fully local – no network, no external dependency, no data leaving the database. Part 3 introduces all three. The UDF itself runs inside Neo4j, but the inference does not: the text crosses the network boundary to TypeSafe AI, and only the structured result returns to Neo4j. If the Jev API is unreachable, the UDF throws. In a production deployment, that's an operational risk worth planning for: the HttpClient in this example has no explicit timeout configured; in production, set a connect timeout and a per-request timeout to prevent a slow or unresponsive API from blocking Neo4j worker threads indefinitely. A fallback return value and monitoring on the call path are also worth considering. Note also that the UDF makes a synchronous blocking HTTP call inside database query execution; concurrent Cypher invocations increase pressure on Neo4j worker threads, the API's rate limits, and the network. Data residency. Text passed to com.example.jev.sentiment() leaves Neo4j and is sent to TypeSafe AI's API. If your data residency requirements prohibit that, this approach isn't suitable. Review TypeSafe AI's data handling policies before deploying in a regulated environment. Caching as a mitigation. For workloads where the same text is scored repeatedly, results can be stored as node properties on first call and read directly on subsequent queries — skipping the API entirely. That's a natural pattern in Neo4j and largely eliminates the latency and cost concerns for stable datasets. The right use case. This approach is best suited to low-frequency, interactive workloads where explicit uncertainty information matters more than throughput and where data leaving the database is acceptable. For high-volume batch ingestion, Part 1 (the Java VADER UDF) remains a strong fit — local, fast, no network dependency. Part 3 trades local throughput and offline operation for a richer decision output and explicit uncertainty information. Comparing the Three Approaches Part 1: Java VADERPart 2: Wasm/VADERPart 3: JevOutputPolarity scores (float)Polarity scores (float)Typed decision + explicit uncertainty informationLocationFully localFully localNetwork (TypeSafe AI API)Latency< 1ms50-200ms (init)70-500msCostFreeFree~$0.000016/callOfflineYesYesNoExecution boundaryJava UDFWasm runtimeExternal APIUncertainty outputNoNoYesTypical fitHigh-volume batchSandboxed local inferenceInteractive, uncertainty-aware decisions Summary We added a third option to the series: calling TypeSafe AI's Jev from inside a Neo4j Java UDF, returning a typed sentiment decision with explicit uncertainty information directly from Cypher. The implementation has the smallest inference footprint of the three – no Wasm runtime, no bundled lexicon, no native libraries. The JAR contains only the UDF class, a relocated Jackson, and Java's built-in HttpClient. The main configuration consideration is passing the API key to Neo4j's JVM via server.jvm.additional in neo4j.conf, since the Neo4j Desktop process does not automatically receive environment variables exported in your terminal session. The key contribution of Part 3 isn't the mechanics — it's the explicit uncertainty output. For clear-cut sentences, Jev is unambiguous. For genuinely mixed sentiment, it returns both the decision and a structured confidence measure that Cypher can act on directly. That's something neither VADER nor the Wasm implementation provides. A deeper discussion of putting Jev inside Neo4j can be found in the free online book Lessons Learned Putting Jev Inside Neo4j. The full source code is available on GitHub.

By Akmal Chaudhri CORE

As AI-generated code expands across production systems, provenance becomes essential for tracing where each change originated, which model or prompt influenced it, and what evidence supports its review, ownership, and trustworthiness. AI-assisted development changes the economics of software creation. Source code can now be produced faster than many engineering organizations can review, understand, test, secure, and maintain it. The resulting risk is not simply "bad AI code." Recent research is more nuanced, as one 2026 study of real repositories found AI-generated code to be more verbose and to exhibit different duplication patterns, while another maintenance study found lower subsequent maintenance activity for sampled AI-generated files than for comparable human-generated files. A 2025 controlled study likewise found fewer defects in some LLM-generated solutions but also structural problems on harder tasks. The operational conclusion is therefore not that generated code is inherently inferior, but that code volume can grow faster than governance capacity. The language and stack are unspecified, so the implementation patterns below use Git, Java, Python, TypeScript, and GitHub-style CI as representative examples. The central strategy is to treat code generation as an input to an engineering control system so we can record provenance, establish ownership, prevent duplicate abstractions, enforce complexity budgets, map tests to changed behavior, automate small refactorings, constrain dependencies, and make security and repository policy executable. Code Abundance Requires Traceability and Ownership Traditional repositories mostly answer who committed a change and when. AI-assisted repositories also need to answer how a change was produced, which tool or agent contributed, which task authorized the change, and which human or team accepted responsibility. Git trailers are a practical mechanism because Git supports structured key-value metadata at the end of commit messages and can parse that metadata with git interpret-trailers. Shell git commit -m "Add invoice reconciliation" \ --trailer "AI-Origin: assisted" \ --trailer "AI-Tool: coding-agent" \ --trailer "Task: PAY-431" This metadata should describe provenance rather than transfer accountability to a model. Ownership remains organizational. A CODEOWNERS file can map sensitive or high-change paths to responsible teams, and GitHub rules can require approval from those owners before merge. GitHub rulesets can also require status checks, dismiss stale approvals after new pushes, and require approval of the latest reviewable push. Plain Text /payments/ @acme/payments /auth/ @acme/security /platform/ @acme/platform The important repository metric is therefore not generated lines per week. It is generated surface area without durable ownership. A repository can absorb high code generation throughput when every merged path has an accountable maintainer, searchable provenance, and enforceable review boundaries. Quality Budgets Must Stop Net-New Entropy Code abundance makes absolute cleanup programs impractical. Quality gates should instead prevent each change from making the repository worse. Duplication is a useful example. PMD’s Copy/Paste Detector supports multiple languages, can fail CI when duplicates are detected, and can normalize identifiers for structurally similar Java or C++ fragments. Its documentation explicitly recommends removing duplication rather than attempting to keep copied blocks synchronized. Shell pmd cpd --minimum-tokens 100 \ --dir src/main/java \ --ignore-identifiers This check becomes more useful when applied differentially, as existing duplication can be baselined, while newly introduced clones fail the pull request. The same principle applies to complexity. ESLint exposes a cyclomatic-complexity rule with a configurable maximum, allowing TypeScript or JavaScript repositories to turn complexity from a review opinion into a merge condition. JavaScript export default [{ rules: { complexity: ["error", { max: 12 }] } }]; A complexity budget should be paired with automated refactoring rather than only rejection. OpenRewrite operates on semantic trees and provides recipes for framework migrations, security fixes, and consistent source transformations while preserving formatting. That model is particularly suitable for code-abundant repositories because repetitive cleanup can be expressed once and applied consistently. Recent industrial research from Bloomberg also suggests that automated maintenance is easier to review when it arrives as very small pull requests, as its experimental system targeted roughly ten line changes and merged most generated patches in a one-month deployment, although the sample was small and should not be generalized broadly. Tests and CI Need Change-Level Intelligence A large test suite does not automatically create confidence. Maintenance becomes expensive when a changed function cannot be connected to the tests that exercise it. Coverage.py supports dynamic contexts that record which test function executed each covered line, making a code-to-test map available for Python repositories. Plain Text [run] dynamic_context = test_function That mapping enables risk-based CI. Changed files can select directly related tests first, while shared libraries, authentication code, persistence layers, or other high-blast-radius modules can trigger broader suites. Coverage remains a gate rather than a vanity percentage. For Java, JaCoCo can halt a build when configured line, branch, instruction, class, or complexity thresholds are not met. A compact CI stage can combine these controls without treating AI-originated code as a separate species: YAML quality: steps: - run: npm run lint - run: pmd cpd --minimum-tokens 100 --dir src/main/java - run: pytest --cov=src --cov-context=test - run: ./mvnw verify The stronger policy is "no unexplained quality regression," not "all generated code receives maximal testing." Provenance can increase scrutiny for high-risk generated changes, but changed behavior, dependency reach, security sensitivity, and ownership should determine the actual gate. Dependency and Security Hygiene Must Scale With Generation Code-generation workflows capable of editing manifests can expand a repository’s dependency surface as quickly as application code. Dependency review therefore belongs at pull request time. GitHub’s dependency review reports direct and indirect changes from manifests and lockfiles, includes vulnerability information, and can fail a required check when vulnerable packages are introduced. This supports a repository policy in which new dependencies require an explicit purpose, lockfile changes receive review, and unused packages are removed during routine maintenance. Security analysis should operate on every merge path rather than rely on prompt instructions. GitHub code scanning can run CodeQL and other analyzers on pull requests and the default branch, and CodeQL data-flow analysis can identify unsafe flows such as dangerous arguments or sensitive data leakage. Required code-scanning checks can block merging until findings are resolved. Runtime-facing checks provide a complementary boundary. OWASP ZAP’s baseline scan spiders a deployed target, performs passive scanning, supports configurable fail conditions, and is explicitly designed for CI/CD environments. Static analysis can therefore guard source changes while dynamic scanning checks observable application behavior. The important distinction is between generation-time guidance and repository-time enforcement, as model instructions are advisory, while CI gates are reproducible controls. Repository governance ties the system together. Rulesets should require code-owner approval for protected areas, passing quality and security checks, review of dependency changes, and fresh approval after material updates. Such controls keep throughput high without allowing generated volume to bypass the same engineering standards applied to manually written code. Conclusion The maintenance problem created by AI is fundamentally a throughput mismatch. Code production can accelerate immediately, while ownership, review capacity, testing, security analysis, and architectural coherence remain finite. Current research does not justify treating AI-generated code as uniformly worse than human-written code, but it does show meaningful differences in verbosity, structure, duplication, defects, and maintenance behavior that make passive governance increasingly risky. Sustainable code abundance therefore depends on converting maintenance expectations into repository mechanics where provenance is attached to commits, ownership is attached to paths, quality budgets are enforced on new changes, tests are mapped to behavior, refactoring is delivered in small reviewable patches, dependencies are examined before merge, and security checks are made mandatory. The durable advantage will not come from generating the most code. It will come from operating a codebase that can absorb generated code without allowing maintainability to collapse under its volume.

By Uthej Mopathi CORE

Building a RAG application is relatively easy. Scaling it for millions of users and millions of documents is a different engineering problem. At production scale, the architecture matters. Ingestion, search, indexing, permissions, latency, and updates all need to work independently without slowing down user requests. The first design decision is to separate document ingestion from user queries. Q1. Why Should Ingestion and Querying Be Separate? They have different requirements. Ingestion runs in the background. It processes documents, cleans them, removes duplicates, creates chunks, generates embeddings, adds metadata, and updates indexes. Querying runs for every user request. It needs to return relevant results quickly and consistently. Think of it like a library. Ingestion is the librarian organizing new books. Querying is a visitor looking for a book. The visitor should not have to wait while the librarian processes a new shipment. This separation allows the ingestion pipeline to optimize for throughput, while the query pipeline can optimize for low latency. Q2. Why Does Chunking Matter, and Is "Semantic Chunking" Always Right? A chunk is a small slice of a document that gets stored and searched. Where you cut that slice matters a lot. Cut it at a random character count, like every 500 characters, and you might slice a sentence in half, or separate a question from its answer. When that happens, the AI model only sees half the picture and gives a worse answer. Semantic chunking cuts the text at natural breaks, like the end of a paragraph or a change in topic, instead of a fixed length. This keeps ideas whole. One honest caveat, since accuracy matters here: semantic chunking is not free. Because chunks come in different sizes, it is harder to batch-process them efficiently, and search scoring can get uneven when some chunks are tiny and others are huge. Many real systems use a middle ground, a target size that still respects natural sentence and paragraph breaks, instead of pure semantic chunking. Every chunk also needs extra information: which customer it belongs to, who can see it, when it was created, and where it came from. Skip this, and you cannot control access, cannot handle updates properly, and cannot debug problems later. Q3. Why Use "Approximate" Search Instead of Exact Search? When the AI looks for the most relevant chunks, it is comparing the meaning of your question to the meaning of millions of stored chunks. Doing this comparison perfectly, checking every single one, is called exact search. It gives the truest answer, but it is painfully slow once you have millions of chunks. Approximate Nearest Neighbor search, called ANN, is a shortcut. It finds chunks that are very likely the best matches, without checking absolutely everything. It is dramatically faster and uses much less memory. The honest tradeoff: ANN occasionally misses the single best match and returns the second or third best instead. In practice, this loss is small, and the speed gain is large, but it is a real tradeoff, not a magic trick with no downside. Q4. What Does "Hybrid Retrieval" Mean and Why Bother? There are two ways to search: by keyword and by meaning. Keyword search, using a method called BM25, is great at finding exact terms: product codes, error messages, specific names. It struggles when your question uses different words than the document, even if they mean the same thing. Meaning-based search, using vector embeddings, understands intent. It can match "how do I get my money back" to a document titled "refund policy" even though no words overlap. It struggles with exact codes and rare technical terms. Hybrid retrieval runs both searches, combines the results, and then uses a stronger, more careful model to re-rank just the top candidates from that combined list. You get the best of both without paying the cost of running the expensive re-ranker on everything. The standard way to combine two different ranked lists like this is called Reciprocal Rank Fusion. It blends the rankings from both searches fairly, without needing the two scoring systems to be on the same scale, which they usually are not. Q5. The Chatbot Gives a Confident, Wrong Answer. How Do You Actually Debug That? Just calling it "a hallucination" and shrugging does not fix anything. You need to find out exactly where the process broke down. There are only two possible places: retrieval or generation. Step 1: Check retrieval first. Look at what chunks were actually pulled for that question. Is the correct information even in there? If not, the problem happened before the model ever saw the question. Something upstream failed: maybe the chunking cut the answer in half, maybe the embedding did not capture the right meaning, maybe the search weighting favored the wrong chunks. Step 2: Check generation. If the right information was retrieved but the model still got it wrong, the problem is how the model used what it was given. Common causes: the prompt did not clearly tell the model to stick to the provided context, the important information was buried in the middle of a long prompt (models pay more attention to the start and end), or a distracting, irrelevant chunk got pulled in alongside the correct one and confused the model. Build this check into an ongoing habit, not a one-time fix. Keep a running set of real user questions and grade the system against them regularly, so you catch new problems the moment they appear instead of after users complain. Q6. Why Does an AI Agent Call the Wrong Tool, and How Do You Fix It? Usually this is not the model being "dumb." It is the tools being poorly described. If two tools sound similar or overlap in what they do, the model genuinely cannot tell them apart, no matter how capable it is. Fixes that actually work: Give every tool exactly one clear job, with no overlap with other tools.Use strict rules for inputs: required fields, fixed lists of allowed values where possible, and check the input before running anything.Show the model a few examples of correct tool choices, right in the instructions.For anything risky, like sending money or deleting data, add a confirmation step before it actually happens. The tradeoff is real. More rules and more checks mean more time per request, more cost, and more code to maintain. It is worth it for anything with real consequences if it goes wrong. Q7. Why Can AI Training Run in Parallel, But Generating a Response Cannot? A language model builds its answer one word (technically one "token") at a time, and each new word depends on every word it already wrote. To say "New Delhi," it must generate "New" first, because it needs that word in front of it before it can decide what comes next. This forces the model to work in a strict, one step at a time sequence when generating a response. Training is different because the model already knows the correct full sentence in advance; it is learning from real examples. So during training, it can check its predictions for every position in the sentence all at once, in one parallel pass, using a trick called teacher forcing. That is why training scales beautifully across many GPUs while generating a single response does not. One real technique worth knowing: speculative decoding. A small, fast model guesses several words ahead, and the big model checks and confirms them in one pass instead of one word at a time. It does not remove the one-word-at-a-time nature of the model, but it does meaningfully speed up how fast text appears in practice. Q8. What Problem Does an "LLM Gateway" Solve? Once more than one team in a company is calling AI models directly, things get messy fast. No shared view of who is spending what. No consistent security checks. Every team is reinventing the same plumbing. An LLM Gateway is a single front door that every team's requests go through. It handles login and per-team usage limits, picks which model or provider to send a request to, and automatically switches if one goes down, logs everything for cost tracking, and checks requests against safety and policy rules before they reach the model. The biggest win: if the company wants to switch AI providers or upgrade to a newer model, they change it once, behind the gateway. No product team has to touch their own code. Q9. Why Not Just Use One Big, Powerful Model for Everything? Cost and speed. Big models are expensive per word generated and slower to respond. A lot of the work inside a production system does not need that much intelligence: figuring out what a user is asking for, pulling out a name or date, deciding which of a few tools to call. Give that simple work to a small, fast, cheap model. Save the big, expensive model for the parts that genuinely need deep reasoning or high-quality writing, usually the final answer the user actually reads. This only works well if you are checking quality regularly. If the small model starts quietly making mistakes on its part of the job, you will not notice unless you are testing for it. Q10. Latency Is Not Just One Number. What Should You Actually Measure? Two separate things, because users experience them differently and they are caused by different parts of the system. Time to First Token (TTFT). How long the user stares at a blank screen before anything shows up. Mostly caused by how long your prompt is and how much retrieval work has to happen before generation even starts. Tokens Per Second (TPS). Once text starts appearing, how fast it keeps coming. Caused by the model's size and how busy the GPUs currently are. TTFT matters more for how the experience feels. If words start appearing right away, the whole thing feels fast, even if the total time to finish is exactly the same as a version that made you wait first and then streamed quickly. This matches general research on perceived speed in interfaces: how fast something feels and how fast it actually is are not the same thing, and feel usually wins for user satisfaction. Q11. What Changes When You Scale From Ten Million Documents to One Hundred Million? At that size, one machine cannot hold the entire search index anymore, period. This stops being a "make the search better" problem and becomes a distributed systems problem. Every question now has to be sent to several machines at once, called shards, and the results merged back into one final ranked list. Two more things stop being optional. First, incremental indexing: when a document changes, you update just that piece of the index; you do not rebuild everything from scratch every time; that would be far too slow. Second, caching: repeated or popular questions get served from a fast cache instead of running the full search and generation process every single time. Q12. How Should an AI Agent's "Memory" Actually Be Organized? Four separate layers, each solving a different problem. Mixing them together makes the system confusing to build and hard to debug. Short-term memory. The working notes for the task happening right now. Lives inside the model's current context window and disappears when the task ends.Long-term memory. Facts that should stick around, like a user's preferences. Stored outside the model and pulled back in only when relevant.Shared memory. When several agents are working together, this is the common notepad they all read from and write to.Episodic memory. A running, timestamped log of what happened before, so the system can look back and learn from past interactions. This is a useful, commonly used way to think about agent memory, though it is a design pattern, not a fixed rulebook everyone follows the same way. The real cost is honest: better memory makes the agent smarter, but it also means more moving parts to keep consistent and in sync. Q13. What Does Security Look Like Specifically for a RAG System? The core rule to remember: treat every retrieved document like it is untrusted input from a stranger, the same caution you would use for a raw form submission on a website. Never let content pulled from your document store trigger an action on its own without a server-side check first. Why this matters: a bad or compromised document sitting in your corpus could contain hidden instructions aimed at tricking the model, a real attack pattern known as prompt injection through retrieved content. For systems that serve multiple customers, called multi-tenant systems, isolation needs to be enforced at three separate levels, not just checked once in the application code: Data isolation. One customer's documents and embeddings must never be retrievable by another customer. This needs to be enforced at the index itself, not just trusted to application logic that could have a bug.Compute isolation. One very active customer should never be able to hog all the processing power and slow everyone else down.Cache isolation. A cached answer generated for one customer must never accidentally be served to a different customer. These three checks are especially important in regulated industries like banking and finance, where keeping customer data separated is often a legal requirement, not just good practice. Q14. Where Is the Real Value Shifting for Engineers Working in AI? Building a basic AI demo is turning into a baseline skill, something most engineers can put together in an afternoon with existing tools. What actually sets someone apart now is knowing how to run this in production: handling real scale, enforcing real security and tenant separation, and keeping agentic workflows reliable when thousands of real users are hitting the system at once. Closing The question engineers get asked is changing. It used to be "can you build this?" Now it is closer to "can you make this work correctly, safely, and affordably for millions of people, every single day, without it falling over."

By Ram Ghadiyaram CORE

An AI agent can understand a request, develop a plan, and still fail for a mundane reason: It cannot reach the system where the information — or the ability to act — lives. Enterprise context is scattered across customer relationship management (CRM) platforms, ticketing systems, document repositories, analytics tools, and internal applications. Connecting them has traditionally required organizations or software vendors to build and maintain separate integrations for every system. That approach has left many companies with significant integration debt. Every point-to-point connector becomes another piece of code to secure, monitor, update, and repair. AI agents make that debt more visible because a single missing connection can interrupt an otherwise automated workflow. Model Context Protocol, or MCP, offers a much better architectural pattern. It is one of the most important developments in the emerging agent ecosystem. But standardized connectivity is only half of what enterprises need to operate agents successfully. What MCP Changes MCP gives AI applications a consistent way to connect to external data sources and tools. Instead of requiring a separate integration for every pairing of applications, an MCP server can advertise the capabilities it makes available. A compatible AI application can then discover and use those capabilities through a common interface. The USB-C analogy is useful. Before common connection standards, every device required its own proprietary cable. Standardization let many devices use the same interface. MCP applies that principle to AI applications and the enterprise systems they need to reach. That is why MCP is generating so much justified excitement. It reduces integration friction, expands what agents can do, and makes it easier to connect new systems as requirements evolve. Integration work does not disappear entirely, however. It moves. The provider of the underlying application — or an internal team supporting a proprietary system — must still expose useful capabilities through an MCP server. Their quality also depends on the underlying application and API. Putting an MCP wrapper around a poor search service does not produce better search. It simply gives the agent standardized access to the same limitations. But MCP only standardizes the connection. It does not provide the complete environment required to operate enterprise agents. Connectivity Does Not Replace Governance When users connect agents to enterprise applications through MCP, the agent may operate using their identity and existing permissions. A person connecting to Jira, for example, authenticates as themselves, and the agent subsequently calls Jira tools using that person’s delegated access. That creates an important distinction between what a user is technically permitted to do and what an agent acting on the user’s behalf should be allowed to do. A developer may have access to tools that can write or delete code. It does not follow that every agent should receive those capabilities. Enterprise governance therefore needs several layers. Administrators should establish which MCP servers users and agents may access and which tools from those servers are available. Within those boundaries, an agentic AI platform can automatically discover and select the tools relevant to a particular task. A GitHub MCP server might expose dozens of actions, while a code-review agent receives only read-only tools. A specialized service agent might receive only the Jira tools needed to review or create tickets, without access to unrelated systems or more destructive capabilities. These controls should be enforced through the operating environment rather than left to a prompt. Telling an agent not to use a destructive tool is guidance. Preventing that tool from being available is a guardrail. Standardized Access Creates a Tool Problem The easier it becomes to connect enterprise systems, the more tools an agent may theoretically have at its disposal. A single MCP server can expose dozens of actions. Connect an agentic AI platform to 10 or 20 enterprise applications and the available toolset can quickly grow into the hundreds. Giving every tool definition to an agent at once consumes context, increases token usage, and introduces irrelevant choices. If several connected systems expose functions called “get user,” “search,” or “update record,” the agent must determine which one applies. The answer is progressive disclosure: Provide tools only when they become relevant. A specialized agent should begin with a focused set of capabilities rather than unrestricted access to the enterprise. A Jira service agent does not need tools for HubSpot, GitHub, or Microsoft 365. Limiting the available toolset improves security, but it also helps the agent work more efficiently and reduces the likelihood that it selects the wrong tool. Agents also need instructions for using tools effectively. A tool definition may tell an agent that a search capability exists. It may not explain when to use keyword search rather than semantic or hybrid search, how to structure the query, or what to do if the results are incomplete. Skills provide that operating guidance. A skill functions like a handbook that explains how to complete a particular type of task and which tools should be used. Once the skill becomes relevant, the platform can disclose the associated capabilities. No organization would onboard an employee by providing access to every corporate system without explaining the job. Agents should not be treated differently. From Connectivity to Enterprise Readiness MCP is one of the most consequential advances in the emerging agent ecosystem. It gives organizations a standardized way to connect agents to the systems where enterprise work actually happens and can eliminate substantial repetitive integration development. But the connection is the starting point, not the complete architecture. To turn access into dependable business execution, enterprises still need an agentic AI platform that can govern tools, provide agents with the right instructions, manage context and memory, orchestrate complex work, and maintain visibility into what agents are doing. The opportunity is not to choose between MCP and an agentic AI platform. It is to combine them: MCP provides a universal way for agents to reach enterprise systems, while the platform provides the structure agents need to use those connections safely and effectively.