Tools

A pod stuck in Pending is one of the most misleading states in Kubernetes. Nothing looks wrong. There's no error banner, no red text screaming at you. The scheduler just quietly gave up finding a home for it and is waiting for the world to change. I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about. This is capacity spillover: demand outgrows what the current set of nodes can absorb, and the cluster needs somewhere for the overflow to land. Handle that well and it's a blip nobody notices. Handle it badly, and it's an outage that shows up in a customer complaint before it shows up in your dashboard. I'll be honest, a few years ago I would have called this an edge case worth a paragraph in the docs and moved on. Not anymore. Ask around, and you'll find most people running workloads in the cloud right now have a story about a specific instance type just not being there when they needed it, GPU-backed machines most visibly, but I've seen it happen with fairly ordinary compute SKUs too, usually right in the middle of a demand spike nobody scheduled around. That's really the whole case for spillover, once you strip away the architecture diagrams. Without it, what you get during a capacity crunch is close to random: whichever pod happens to hit the scheduler first grabs whatever's left, and everyone else waits and hopes. With it, the randomness mostly goes away. Workloads land where you decided they should, in the order you decided matters, instead of wherever the scheduler happened to find room first. New pod needs scheduling -> primary node pool checked -> if room is available, schedule normally; if the pool is full, spill to the secondary pool (overflow capacity). Why Scheduling Failures Happen in the First Place The scheduler's job sounds almost too simple: find a node with enough free CPU and memory for the pod's requests. In practice, a pod can fail to schedule for reasons that have nothing to do with the cluster actually running out of room. Here's the part that trips people up the first time they see it. A cluster can have plenty of free CPU in aggregate and still fail to place a memory-heavy pod, because no single node has enough memory available on its own. Resource requests get evaluated per node, not pooled across the whole cluster. I've had a coworker insist this was a bug the first time they saw it happen. It isn't. It's just how bin-packing works when nobody told you that's what you were looking at. Taints, tolerations, and node affinity narrow the field further. A pod that needs a GPU node, or needs to steer clear of a specific zone, only ever sees the slice of the cluster that matches. Spare capacity can be sitting right there, and the pod simply isn't allowed near it. And then there's the timing problem, which is the one that actually pages people. Even when the cluster genuinely needs more nodes and the autoscaler agrees, provisioning a node isn't instant. There's a real window between "we need capacity" and "capacity exists," and whatever's waiting in that window experiences it as a failure, not as a system doing exactly what it was designed to do. What Spillover Actually Looks Like "Spillover" gets used loosely, but it covers a few genuinely different situations, and I think that's where a lot of confusion comes from. People nod along in a meeting assuming everyone means the same thing, and they don't. The first one, and honestly the one most teams reach for first, is spilling to a different node pool entirely. You run a primary pool sized for steady-state load, and a secondary pool, usually spot or preemptible, that only spins up once the primary's full. Interruption-tolerant stuff gets pushed there when things get tight, which keeps the expensive, stable pool free for whatever actually can't afford to get evicted. It's an easy sell to whoever owns the cloud bill, which is probably why it's the version everyone sets up first. Then there's spilling across zones, or in the worst cases, regions. A single zone running out of a specific instance type during a demand spike isn't some rare event you plan a tabletop exercise around; it's just a Tuesday at this point. So you spread the workload wider. Fine in theory. I've watched a "let's just spread it across zones" fix get shipped on a Friday and turn into a cross-region latency ticket by the following Wednesday, because nobody had checked whether the workload actually tolerated that. And there's a third flavor that's less about running out of room and more about not wanting to spend the money. Technically, there's room on the expensive reserved instances. The better default is letting things spill onto cheap infrastructure first, and only reaching for the premium stuff when nothing cheaper is left over. Build it the other way, and you won't find out until the next billing cycle, which is a genuinely bad time to find out. Designing for It Instead of Reacting to It The cluster autoscaler, or something like Karpenter, handles the mechanics of adding nodes once pods can't be placed anywhere. That's useful, but it's also reactive by definition, closer to a fire alarm than a smoke detector. By the time it fires, everyone already knows something's wrong. A few things actually close the gap between "capacity ran out" and "someone got paged," in my experience. Pod priority and preemption are the ones people underestimate. Not every workload deserves an equal claim on scarce capacity, and treating them as if they do just means everything fails to schedule at once, in whatever order the scheduler happened to try them, which helps nobody. Give the stuff that actually matters a higher PriorityClass and let it evict lower-priority pods when things get tight. Over-provisioning on purpose sounds wasteful until you sit with it for a second. A handful of low-priority placeholder pods, doing nothing except holding a spot, get evicted the instant real demand shows up, and the real workload schedules immediately instead of waiting on a node to boot. Small constant cost, much faster response when it actually counts. Provisioning speed is the one that surprised me. Tools that can add nodes directly, without needing to fit some pre-defined pool shape first, shrink that window between "pod can't schedule" and "a node exists to put it on" a lot more than I expected going in. That window is basically where the pager lives. And testing failure on purpose, which is the step almost everyone skips. It's easy to assume spillover works because it's configured correctly on paper. Deliberately draining a node pool in a test environment and watching whether pods actually land where they're supposed to, in the order they're supposed to, finds a category of misconfiguration you will never catch by reading YAML, however carefully you read it. The Part That's Easy to Skip Past Spillover design tends to get treated as a purely technical exercise. Pick the autoscaler, define the pools, set the priorities, done. The harder question usually isn't technical at all: who decides which workloads are allowed to spill onto cheaper or less reliable capacity, and who's on the hook when a decision made for cost reasons turns into someone else's latency problem three sprints later. Getting the mechanics right is necessary. It's not enough on its own, because nobody's actually agreed on the policy those mechanics are supposed to enforce, and that conversation almost always happens after the first bad incident instead of before it. It's happened to every team I've worked with in some form. I'd rather have that conversation early, even if it's a little awkward and nobody wants to own the answer yet. It beats the alternative meeting, the one scheduled at 9 am the morning after something broke.

Azure Key Vault is the well-behaved citizen of the Azure secrets world. Every secret is versioned; you get soft-delete and purge protection; access is governed by RBAC or access policies; and DefaultAzureCredential means your app authenticates the same way locally (az login) and in production (Managed Identity/Workload Identity on AKS) with zero code changes. And yet the same old gap is there. Your Go service calls GetSecret once at startup, decodes the value into a struct, and treats it as immutable for the life of the process. When someone rotates the secret in Key Vault — a new version, the old one disabled — your service has no idea. It keeps using the value it read at boot until the next deploy or pod restart. The rotation was seamless in Azure and invisible to your app, which is exactly the wrong combination. mamori is a Go library I built to fix this class of problem across every backend. Point it at a Key Vault secret, and it watches for new versions, pulls the change, re-validates your whole config, and swaps it in atomically — while your service keeps running. This is the Azure entry in the mamori launch series. For the full model, start with Introducing mamori: Typed, Watchable Config & Secrets for Go; here we focus on Key Vault. The azure-kv Scheme The providers/azure module registers the azure-kv scheme. The grammar is azure-kv://<vault-name>/<secret-name>[#json-key][?version=<v>]: Plain Text azure-kv://my-vault/db-password # latest enabled version azure-kv://my-vault/api-key?version=abc123 # pin to a specific version id azure-kv://my-vault/creds#password # one key out of a JSON secret One Key Vault quirk to keep in mind: versions are identified by opaque hex IDs, not the sequential 1, 2, 3 you get in GCP Secret Manager. So ?version= here pins to an id like abc123def456..., not a number. Leave ?version off and mamori tracks whatever the latest enabled version resolves to — which is what you want for credentials that rotate. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/azure Register it with a blank import. It authenticates through DefaultAzureCredential, so it picks up your az login locally, a Managed Identity on a VM/App Service, or Workload Identity on AKS — automatically, in that order: Go import _ "github.com/xavidop/mamori/providers/azure" The identity you bind needs read access to the vault's secrets. With Key Vault's RBAC model, that's the built-in Key Vault Secrets User role, scoped to the vault (or to individual secrets): Shell az role assignment create \ --role "Key Vault Secrets User" \ --assignee "<managed-identity-object-id>" \ --scope "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/my-vault" If your vault still uses the classic access-policy model instead of RBAC, grant Get (and List) on secrets. Load, Then Watch the Rotation Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/azure" ) type Config struct { // Tracks the latest enabled version: rotations flow in automatically. DBPassword secret.String `source:"azure-kv://my-vault/db-password"` // Pinned to an immutable version id: deliberate, reproducible. SigningKey secret.String `source:"azure-kv://my-vault/signing-key?version=abc123"` // One field out of a JSON secret. APIToken secret.String `source:"azure-kv://my-vault/creds#api_token"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) // live, no restart log.Println("Key Vault: db-password rotated and applied") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // last VALID config, lock-free _ = cfg } Every secret field is a secret.String, so it redacts itself in fmt, slog, and JSON — a stray log.Printf("%+v", cfg) prints [REDACTED] — and only reveals plaintext through an explicit .Reveal(). mamori also ships a go vet analyzer that fails your build if an azure-kv:// value lands in a plain string instead of a secret.String. What Reconciliation Guarantees When a new version of db-password becomes the latest enabled one, mamori's poll cycle detects the version change, fetches it, and then: Re-validates the entire struct – a value that fails a validate rule is rejected atomically; Get() keeps returning the last good config and OnError receives a *ValidationError. A bad secret can't break your service.Swaps atomically – w.Get() moves from old to new in one lock-free store; no reader ever sees a half-updated config.Fires a diff-aware callback – ev.Fields tells you exactly which fields moved, so you only rebuild what changed. Key Vault has no push channel for "a new version was created", so mamori polls — politely: the interval is jittered by ±20% (tunable with WithJitter) so a fleet of pods doesn't hammer the vault on the same tick, and change detection compares the version id before pulling the payload, so an unchanged secret is cheap. If Key Vault is briefly unreachable, mamori retries with backoff and keeps serving the last good value; wire WithStale(10*time.Minute) and check for a *mamori.StaleError in OnError if you want to page only on a prolonged outage rather than a transient blip. Structured Secrets With flatten Storing a whole connection blob as one JSON secret? Decode it straight into a nested struct: Go type Config struct { DB DBConfig `source:"azure-kv://my-vault/db-conn" flatten:"json"` } type DBConfig struct { Host string `mapstructure:"host"` Port int `mapstructure:"port"` Password secret.String `mapstructure:"password"` } One Key Vault call, one JSON document, cleanly typed — with the password still redacting itself. When You Don't Need This If a secret only ever changes on deploy, a plain mamori.Load at boot is all you need — no watching, no poll loop. mamori earns its keep the moment a Key Vault secret rotates while your process is running and you'd rather rebuild a client than recycle a pod. Wrapping Up Azure Key Vault rotates your secrets cleanly; mamori makes your Go service actually notice. It watches Key Vault, reconciles a new version into your running process — typed, validated, redacted — and lets you pin the versions you want frozen. Shell go get github.com/xavidop/mamori/providers/azure Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early and moving fast, so issues and feedback are very welcome. Happy coding!

By Xavier Portilla Edo CORE

AWS Secrets Manager has a lovely feature: automatic rotation. You attach a rotation Lambda, set a schedule, and every 30 days your RDS password is rotated for you. Secure by default, no human involved. There's just one problem, and it's the one nobody mentions in the demo: your application has no idea it happened. Your Go service read SecretString once, at startup, and stashed the password in a struct. Thirty days later Secrets Manager quietly rotates it, the old value stops working, your connection pool starts throwing auth errors, and you find out via a pager alert at 3 am. The "fix" everyone reaches for is to redeploy on a schedule, or restart pods so the app re-reads the secret — which turns a security feature into an operational chore. mamori is a Go library I built to close exactly this gap. It watches AWS Secrets Manager (and SSM Parameter Store) and reconciles a rotation into your running process — typed, validated, redacted — so a rotation triggers a connection-pool rebuild instead of a restart. This is the AWS-focused companion to the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go. Read that first for the full model; here we go deep on AWS. The Two AWS Schemes The providers/aws module registers two ref schemes: RefReads fromaws-sm://<name>[#json-key]Secrets Manageraws-ps://<name>[#json-key]SSM Parameter Store A few real refs — note the triple slash on Parameter Store, because SSM parameter names are absolute paths that start with /: Plain Text aws-sm://prod/api-key # whole secret value aws-sm://prod/db#password # one key out of a JSON secret aws-ps:///myapp/log-level # SSM parameter (triple slash!) aws-ps:///myapp/db#password # one key out of a JSON SSM parameter The rule of thumb: secrets (passwords, tokens, keys) belong in Secrets Manager (aws-sm), and plain, non-sensitive config (log levels, worker counts, feature toggles) belongs in the cheaper Parameter Store (aws-ps). mamori lets you mix both in the same struct. Setup Shell go get github.com/xavidop/mamori go get github.com/xavidop/mamori/providers/aws Register it with a blank import for zero-config (it uses the standard AWS credential chain — environment, shared config, IRSA on EKS, EC2 instance role): Go import _ "github.com/xavidop/mamori/providers/aws" Or, when you need to pin a region or pass a custom client, construct the provider explicitly and hand it to Load/Watch with WithProvider — this takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) Your runtime's IAM role needs the obvious permissions, scoped tightly: JSON { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:prod/*" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:eu-west-1:123456789012:parameter/myapp/*" } ] } Load, Then Watch the Rotation Here's a service that pulls its database password from Secrets Manager and its operational knobs from Parameter Store: Go package main import ( "context" "log" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" _ "github.com/xavidop/mamori/providers/aws" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` APIKey secret.String `source:"aws-sm://prod/api-key"` LogLevel string `source:"aws-ps:///myapp/log-level" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"aws-ps:///myapp/workers" default:"4" validate:"gte=1,lte=256"` } func main() { ctx := context.Background() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(60*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { // A rotation just landed — rebuild the pool, no restart. pool.Rotate(ev.New.DBPassword.Reveal()) log.Println("database password rotated and applied live") } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() log.Printf("started with %d workers at %s", cfg.Workers, cfg.LogLevel) } DBPassword and APIKey are secret.String, so they redact themselves everywhere — log.Printf("%+v", cfg) prints [REDACTED], json.Marshal emits "[REDACTED]", and slog logs them as [REDACTED]. The plaintext only ever escapes through an explicit .Reveal(). mamori even ships a go vet analyzer that fails the build if you put an aws-sm:// value into a plain string instead of a secret.String. Polling Done Right (Jitter, Batching, Atomicity) Secrets Manager has no push notification for value changes, so mamori polls — but it polls the way you'd want a fleet to poll: Jittered intervals. The poll interval is spread by ±20% by default, so 200 pods don't all hit GetSecretValue on the same tick and blow through your API rate limit. You can tune the spread with WithJitter.Change detection is cheap. mamori compares the secret's version (Secrets Manager's VersionId) before doing anything, so an unchanged secret costs one lightweight check, not a full re-decode.Batching. The AWS provider resolves multiple refs from the same account in as few API calls as possible, automatically — so a struct with ten aws-sm:// fields doesn't mean ten times the API traffic.Atomic + validated. When a rotation is detected, mamori re-validates the entire struct and only swaps it in if it's valid. OnChange never fires with a half-updated or invalid config. Compare this to the AWS Secrets Manager caching client: it caches and refreshes one secret at a time, with no struct composition, no validation, no diff-aware callback, and no config sources beyond Secrets Manager. mamori gives you the whole config surface — secrets and Parameter Store and env and files — in one typed struct with one reconciliation loop. Tolerate a Blip, Page on a Real Outage Secrets Manager occasionally throttles or has a transient error. You don't want that to page anyone — a blip should just retry with backoff and keep serving the last good value. But if Secrets Manager is genuinely unreachable for ten minutes, that's worth waking someone up. WithStale draws that line: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { pager.Page("AWS config stale >10m", stale) return } metrics.Inc("aws_transient_error") // blips: just a metric }), ) Mixing Sources in One Struct The real payoff shows up when your config isn't all in one place. mamori doesn't care that these live in four different systems — it's one struct: Go type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` // Secrets Manager Workers int `source:"aws-ps:///myapp/workers"` // Parameter Store LogLevel string `source:"env:LOG_LEVEL" default:"info"` // environment TLSCert []byte `source:"file:///etc/tls/tls.crt"` // local file } Wrapping Up Automatic rotation is only half a feature if your app has to restart to notice. mamori supplies the other half: it watches AWS Secrets Manager and Parameter Store and reconciles a rotation into your running Go service — typed, validated, redacted, no bounce. Shell go get github.com/xavidop/mamori/providers/aws Full introduction: Introducing mamoriDocs: mamorigo.dev/docsSource: github.com/xavidop/mamori mamori is MIT-licensed and at v0.1.0 — early days, so feedback and issues are genuinely welcome. Happy coding!

By Xavier Portilla Edo CORE

Every few weeks, a new name shows up in security news. Project Glasswing. Daybreak. Lightwell. CodeMender. Big Sleep. Each one is presented as the next answer to AI-driven cyberattacks. It is easy to read this as a race between AI labs, cloud providers, and security vendors, and to ask which one will win. That is the wrong question. Each of these programs strengthens a different part of the same system. Some improve discovery. Some help engineers understand which bugs actually matter. Some generate patches. Some govern who is allowed to use powerful cyber tools at all. None of them, on its own, is a complete answer. The useful question for a security team is not which vendor we should bet on. It is which part of our own system is weakest, and which of these pieces fills that gap. This article lays out the evidence for why the pace of the problem has changed, walks through the lifecycle every vulnerability goes through, maps the current tools, programs, and rulebooks onto that lifecycle, and then treats AI agents themselves as a new class of identity that needs the same governance as any other. Along the way, I point to my own earlier articles where they cover the underlying engineering discipline in more depth. Why the Old Patch Cycle No Longer Works The pressure is not hypothetical. A few figures, each traceable to its source, show the shape of the problem. SignalWhat it showsSourceA record 48,185 CVEs were published in 2025, a 263 percent rise from 2020The volume of vulnerabilities to sort through has grown far faster than security teams haveCloud Security Alliance, "The Collapsing Exploit Window"Mandiant's M-Trends 2026 puts mean time to exploit at roughly negative seven days for high-value targetsAttacks now often begin before a fix is even publicMandiant M-Trends 2026, Google CloudCISA's Binding Operational Directive 26-04 requires a three-day fix for vulnerabilities that meet four risk criteriaThe US federal government has formally shortened its own patch deadlines because of AI-accelerated exploitationCISA, Binding Operational Directive 26-04The average KEV catalog patch deadline in 2026 is 14.4 days, down from 19.7 the year beforeDeadlines across the board are compressing, not just for the most severe bugsFederal News Network, May 2026AI agents found and helped fix 1,072 Chrome security bugs across two releases in June 2026, more than the previous 23 releases combinedDiscovery tooling has genuinely gotten faster, not just louder in marketingGoogle, "Stronger with every update" None of this changes how many hours are in your team's week. The number that actually matters is the one CISA is trying to compress by regulation: the time between a fix exists and the fix is live in production. Every program in this article is, in one way or another, trying to shrink that number. The Lifecycle Every Vulnerability Goes Through Before comparing tools, it helps to agree on the steps a single weakness goes through, from discovery to a proven fix. Lifecycle of a vulnerability or a bug Discovery has become cheap. Deciding, fixing safely, and rolling out remain slow, mostly because they need judgment, testing, and coordination across teams. That is where most vulnerability management programs still struggle, and it is where the new wave of AI security tools is trying to help. This lifecycle is also a loop, not a straight line. A better way to see the whole program is as a continuous cycle that feeds back into itself. Vulnerability management Discovery finds what might be wrong. Context asks whether the vulnerable component is actually reachable, deployed, and exposed. Prioritization ranks it against everything else competing for the same engineering hours. Remediation turns a finding into an engineering change: a dependency bump, a code fix, a rotated credential, a corrected IAM policy. Governance makes sure the whole process is reviewable, approved where it needs to be, and evidenced. Learning feeds what was found back into better defaults so the same class of issue shows up less often. I covered the review and testing discipline that sits behind the remediation and governance stages in Code Review Core Practices, which stays relevant even when the patch is drafted by an AI agent rather than a person. Three Programs, Three Different Jobs Three named programs currently dominate the conversation. Each one is strongest at a different point in the lifecycle above. Project Glasswing: Discovery at the Infrastructure Layer Anthropic launched Project Glasswing in April 2026. Launch partners include Amazon Web Services, Apple, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks, working with Claude Mythos Preview on defensive tasks. Anthropic committed up to 100 million dollars in usage credits and 4 million dollars in donations to open source security groups, and later extended access to roughly 150 more organizations across more than fifteen countries (Anthropic, Project Glasswing). The program targets critical infrastructure and widely used open source software, giving trusted defenders early access to a strong model before similar capability becomes broadly available. Mozilla has reported that the underlying model helped it find and patch 271 vulnerabilities in a recent Firefox release. OpenAI Daybreak: Tiered Access for Defenders OpenAI announced Daybreak on May 11, 2026, describing it as bringing together its frontier models, Codex Security, and a network of security partners to accelerate cyber defense and build security into software from the design stage rather than only patching after the fact (OpenAI, Daybreak). Access is split into two tiers that require separate approval. Daybreak's tiered access In September 2026, OpenAI added a subsidized access tier for organizations that keep essential services running, describing it as billions of dollars in access over roughly six months. Project Lightwell: Fixing Open Source Without Breaking Production IBM and Red Hat announced Project Lightwell on May 28, 2026, as a five-billion-dollar commitment backed by a global force of more than 20,000 engineers, and moved it to commercial launch on July 8 (IBM Newsroom, May 28, 2026; IBM Newsroom, July 8, 2026). It targets a specific and common problem. A library has a known bug. The official fix is a newer version, but the newer version breaks a certified production setup. Lightwell Network launched with a catalog of more than 6,500 remediated, signed, and certified dependencies across ecosystems such as Java and Python, giving teams a patched version of the release they already run instead of forcing a risky upgrade. Lightwell Clearinghouse Premier, in limited availability, adds coordinated patch embargo handling. In August 2026, IBM and Red Hat extended free access to more than 185 universities and 100 NGOs and think tanks that lack the engineering capacity to keep pace on their own (IBM Newsroom, August 4, 2026). Comparing the Three GlasswingDaybreakLightwellLed byAnthropic and partnersOpenAI and partnersIBM and Red HatCore questionWhere are the bugs in critical software?How can defenders use strong AI safely?How do we fix open source bugs without breaking production?Strongest lifecycle stepsFind, patchFind, check, proveFix, test, roll outAccess modelInvited partners, expandingApproved defenders, two tiersPaid subscription, with free access for universities and NGOsPrimary scopeCritical and open source infrastructureAn organization's own code and systemsOpen source dependencies in productionMain limitationMostly invite-basedSeparate approval for each tierCovers dependencies, not custom code Tools That Find and Patch Code Directly Alongside the three named programs, a set of tools now work directly inside the discovery and fix stages. ToolOwnerFunctionLifecycle stepSourceBig SleepGoogle DeepMind and Project ZeroSearches for memory safety bugs in widely used codeFindGoogle, Chrome security blogCodeMenderGoogle DeepMindPatches new bugs and rewrites existing code to remove whole classes of bugsFind, fixDeepMind, "Introducing CodeMender"Codex SecurityOpenAIBuilds an editable threat model from a repository and generates and tests patchesFind, checkOpenAI, DaybreakClaude SecurityAnthropicPublic beta for organizations outside GlasswingFind, checkAnthropic, Project Glasswing DeepMind is explicit that every CodeMender patch is reviewed by a person before it ships. That single fact matters more than the raw bug counts. It matches a habit I described in Ansible Code Scanning and Quality Checks With SonarQube: automated scanning finds candidates, but a human still signs off before anything reaches production. There is also support work happening around the big three programs, aimed at a second-order problem: faster discovery means more reports than open source maintainers can review. Anthropic has partnered with the Open Source Security Foundation's Alpha-Omega project to help maintainers triage incoming reports, and OpenAI's Daybreak commitment includes direct funding for groups such as Trail of Bits. Palo Alto Networks has also worked with IBM and Red Hat so its network-level protections can block attacks while a Lightwell fix is tested and rolled out, a practice known as virtual patching. It buys time. It does not replace the fix. What This Sets Up Discovery is now genuinely fast. Three well-funded programs and a handful of dedicated tools are throwing real capability at the find-the-bug and build-the-fix steps of the lifecycle. But finding more bugs faster does not, by itself, get you a faster or safer program. It just moves the bottleneck. The next problem is knowing which of those findings actually matter, making sure the AI agents doing this work are governed like the privileged systems they are, and having a plan that does not depend on an invitation to any of these programs. That is where Part 2 picks up. Further Reading Code Review Core PracticesAnsible Code Scanning and Quality Checks With SonarQube

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Every Go service I have ever shipped ends up growing the same organ. It starts small: a couple of os.Getenv calls. Then a secret needs to come from AWS Secrets Manager, so you add the SDK. Then someone wants that secret to rotate without a deploy, so you add a time.Ticker. Then the ticker needs a mutex because two goroutines read the config while it is being replaced. Then a bad value slips in and takes down production, so you add validation. Then a secret shows up in a log line during an incident, and now you are adding a redaction wrapper too. By the end you have a bespoke ConfigManager in every repo: a loader, a fetcher, a refresh loop, a mutex, and a prayer. It is never quite the same twice, it is rarely tested well, and it is exactly the kind of code nobody wants to own. I got tired of writing it, so I built a library that does it once, properly. It is called mamori; it is open source, and this article is the full tour: what it is, why it exists, how to use it end to end, and when you should (and shouldn't) reach for it. The Restart-to-Rotate Problem Here is the assumption baked into most Go config code: configuration is static. You read it at boot, decode it into a struct, and treat that struct as immutable for the lifetime of the process. If something changes, you redeploy. That assumption was fine when "config" meant a log level and a port. It is not fine anymore. In a modern service, a meaningful amount of your config is dynamic by nature: Database passwords and API tokens rotate on a schedule (and increasingly, automatically).Vault hands you dynamic credentials with a lease that expires in an hour.Feature flags flip in the middle of the day, on purpose.TLS certificates get reissued. The industry answer to "config is dynamic" has mostly been to push the problem into the platform. On Kubernetes, the External Secrets Operator syncs your cloud secret into a Secret, and then... you still restart the pod to pick it up, or you mount it as a file and hope your app re-reads it. The dynamic part gets flattened back into a static one at the worst possible layer. mamori takes the opposite position: your process should be able to observe a change at the source and react to it, in memory, without bouncing. Rotating a password should rotate a connection pool, not recycle a pod. What Is mamori? mamori (守り, Japanese for "protection" or "safeguard") is an embedded Go library — not an operator, not a server, not a sidecar. It lives inside your process. Its tagline says it plainly: Typed, watchable config & secrets for Go. Load configuration and secrets from anywhere into validated Go structs — and keep them reconciled at runtime, without a restart. There are three concepts you actually touch, and they map cleanly onto three types: Refs. Struct fields carry a source tag containing a URL-like reference to where the value lives — env:LOG_LEVEL, aws-sm://prod/db#password, vault://secret/app#token. A ref names what you want, not how to fetch it.Providers. A provider resolves one scheme (aws-sm, vault, k8s-secret, …) into a value. Providers register themselves using the same pattern as database/sql drivers, so the core module carries zero cloud-SDK dependencies — you only pull in the providers you use.The Reconciler. This is the interesting part. Watch resolves every ref once (fail-fast at startup), then keeps watching each source. When a value changes, it re-validates the entire struct, and only if the new snapshot is valid does it atomically swap it in and hand you a diff-aware callback. Your app never sees a half-updated, invalid config. If you have used IOptionsMonitor<T> in .NET, w.Get() is the direct analogue: always the latest valid configuration, lock-free. How mamori Compares I want to be honest about prior art, because none of these primitives are new — mamori's contribution is composing them. gocloud.dev/runtimevar is the closest primitive. It watches a single variable beautifully, but it is one variable at a time, with no struct composition, no tags, no validation, no secret hygiene. mamori adds all of that on top of the same idea.Viper/koanf are excellent, but they are config-first with secrets bolted on. mamori is deliberately secrets-first, with config included: secret types redact by default, and there is a go vet analyzer that yells at you if you store a secret in a plain string.envconfig/caarlos0/env are load-once. No watching, env only.The External Secrets Operator is complementary, not competitive. mamori is for apps that want to skip the Kubernetes Secret hop entirely, or that do not run on Kubernetes at all. It keeps no persistent external state, so there is no finalizer lifecycle to babysit. And to be equally clear about what mamori is not: it is not a secrets store (no encryption at rest, no server), not a sync engine between stores, not a general feature-flag platform, and not cross-language. It is a Go-idiomatic library, and it stays in its lane. Getting Started The core module has built-in env:, file://, and dotenv: providers, so you can do something useful with zero extra dependencies: Shell go get github.com/xavidop/mamori Heads up: mamori requires Go 1.26 or newer. It leans on modern generics ergonomics, so this is a hard floor, not a suggestion. The whole API is driven by a struct and its tags. You describe what you want; mamori figures out how: Go package main import ( "context" "log" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) type Config struct { DBPassword secret.String `source:"aws-sm://prod/db#password"` LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` TLSCert []byte `source:"file:///etc/tls/tls.crt"` } func main() { cfg, err := mamori.Load[Config](context.Background()) if err != nil { log.Fatal(err) } // cfg is fully resolved, defaulted, and validated. } The signature is exactly what you'd hope for: Go func Load[T any](ctx context.Context, opts ...Option) (T, error) Load resolves every ref once, applies defaults, validates, and returns your typed config. It fails fast: on any resolve or validation error, it returns the zero value and a non-nil error. It never returns a partially-populated struct — you either get a complete, valid config or an error. A few things worth knowing about the tags: TagMeaningsource:"..."The ref: where the value comes from.default:"..."Used only when the ref resolves to not-found — never on an error.validate:"..."go-playground/validator/v10 syntax, run on load and on every update.flatten:"json|yaml|env"Decode one payload into a nested struct.optional:"true"Tolerate not-found with no default (field keeps its zero value). That default-only-on-not-found rule is deliberate and important: if AWS Secrets Manager is having a bad day, you want an error, not a silent fallback to a default password. Watching and Reconciliation This is why mamori exists. Swap Load for Watch and your config becomes a living thing: Go func Watch[T any](ctx context.Context, opts ...Option) (*Watcher[T], error) You read the current config with a lock-free w.Get(), and you react to changes with an OnChange callback: Go w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(30*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { if ev.Changed("DBPassword") { pool.Rotate(ev.New.DBPassword.Reveal()) } for _, f := range ev.Fields { log.Printf("%s changed: %s -> %s", f.Path, f.OldVersion, f.NewVersion) } }), mamori.OnError(func(err error) { metrics.Inc("config_error") }), ) if err != nil { log.Fatal(err) } defer w.Close() cfg := w.Get() // always the last VALID config, no lock needed The Change[T] you get in the callback carries the old value, the new value, and a per-field diff, plus a Changed(path) helper so you only do expensive work (like rebuilding a connection pool) when the field you care about actually moved. The guarantees are the whole point, so let me spell them out: Atomicity. OnChange only ever fires with a fully re-validated snapshot. If an incoming value fails validation, the update is rejected: Get() keeps returning the last good config and OnError receives a *ValidationError. Your config never transitions into a broken state mid-flight.Push where possible, poll where not. Providers whose backend can push changes (Kubernetes informers, Vault lease watchers, Consul blocking queries, fsnotify for files) push natively. Everything else is polled on WithPollInterval, jittered by ±20% by default so a fleet of pods doesn't stampede your secrets backend on the same tick. If a native watch fails to start, mamori quietly falls back to polling.Coalescing. Rapid-fire changes within a debounce window (500ms default, overridable per-field with ?debounce=) collapse into a single event.Lease-aware. If a provider reports an expiry (Vault dynamic creds), mamori schedules a refresh before the lease dies, rather than waiting for the next poll.First event. Watch resolves the initial config before it returns, so OnChange fires only on subsequent changes. Read the initial state from w.Get(). There's a nice operational escape hatch too. Transient backend outages shouldn't page anyone — but prolonged staleness should. WithStale draws that line for you: Go w, err := mamori.Watch[Config](ctx, mamori.WithStale(10*time.Minute), mamori.OnError(func(err error) { var stale *mamori.StaleError if errors.As(err, &stale) { alert.Page("config stale", stale) return } metrics.Inc("config_transient_error") }), ) A blip retries with exponential backoff and keeps the last good value. Ten minutes of failure escalates to a hard *StaleError you can actually page on. Secrets That Behave Like Secrets Configuration and secrets are treated differently, on purpose. Any field that holds a secret should be a secret.String or secret.Bytes instead of a plain string or []byte. These types redact themselves everywhere a value normally leaks: Go s := secret.NewString("hunter2") fmt.Println(s) // [REDACTED] fmt.Sprintf("%v", s) // [REDACTED] fmt.Sprintf("%#v", s) // [REDACTED] json.Marshal(s) // "[REDACTED]" slog.Info("login", "pw", s) // pw=[REDACTED] s.Reveal() // "hunter2" <- the only way to read the plaintext It works because the type implements fmt.Stringer, fmt.GoStringer, json.Marshaler, and slog.LogValuer. The single, greppable access point is Reveal(). That is a feature: in code review or an audit, grep Reveal shows you every place a secret is actually unwrapped. (secret.Bytes.Reveal() returns []byte rather than a string, for keys and certificates.) And because "just remember to use secret.String" is exactly the kind of discipline that erodes under deadline pressure, mamori ships a go vet analyzer that enforces it: Shell go install github.com/xavidop/mamori/tools/reconcilevet/cmd/reconcilevet@latest go vet -vettool=$(which reconcilevet) ./... reconcilevet flags any field whose source points at a secret-bearing scheme (aws-sm, gcp-sm, azure-kv, vault, op, sops, k8s-secret) but is stored in a plain string or []byte. The leak becomes a build failure, not a postmortem. Providers: More Than 30 Sources The core stays dependency-free; each cloud backend is its own module you opt into. A ref's scheme picks the provider: Shell go get github.com/xavidop/mamori/providers/aws # aws-sm:// aws-ps:// go get github.com/xavidop/mamori/providers/vault # vault:// go get github.com/xavidop/mamori/providers/gcp # gcp-sm:// go get github.com/xavidop/mamori/providers/azure # azure-kv:// go get github.com/xavidop/mamori/providers/k8s # k8s-secret:// k8s-cm:// go get github.com/xavidop/mamori/providers/consul # consul:// go get github.com/xavidop/mamori/providers/doppler # doppler:// go get github.com/xavidop/mamori/providers/onepassword # op:// go get github.com/xavidop/mamori/providers/sops # sops:// That is a small slice. All in, there are more than 30 providers: secret managers (AWS Secrets Manager, AWS Parameter Store, Vault, GCP, Azure Key Vault, Doppler, 1Password, SOPS), databases (Postgres, MySQL, SQLite, MongoDB, DynamoDB, Cosmos DB, Redis), KV stores (Consul, etcd), Kubernetes Secrets and ConfigMaps, Firebase/Firestore/Remote Config, object storage (S3, GCS, Azure Blob), and a whole shelf of feature-flag platforms (LaunchDarkly, Unleash, Flagsmith, ConfigCat, Split, GrowthBook, Flipt, GO Feature Flag). Registration follows the database/sql convention — a blank import runs the provider's init(), which calls mamori.Register(...): Go import _ "github.com/xavidop/mamori/providers/vault" When you need explicit configuration (a region, a custom client), construct the provider and pass it with WithProvider. That takes precedence over the registry for that call: Go import awsprov "github.com/xavidop/mamori/providers/aws" cfg, err := mamori.Load[Config](ctx, mamori.WithProvider(awsprov.NewSecretsManager(awsprov.WithRegion("eu-west-1"))), ) The ref grammar is compact — scheme://path[#key][?opts] — with a couple of quirks worth internalizing: The #key fragment selects one field out of a JSON payload: aws-sm://prod/db#password.AWS Parameter Store, etcd, sops, and file preserve absolute paths, so they use a triple slash: aws-ps:///myapp/log-level, file:///etc/tls/tls.crt.Doppler is the one provider where the # fragment is required: doppler://backend/prd#STRIPE_API_KEY. When a payload is a structured blob, you can decode it straight into a nested struct with flatten: Go type Config struct { Redis RedisConfig `source:"aws-sm://prod/redis" flatten:"json"` } type RedisConfig struct { Addr string `mapstructure:"addr"` Password secret.String `mapstructure:"password"` DB int `mapstructure:"db"` } One AWS Secrets Manager call, one JSON blob, cleanly typed — with the password still redacting itself. Writing Your Own Provider If your source of truth is homegrown — an internal config service, a weird legacy database table — you can teach mamori about it. The interface is intentionally tiny: Go type Provider interface { Scheme() string Resolve(ctx context.Context, ref Ref) (Value, error) } A minimal implementation looks like this: Go func (p *Provider) Resolve(ctx context.Context, ref mamori.Ref) (mamori.Value, error) { raw, err := p.fetch(ctx, ref.Path) if isNotFound(err) { return mamori.Value{}, mamori.ErrNotFound // MUST satisfy errors.Is } if err != nil { return mamori.Value{}, err } if ref.Key != "" { raw, err = mamori.SelectKey(raw, ref.Key) // implements #key against a JSON payload if err != nil { return mamori.Value{}, err } } return mamori.Value{ Bytes: raw, Version: mamori.VersionHash(raw), // change detection; use a native revision if you have one Sensitive: true, // true for secret-bearing sources }, nil } func init() { mamori.Register(New()) } // database/sql pattern; panics on a duplicate scheme The Version field is what powers cheap change detection: if your backend exposes a revision (a VersionId, a Vault version, an mtime hash), return it; otherwise mamori.VersionHash(bytes) gives you a stable FNV hash for free. Two optional interfaces let you do better when the backend supports it: Go type WatchableProvider interface { Provider Watch(ctx context.Context, ref Ref) (<-chan Update, error) } type BatchProvider interface { Provider ResolveBatch(ctx context.Context, refs []Ref) (map[string]Value, error) } Implement Watch only if your backend can genuinely push — otherwise let mamori's poller do the work, so behavior stays consistent across providers. Implement ResolveBatch if you can fetch many refs in one round trip (the AWS provider does this automatically). Best of all, you don't have to trust yourself to get the contract right. There's a conformance kit — github.com/xavidop/mamori/providertest — that exercises resolution, not-found typing, version monotonicity, concurrency, context cancellation, native watching, goroutine hygiene, and even asserts you never accidentally log the payload. Point it at your provider, and it tells you if you've broken the rules. Middleware: Cache, Failover, Audit, Rate-Limit Because every provider is just a Provider, you can wrap one in another. The middleware package ships the decorators you'd otherwise reinvent: MiddlewareWhat it doesCache(ttl, inner)Memoize successful resolves. Errors and not-found are not cached.Failover(primary, replicas...)Try the primary, fall back to replicas on transport errors. Not-found is authoritative.RateLimit(rps, inner)Cap resolves per second so you don't trip your secrets backend's throttle.Audit(logger, inner)Log scheme, ref, latency, and outcome — never the value.Prefix(prefix, inner)Namespace every ref path — handy for multi-tenant setups. They compose by nesting, so you can read the stack top-down: Go import "github.com/xavidop/mamori/middleware" mamori.WithProvider( middleware.Cache(5*time.Minute, middleware.Audit(logger, middleware.RateLimit(10, middleware.Failover(primarySM, replicaSM), ), ), ), ) Crucially, the shipped middleware preserves the WatchableProvider and BatchProvider capabilities of whatever they wrap — decorating a provider never silently downgrades it from push to poll. Observability Out of the Box The core takes no OpenTelemetry dependency (keeping it lean), but there's a bridge module when you want metrics and traces: Shell go get github.com/xavidop/mamori/x/otel Go import ( "go.opentelemetry.io/otel" mamoriotel "github.com/xavidop/mamori/x/otel" "github.com/xavidop/mamori" ) meter, _ := mamoriotel.NewMeter(otel.Meter("mamori")) tracer := mamoriotel.NewTracer(otel.Tracer("mamori")) w, err := mamori.Watch[Config](ctx, mamori.WithMeter(meter), mamori.WithTracer(tracer), ) You get a mamori.resolve.duration histogram, mamori.refresh.count and mamori.watch.errors counters (all tagged by scheme), and a span per resolve. And because WithMeter/WithTracer take tiny internal interfaces, you can implement them against Prometheus, statsd, or a test recorder without pulling OTel into your binary at all. A Complete, Runnable Example Here is the whole loop — load, watch, react — in one file you can run today (it's the examples/basic program from the repo): Go package main import ( "context" "fmt" "log/slog" "os" "time" "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" ) const tokenPath = "/tmp/mamori-example-token" type Config struct { LogLevel string `source:"env:LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"` Workers int `source:"env:WORKERS" default:"4" validate:"gte=1,lte=256"` APIToken secret.String `source:"file:///tmp/mamori-example-token"` } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) _ = os.WriteFile(tokenPath, []byte("initial-token"), 0o600) defer func() { _ = os.Remove(tokenPath) }() ctx, cancel := context.WithCancel(context.Background()) defer cancel() w, err := mamori.Watch[Config](ctx, mamori.WithPollInterval(2*time.Second), mamori.OnChange(func(ev mamori.Change[Config]) { for _, f := range ev.Fields { logger.Info("config changed", "field", f.Path, "from", f.OldVersion, "to", f.NewVersion) } if ev.Changed("APIToken") { logger.Info("rotating clients with the new API token", "token", ev.New.APIToken) } }), mamori.OnError(func(err error) { logger.Warn("reconcile error", "err", err) }), ) if err != nil { logger.Error("watch failed", "err", err) os.Exit(1) } defer func() { _ = w.Close() }() cfg := w.Get() logger.Info("loaded config", "logLevel", cfg.LogLevel, "workers", cfg.Workers, "apiToken", cfg.APIToken) // Simulate a rotation a few seconds in. go func() { time.Sleep(3 * time.Second) _ = os.WriteFile(tokenPath, []byte("rotated-token"), 0o600) }() time.Sleep(8 * time.Second) logger.Info("final config", "apiToken", w.Get().APIToken) } Shell LOG_LEVEL=debug WORKERS=8 go run ./examples/basic Notice the token field in the log output: it prints [REDACTED] every time, even though the program is actively rotating it. That's the whole philosophy in one demo — live rotation, no restart, and nothing leaks. When to Reach for mamori (and When Not) Reach for it when: Your config or secrets genuinely change while the process runs, and you want to react in memory instead of bouncing pods.You pull from more than one source and are tired of a different SDK and refresh strategy for each.Secret hygiene matters and you want redaction and a go vet guardrail for free.You run outside Kubernetes, or you're on Kubernetes but want to skip the Secret-sync hop. Probably skip it when: Your config is a log level and a port from the environment, read once at boot. os.Getenv is fine — don't add a dependency.You need a secrets store (encryption at rest, a server component) — that's Vault's job, not mamori's.You're syncing secrets between stores — that's what ESO, vals, and teller are for. Wrapping Up mamori is the library I wish I'd had every time I typed type ConfigManager struct for the fifth time. It takes the boring, error-prone, security-sensitive plumbing — load, validate, watch, redact, reconcile — and makes it a typed struct and one function call. It's open source under the MIT license and currently at v0.1.0 — early, pre-1.0, and moving fast, so feedback and issues are genuinely welcome. Give it a spin: Shell go get github.com/xavidop/mamori Docs: mamorigo.dev/docsHomepage: mamorigo.devSource: github.com/xavidop/mamori Provider Deep Dives If you live on a particular platform, I've written focused, hands-on companions: Hot-reloading Kubernetes Secrets and ConfigMaps in Go — no pod restartsRotating AWS Secrets Manager secrets without restarting your Go serviceGCP Secret Manager, live-reloaded in GoAzure Key Vault secrets that reconcile themselves in Go If you build something with it, or write a provider for a backend I haven't covered yet, I'd love to hear about it. Happy coding!

By Xavier Portilla Edo CORE

Context The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP. One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed. The service used placement constraints, max replicas per node = 1, and a Traefik health check configured for /actuator/health on port 8080. This meant we relied on external health checking and strictly controlled replica placement across nodes. Swarm-level load balancing was intentionally disabled. We needed routing control at the Traefik layer, along with the ability to quickly remove unhealthy backends based on their health check status. For this production setup, that provided more flexibility than the built-in Swarm load-balancing logic. What Happened The issue was neither a widespread outage nor a failure of the entire service. One replica became unreachable to Traefik over the overlay network: a Traefik instance on one of the manager nodes could not establish a connection to that replica’s backend endpoint. The second replica continued to work normally, so the service as a whole remained available. For regular traffic, the incident did not affect service availability: Traefik continued routing requests to the healthy replica and removed the unavailable backend from routing. During targeted manual testing of the unhealthy replica’s backend endpoint through Traefik, requests ended in a timeout or HTTP 503. The container appeared healthy, but only locally. The service responded inside the container, and /actuator/health returned UP. However, other containers and Traefik instances on remote nodes could not establish a connection to that replica through its IP address in mgp_default. Initial Checks The container had the same DNS name in both networks. This ruled out name mismatch as an independent cause of the unavailability: the issue occurred specifically at the network-connectivity level to the replica IP in mgp_default. First, we identified the problematic container and inspected its state: Shell docker ps --filter "name=mgp_router-app" --q docker inspect 6bd15b62f891 Next, we checked the networks connected to the container: Shell docker inspect 6bd15b62f891 --format '{{json .NetworkSettings.Networks}' | jq The container was attached to two networks: ingress with IP 10.22.3.137/22;mgp_default with IP 10.29.72.232/13. We then entered the container and checked whether the application was listening on the expected port: Shell docker exec -it 6bd15b62f891 bash ss -tuln The health endpoint also responded correctly: Shell curl "http://10.29.72.232:8080/actuator/health" The expected response was: Shell {"status":"UP"} At this point, it was clear that the issue was not the application itself, its startup, or its bind address inside the container. The service was alive, but that alone said nothing about its reachability from outside the container. Checking Connectivity Between Nodes The next step was critical: We checked whether another node in the cluster could reach the container using its overlay IP. Shell docker exec -it 373354916bfc bash ping 10.22.3.137 This was the IP from the ingress network, and the test confirmed ICMP reachability of the address in that network. That indicated the issue did not look like a complete loss of inter-node connectivity. The situation was different for the mgp_default IP: pinging it from other nodes failed, even though this overlay network was supposed to provide inter-container communication in Docker Swarm. This distinction was important. ICMP reachability of the address in ingress remained available, while connectivity to the replica through mgp_default was broken. Therefore, the problem was not the container itself or a complete network failure; it was a specific break in the network path used to reach that replica. What Docker Logs Showed The dockerd logs on the affected node provided another useful signal. They contained repeated messages related to Swarm network synchronization: Shell Jun 02 11:08:42 node-router2 dockerd: time="2025-06-02T11:08:42.462965233Z" level=warning msg="bulk sync to node 091d01df89ae failed: failed to send a TCP message during bulk sync: dial tcp 10.170.0.145:7946: i/o timeout" Jun 02 11:17:02 node-router2 dockerd: time="2025-06-02T11:17:02.581936915Z" level=error msg="Bulk sync to node 091d01df89ae timed out" The logs also contained NetworkDB stats entries: Shell Jun 02 11:07:08 node-router2 dockerd: time="2025-06-02T11:07:08.012509772Z" level=info msg="NetworkDB stats node-router2(2642aeb5172b) - netID:q9athb3moj967oxmb3y8xw4lj leaving:false netPeers:58 entries:13563 Queue qLen:2 netMsg/s:7" These messages do not prove a specific bug or root cause, but they indicate problems with exchanging network state between nodes. Combined with the endpoint being unreachable through mgp_default, they strengthened the working hypothesis of an overlay-networking issue. This was an important diagnostic signal: the Docker daemon was running, and the container was started, but the cluster’s network state was not ideal and was likely becoming inconsistent during node-to-node state exchange. Investigating the Hypotheses Hypothesis 1: The container itself was the problem. This hypothesis was not confirmed. A curl request inside the container to http://10.29.72.232:8080/actuator/health returned {"status":"UP"}, and ss -tuln showed that the service was listening on the required port. Hypothesis 2: The problem was port publishing or binding. This hypothesis was not confirmed. Even though port 10005 was reachable externally, that did not explain why Traefik could not reach port 8080 over the overlay network. The container worked locally, while the issue occurred on the network path between nodes. Hypothesis 3: Traefik itself was the problem. Probably not. Traefik was not the source of the failure; it was the first component that detected that the backend was unavailable from its node. Hypothesis 4: The Docker Swarm overlay network was the problem. This became the primary working hypothesis. The replica was reachable from inside its container but was not properly reachable from other nodes through its overlay IP. Shell curl http://10.29.72.232:8080 nc -vz 10.29.72.232 8080 The curl request worked locally, while nc -vz 10.29.72.232 8080 failed from other nodes. This indicated that traffic was not being routed correctly through the overlay network. An MTU/VXLAN issue was also considered. This is a plausible class of problems for Docker overlay networks, especially when there is an MTU mismatch or a Path MTU Discovery issue. In this case, however, it remained only a hypothesis rather than a confirmed root cause. Recovery We restored the problematic replica by forcing Docker Swarm to recreate the task through a rolling update: Shell docker service update --force \ --update-parallelism 1 \ --update-order start-first \ --update-delay 30s \ --detach=true \ mgp_router-app --force triggered an update even though neither the image nor the service configuration had changed. Swarm recreated the task for the affected replica and reattached it to mgp_default; after that, Traefik could reach it again over the overlay network. The rollout parameters were selected to avoid affecting service availability: --update-parallelism 1 – update one replica at a time;--update-order start-first – start the new task before stopping the old one;--update-delay 30s – give the new task time to start and pass its checks. Restarting Docker on the node and placing the node into drain mode were considered fallback measures, but they were not required to resolve this incident. Why It Happened The exact upstream root cause cannot be stated without additional evidence. However, the most well-supported explanation is that a local network-state inconsistency occurred in the Docker Swarm overlay network. As a result, one replica remained operational inside its container, but became unreachable through its overlay IP. The most plausible explanations include: the overlay network on the node was not fully ready when the container started;an issue in Docker libnetwork;an outdated VXLAN routing table;an iptables issue or a race condition while updating network rules;a node restart without restarting the Docker daemon;an issue with a Swarm endpoint that was not properly advertised or connected. These situations are particularly unpleasant because Docker continues to consider the container running and its internal health check green, while the network path to the replica is already partially broken. Key Takeaways The main conclusion is simple: a healthy container does not necessarily mean a reachable replica. This is critical in production because: a local curl inside the container does not prove external reachability;an internal health check does not guarantee that Traefik can reach the backend;Swarm may remain formally healthy even when one replica has dropped out of normal network connectivity. Therefore, in such clusters, it is necessary to check not only container state but also inter-node reachability over the overlay network. What to Monitor To prevent similar incidents, it is useful to monitor not only service health but also the cluster’s network plane. The minimum set of checks includes: reachability of container overlay IPs from other nodes;health checks through Traefik;bulk sync and NetworkDB errors in Docker logs;recurring issues on the same node;consistency between a container’s local health and its external reachability. It is also useful to have a script that specifically checks connectivity between nodes through the overlay network, not just through ingress or localhost. Conclusion This case demonstrated several important points. First, a container can be healthy locally while remaining unreachable externally. This is especially important for services that handle user traffic. Second, Traefik is a good early indicator of network problems. It can reveal that a backend on one node has stopped responding. Third, Docker Swarm overlay networking can degrade locally. In that case, the cluster may appear operational, while one replica has already dropped out of normal routing. Fourth, such incidents must be diagnosed along the network path, not only through application state. First, check the container from the inside, then verify reachability from other nodes, then inspect Docker logs, and only after that draw conclusions about the root cause.

A Kafka recovery often looks successful before it actually is. The replay job exits. Consumer lag drops. Error rate returns to normal. The dashboard is green enough for everyone to want the incident closed. Then somebody checks the downstream projection, account balance, inventory state, enrichment table, or audit report and finds the real problem: data moved, but state did not converge. That is the failure this article is about. Consumer lag is a progress signal, not a recovery verdict. It proves position movement. It does not prove terminal outcomes, idempotent side effects, owned failures, or source-to-target convergence. If an incident review ends with “lag went back to zero,” the team has proved movement, not recovery. Declare the Recovery Cohort First The first observability decision is not which metric to graph. It is what work the recovery run is supposed to prove. A useful cohort names: the topic, partitions, and offset rangethe event types included in the runthe business-key scopethe expected record countthe recovery reasonthe ownerand the stop conditions A compact contract object makes that scope reviewable: YAML recovery_run: id: rec-2026-09-05-01 reason: projection_drift_after_consumer_restart owner: billing-platform-oncall run_type: incident_repair source: topic: billing-events event_types: - InvoicePosted - CreditApplied offsets: - partition: 8 from: 492100 to: 492881 - partition: 9 from: 388010 to: 388522 business_scope: key_type: account_id selector: incident_account_hashes_2026_09_05 expectations: expected_record_count: 1294 reconciliation: source_of_truth: billing_ledger checks: - account_day_balance_matches - no_missing_decision_events - no_unowned_final_failures safety: dry_run_required: true max_retry_attempts: 3 stop_on_reconciliation_mismatch: true The object does not have to be beautiful. It has to be unambiguous. During an incident, ambiguity is expensive: “replay the billing topic” is not a scope, and “check the dashboard” is not evidence. The cohort also forces a design decision: do you need partition/offset proof, business-key proof, or both? Offsets are excellent for Kafka lineage. Business keys are what downstream systems usually reconcile. A serious recovery plan connects them instead of pretending one replaces the other. Count Terminal Outcomes, Not Processed Records Processed count is a dangerous comfort metric. A record can be processed and still be wrong. It can be skipped correctly. It can be suppressed as a duplicate. It can be retried. It can land in a dead letter queue. It can be quarantined because replaying it would violate a business rule. The recovery run should account for outcomes explicitly: OutcomeTerminal?MeaningAPPLIEDYesThe expected side effect was performed.UPDATEDYesExisting target state was corrected or brought current.SKIPPEDYesThe record was intentionally ignored under a documented rule.DUPLICATE_SUPPRESSEDYesIdempotency prevented a repeat side effect.FAILED_RETRYABLENoProcessing failed and remains inside retry policy.FAILED_FINALYesProcessing failed and needs owner action.QUARANTINEDYesThe record is retained but should not be replayed without domain review. The key equation is simple: expected = applied + updated + skipped + duplicate_suppressed + failed_final + quarantined Retryable failures should not count as terminal until the retry policy is exhausted or the run is stopped. If the math does not close, the recovery has not been proven. Keep Metrics Bounded Recovery runs create tempting labels: run IDs, account IDs, order IDs, offset ranges, incident IDs. Put those blindly into metrics and the observability system becomes the next incident. Use metrics for bounded dimensions and put high-cardinality detail in decision logs, traces, audit tables, or exemplars. MetricBounded labelsPurposekafka_recovery_records_expectedtopic, event_type, run_typeDefines the denominator.kafka_recovery_records_unaccountedtopic, event_type, run_typeShows records not yet terminal.kafka_recovery_records_terminal_totaloutcome, event_type, run_typeShows how records completed.kafka_consumer_duplicate_suppressed_totalconsumer_group, event_type, run_typeProves replay did not duplicate side effects.kafka_recovery_reconciliation_mismatchescheck_name, run_typeBlocks false recovery success. That split matters. Metrics tell you that recovery is unhealthy at a class level. The decision record tells you which run, offsets, business keys, and owners are involved. Emit Decisions You Can Inspect A useful recovery log does not just say that a handler ran. It says why a record took a path. JSON { "recoveryRunId": "rec-2026-09-05-01", "eventId": "evt-1001", "businessKeyHash": "bkh_7f3a", "topic": "billing-events", "partition": 8, "offset": 492881, "eventType": "InvoicePosted", "outcome": "DUPLICATE_SUPPRESSED", "reason": "idempotency_key_exists", "schemaVersion": "v4", "consumerGroup": "usage-aggregator", "runType": "incident_repair", "decidedAt": "2026-09-05T18:42:31Z" } Notice what is missing: raw payloads, customer identifiers, tokens, and private business data. Recovery logs should explain decisions without becoming a second data-leak surface. The Java side can stay small. The following Java 17/Spring/Micrometer sketch is illustrative, not a complete production service. It assumes repository, meterRegistry, objectMapper, and log are class fields: Java record RecoveryDecision( String recoveryRunId, String eventId, String businessKeyHash, String topic, int partition, long offset, String eventType, RecoveryOutcome outcome, String reason, String consumerGroup, String runType, Instant decidedAt ) {} void emit(RecoveryDecision decision) { repository.insert(decision); Tags tags = Tags.of( "event_type", decision.eventType(), "run_type", decision.runType(), "outcome", decision.outcome().name() ); meterRegistry .counter("kafka_recovery_records_decision_total", tags) .increment(); if (decision.outcome() != RecoveryOutcome.FAILED_RETRYABLE) { meterRegistry .counter("kafka_recovery_records_terminal_total", tags) .increment(); } log.info("kafka_recovery_decision={}", toJson(decision)); } String toJson(RecoveryDecision decision) { try { return objectMapper.writeValueAsString(decision); } catch (JsonProcessingException error) { return "{\"serialization_error\":\"recovery_decision\"}"; } } That emitter is not the recovery system. It is the instrumentation boundary. It makes the record decision visible in the durable decision log, the metric stream, and the incident timeline. Reconciliation Is the Verdict Layer The hardest part of recovery observability is deciding what “correct” means. For some domains, a count is enough. For most serious domains, it is not. A count can match while the wrong keys, amounts, versions, or timestamps remain wrong. The source of truth must be named. The tolerance window must be named. Late arrivals, clock skew, eventually consistent sinks, and manual exceptions must be handled explicitly. Otherwise reconciliation becomes another dashboard panel that looks precise while hiding the policy decision. For example, if the recovery cohort is supposed to repair account/day billing state, compare the source ledger to the target projection for the same cohort. This is PostgreSQL-style SQL; use an equivalent pattern for your database: PLSQL WITH expected AS ( SELECT account_hash, business_date, SUM(amount_cents) AS expected_amount_cents FROM billing_ledger WHERE recovery_run_id = :recovery_run_id GROUP BY account_hash, business_date ), actual AS ( SELECT account_hash, business_date, amount_cents AS actual_amount_cents FROM billing_daily_projection WHERE recovery_run_id = :recovery_run_id ) SELECT COALESCE(e.account_hash, a.account_hash) AS account_hash, COALESCE(e.business_date, a.business_date) AS business_date, COALESCE(e.expected_amount_cents, 0) AS expected_amount_cents, COALESCE(a.actual_amount_cents, 0) AS actual_amount_cents FROM expected e FULL OUTER JOIN actual a ON e.account_hash = a.account_hash AND e.business_date = a.business_date WHERE COALESCE(e.expected_amount_cents, 0) <> COALESCE(a.actual_amount_cents, 0); Also check whether every selected event produced a decision: PLSQL SELECT c.event_id FROM kafka_recovery_cohort_event c LEFT JOIN kafka_recovery_decision_log d ON c.recovery_run_id = d.recovery_run_id AND c.event_id = d.event_id WHERE c.recovery_run_id = :recovery_run_id AND d.event_id IS NULL; Those queries are intentionally plain. Recovery confidence should not depend on a heroic debugging session. It should depend on executable checks that can run before anyone is asked to close the loop. Alert on Evidence Failure, Not Just Kafka Symptoms Recovery alerts should catch bad recovery, not only broken Kafka. Page when terminal outcomes do not close after the deadline, reconciliation fails after the check has completed, or an active recovery run stops making progress. Do not page simply because an in-progress recovery still has work remaining. YAML groups: - name: kafka-recovery rules: - alert: KafkaRecoveryUnaccountedRecords expr: | kafka_recovery_records_unaccounted > 0 and on(topic, event_type, run_type) kafka_recovery_deadline_reached == 1 for: 15m labels: severity: page annotations: summary: "Kafka recovery has overdue unaccounted records" - alert: KafkaRecoveryRunTooOld expr: | time() - kafka_recovery_last_progress_timestamp_seconds > 1800 and on(topic, event_type, run_type) kafka_recovery_run_active == 1 for: 5m labels: severity: ticket annotations: summary: "Kafka recovery run has stopped making progress" The alert is not the verdict. It tells the operator where to look. The verdict should still come from the recovery evidence record. Test the Evidence Path Recovery safety should not only be described. It should be testable. At minimum, test the evidence path with a small matrix: TestInjectionExpected proofDuplicate replayProcess the same event_id twice.One APPLIED, one DUPLICATE_SUPPRESSED, no second side effect.Missing decisionSelect an event into the cohort but skip emission.no_missing_decision_events fails.Projection driftChange target state after replay.Reconciliation returns mismatched keys.Poison recordSend a record that violates a domain rule.Outcome is QUARANTINED with owner and reason.Stalled runStop progress after partial terminal outcomes.Unaccounted-record or stale-progress alert fires. Run these tests in two places. First, keep a small contract-test suite around the recovery library or consumer adapter so decision emission cannot regress. Second, run a replay drill in a non-production environment with a deliberately small cohort. The drill should prove that the metrics, decision log, reconciliation query, alerts, and final verdict all agree before the same pattern is trusted during an incident. Recovery Definition of Done Before closing a Kafka recovery run, require evidence for each item: CheckEvidenceCohort declaredTopic, partitions, offsets, event types, expected count, owner, and stop conditions are recorded.Terminal math closesExpected records equal terminal outcomes, excluding retryable in-flight failures.Idempotency is visibleDuplicate suppressions and side-effect decisions are emitted as bounded metrics and durable decision records.Failure ownership existsFinal failures and quarantined records have reasons, owners, and next actions.Reconciliation passedThe named source of truth matches the target state for the approved scope, or exceptions are explicitly owned.Verdict recordedThe run ends as RECOVERED, RECOVERED_WITH_EXCEPTIONS, BLOCKED, QUARANTINED, or FAILED. This full pattern is not free. Structured decision logs add write volume. Reconciliation consumes compute. Audit tables need retention and access control. Quarantine creates human workflow. There are also cases where the full pattern is too much. A stateless consumer that only updates an internal cache may not need decision tables, reconciliation jobs, and formal verdicts. Start with the recovery definition of done, then choose the minimum evidence that proves the business risk you actually have. Kafka gives teams a powerful recovery primitive because the log can be replayed. That does not make recovery safe by itself. Lag tells you the consumer moved. Recovery evidence tells you whether the system became correct again.

CI failures often expose a narrow defect through a broad stream of evidence such as compiler output, test reports, stack traces, changed files, dependency metadata, and build logs. Large language models can turn that evidence into candidate repairs, and recent industrial research has shown that LLM-assisted CI repair can resolve a meaningful share of compilation failures when the model receives carefully selected context. The difficult part is not generating a patch. It is ensuring that automated repair changes only code justified by the failure, leaves protected files untouched, and proves that the original failure is actually fixed. Automated program repair research has repeatedly identified patch overfitting as a core risk: a patch can satisfy available tests without being semantically correct. Capture Failure Evidence Before Asking for a Fix A repair agent should start from structured CI evidence rather than a raw prompt containing an entire build log. GitHub Actions artifacts can persist test output, logs, and other files after a job completes, making them a practical handoff format between the failing job and a later repair job. For a Maven project, the failing job can retain both console output and Surefire reports: YAML - name: Run tests run: ./mvnw test 2>&1 | tee ci.log - name: Upload failure evidence if: failure() uses: actions/upload-artifact@v4 with: name: failure-evidence path: | ci.log target/surefire-reports/ The repair process should normalize this data into a compact incident object containing the failing test, exception type, stack trace, failing command, commit SHA, and original pull-request diff. Current surveys of LLM-based software repair describe retrieval and analysis augmentation as important layers for repository-level repair, reinforcing the value of selecting failure-relevant context before generation. Before patch generation, the failure should be reproduced in an isolated checkout. Git worktrees support linked working directories and can create a detached worktree for experimental changes without disturbing the primary checkout. A repair job can create a disposable workspace and rerun only the failing test: Shell git worktree add --detach ../repair "$HEAD_SHA" cd ../repair ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test Maven Surefire supports selecting a specific test class and, for supported providers, individual test methods. If the failure cannot be reproduced, automatic patching should stop. A passing rerun provides no reliable oracle for judging a generated fix. Turn Failure Localization Into an Edit Boundary The central control is an explicit distinction between readable and editable files. A failing test, configuration file, or build script may be necessary for diagnosis without being safe for autonomous modification. A scope document can represent that boundary: YAML failure: PaymentServiceTest.shouldRetryTimeout readable: [src/test/java/com/acme/payment/**, src/main/java/com/acme/payment/**, pom.xml] editable: [src/main/java/com/acme/payment/**] protected: [src/test/**, pom.xml, .github/**, db/migration/**] The editable set can be derived from the original PR diff, stack-trace locations, and a small neighborhood of related production symbols. Git provides machine-readable changed-path output through git diff --name-only, while pathspecs can limit comparisons to selected repository regions. The model can inspect broader context, but the orchestrator should reject any patch that escapes the approved surface. A validator needs a repair baseline captured immediately before the model edits the worktree: Shell REPAIR_BASE=$(git rev-parse HEAD) git diff --name-only "$REPAIR_BASE" > repair-files.txt while read -r path; do case "$path" in src/main/java/com/acme/payment/*) ;; *) echo "Out-of-scope change: $path"; exit 1 ;; esac done < repair-files.txt This check matters because prompt instructions such as "modify only relevant code" are advisory, while a diff gate is enforceable. Git also provides exit-status behavior through git diff --exit-code, which is useful for automated policy checks. Generate a Patch, Not an Open-Ended Coding Session The repair model should receive a bounded request like normalized failure data, relevant source fragments, the original PR diff, editable paths, protected paths, and the exact verification command. The response should be structured around a patch rather than unrestricted shell execution. Java RepairRequest request = new RepairRequest( failure, relevantSource, originalDiff, editablePaths, protectedPaths, verificationCommand ); Patch patch = repairModel.generate(request); patchValidator.validate(patch); patchApplier.apply(patch); The model proposes the smallest plausible correction where the orchestrator remains responsible for permissions and validation. This separation reflects a broader trend in LLM-based program repair toward procedural and agentic workflows that combine generation with retrieval, analysis, execution feedback, and external control. For an initial implementation, tests and CI configuration should remain read-only. A failure that genuinely requires changing a test, dependency declaration, migration, or workflow can be escalated for manual review rather than allowing the agent to broaden its own permissions. Validate More Than a Green Build A candidate patch should first rerun the exact failing test. Only after that test passes should validation expand to the affected module and then the normal CI suite. Shell ./mvnw -Dtest=PaymentServiceTest#shouldRetryTimeout test ./mvnw -pl payment test ./mvnw verify Passing tests alone are not proof of correctness. Automated program repair research documents the overfitting problem, where generated patches satisfy the available test suite without correctly repairing the underlying behavior. Scope validation therefore acts as an additional safeguard. A successful repair should preserve protected files, avoid dependency changes, keep the patch reviewable, and retain the original failing test unchanged. A final guard can verify that tests were not modified during repair: Shell git diff --quiet "$REPAIR_BASE" -- src/test || { echo "Repair modified protected tests" exit 1 } The resulting output should be an evidence package rather than an opaque "fixed" status, but contain the root-cause summary, exact patch, changed-file list, focused-test result, module-test result, and full-CI result. That evidence makes the automated repair reviewable. To ensure a patch does not escape its intended scope, the workflow can run the agent in an isolated Git worktree and enforce explicit path-based allowlists. For example, a CI script might use git worktree add -d /tmp/agent main so the agent’s changes occur in a separate directory. After the agent generates a commit, the pipeline can run git diff --name-only origin/main...HEAD and check that all file paths match the allowlist (for instance, grep -qvE '^(src/payment/|config/)' rejects unauthorized modifications). Protected paths (e.g. .git/) would instantly be blocked by the allowlist. The pipeline then reruns the failing test class in isolation (mvn -Dtest=PaymentServiceTest test) to confirm the issue is fixed, and executes the full test suite. A quick diff or mutation-based check can detect if any test assertions were weakened or disabled. Finally, a canary merge or smoke-test job automatically undoes the change on failure. For instance, a GitHub Actions step might include the following so that any failing health check immediately triggers an automated rollback: SQL - name: Rollback if smoke tests fail if: steps.smoke.outcome == 'failure' run: ./rollback.sh These combined safeguards (worktree isolation, allowlist enforcement, retests, and rollback triggers) help guarantee the AI-generated patch is minimal, correct, and reversible. Keep the Repair Job Less Privileged Than the Code It Examines An autonomous repair workflow processes repository content and may execute code originating from a pull request, so CI permissions need careful separation. GitHub warns that privileged workflow_run and pull_request_target workflows must not blindly check out untrusted pull-request code, and recommends granting GITHUB_TOKEN only the minimum required permissions. A safer pattern is to perform diagnosis and patch generation with read-oriented credentials, store the candidate patch as an artifact, and use a separate trusted step to create a branch or pull request. Workflow artifacts should also be treated cautiously when produced by untrusted jobs. This prevents the repair agent from turning a build failure into a path toward repository write access. Conclusion A useful CI repair agent is not an LLM attached directly to a failing build. It is a constrained repair pipeline that captures structured failure evidence, reproduces the defect, narrows the editable surface, generates a minimal candidate patch, rejects out-of-scope changes, and validates the result from the failing test through the full CI suite. Industrial evidence shows that LLM-assisted repair can be effective in CI settings, but automated program repair research also shows why a green build cannot be treated as sufficient proof. Enforced edit boundaries, isolated execution, layered testing, and least-privilege automation convert model-generated code from an uncontrolled suggestion into a reviewable, evidence-backed repair mechanism.

By Uthej Mopathi CORE

Kafka is a distributed event streaming platform that enables real-time data ingestion, storage, and processing of continuous event streams. To do that, Kafka has a complex internal structure. In this part, I will try to cover an overview of the full internal structure. Overview Kafka uses a publish-subscribe model where producers produce events and consumers consume messages. Kafka producers publish events to the broker, and consumers consume the events from the broker. In the meantime, the broker has a different layer of architecture that allows Kafka to be an event streaming platform. The producer, broker, and consumer-related metadata are managed by the Kafka controller, which is elected by ZooKeeper or KRaft. Key Terms Topic: A topic is a logical name or category where Kafka stores a stream of related events/messages. Think of a topic as a database table for events or a channel where producers publish messages and consumers subscribe to read them. Partition: A partition is a physical subdivision of a Kafka topic that stores messages in an ordered, immutable sequence. It is the basic unit of parallelism and scalability in Kafka. Segment: A segment is a physical file inside a Kafka partition where messages are stored on disk. A partition is not stored as one huge file. Kafka divides it into multiple smaller segment files. Offset: An offset is a unique sequential number assigned to each message within a Kafka partition. It represents the position of a message in that partition. Metadata: Metadata is information about the Kafka cluster, not the actual message data. It describes the structure and state of the Kafka cluster. Controller: The Kafka Controller is a special broker responsible for managing cluster-level operations. In a Kafka cluster, one broker is elected as the controller. ZooKeeper (OLD): ZooKeeper is an external distributed coordination system used by older Kafka versions to manage cluster metadata and controller election. Kraft (Modern): KRaft (Kafka Raft metadata mode) is Kafka's new architecture that removes ZooKeeper and manages metadata internally using the Raft consensus algorithm. Leader: A leader is the broker that owns the primary copy of a partition and handles all read and write operations for that partition. Only one broker can be the leader for a partition at a time. Replica: A replica is a copy of a partition stored on another broker to provide fault tolerance. Internal Details With the Kafka Library At first, download Kafka from this link. After extracting, we will get the following directory structure: Here, each folder has its own functionality. The bin folder contains all the executable scripts. These scripts can be used from the command line to manage Kafka components and perform various tasks, including those listed below: Start serversStop serversTopic managementCluster managementAct as a publisher/consumer Another important folder is the config folder, where we can configure the broker and controller nodes. Initially, this folder contains the default configuration files for the Kafka components. Here we find property files named controller.properties, server.properties, broker.properties, consumer.properties, producer.properties, and some other configuration files. In a Kafka installation, these .properties files are configuration files used by different Kafka components and client applications. However, not all of them are used by the Kafka server itself. We can think of the configuration files in three major categories: Servers - server.properties - broker.properties - controller.propertiesClients - producer.properties - consumer.propertiesTools - other properties file 1. server.properties This is the main configuration file for a Kafka broker. It controls how the Kafka broker runs, including: Broker ID/node identityListener and network configurationLog directoriesNumber of network and I/O threadsDefault number of partitionsReplication settingsRetention policiesMessage size limitsController configuration in KRaft modeCluster-related settings For example: Properties files node.id=1 process.roles=broker,controller listeners=PLAINTEXT://:9092,CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 log.dirs=/tmp/kafka-logs num.partitions=1 log.retention.hours=168 In modern Kafka using KRaft, server.properties can configure both the broker and controller when a combined node is used. 2. broker.properties This is generally used as a separate broker configuration file when you want to maintain configuration for a broker independently. For example, you might create: Plain Text config/ ├── broker.properties ├── controller.properties and start Kafka with a particular configuration: Properties files kafka-server-start.sh config/broker.properties The exact files supplied with Kafka can vary depending on the Kafka version/distribution. broker.properties is not a universal mandatory filename in Kafka; Kafka mainly cares about the properties file you pass to the startup command. 3. controller.properties This is used for configuring a KRaft controller. Kafka's modern architecture doesn't require ZooKeeper. Instead, Kafka uses the KRaft consensus mechanism. A controller is responsible for managing cluster metadata, such as: Which brokers are part of the clusterTopic metadataPartition informationLeader assignmentsCluster configurationController elections A dedicated controller configuration could contain things such as: Properties files process.roles=controller node.id=1 listeners=CONTROLLER://:9093 controller.quorum.voters=1@localhost:9093 controller.listener.names=CONTROLLER A production Kafka cluster may use dedicated controller nodes rather than running broker and controller roles together. 4. producer.properties This is not a broker configuration. It contains configuration for a Kafka Producer client — the application that sends messages to Kafka. For example: Properties files bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer Other producer settings can control: acksretriesbatch.sizelinger.mscompression.typebuffer.memoryenable.idempotencehghg For example: Properties files acks=all retries=10 compression.type=snappy A producer application reads these settings when connecting to Kafka. 5. consumer.properties This is also a client configuration, not a broker configuration. It controls how a Kafka Consumer reads messages. Example: Properties files bootstrap.servers=localhost:9092 group.id=my-consumer-group key.deserializer=org.apache.kafka.common.serialization.StringDeserializer value.deserializer=org.apache.kafka.common.serialization.StringDeserializer auto.offset.reset=earliest Important consumer settings include: bootstrap.serversKafka broker addressesgroup.idConsumer group identitykey.deserializerConverts message key bytesvalue.deserializerConverts message value bytesauto.offset.resetWhat to do when no valid offset existsenable.auto.commitControls automatic offset commitsmax.poll.recordsMaximum records returned per pollfetch.min.bytesMinimum data fetched from broker 6. Other Configuration Files Depending on your Kafka version, you may also find files such as: Plain Text config/ ├── connect-distributed.properties ├── connect-standalone.properties ├── connect-log4j.properties ├── tools-log4j.properties └── ... Some are for Kafka's additional components. For example: Kafka Connect Plain Text connect-distributed.properties connect-standalone.properties These configure Kafka Connect workers. Logging Plain Text connect-log4j.properties tools-log4j.properties These configure logging behavior for particular Kafka tools/components. So, server.properties/broker.properties/controller.properties configure Kafka infrastructure, while producer.properties and consumer.properties configure applications that communicate with Kafka. Step by Step: Test Producer and Consumer From CLI We will discuss the Kafka cluster later, but for now we will check how to run the Kafka controller, broker, and an example producer and consumer from the CLI. 1. Initialize Kafka Storage For a new Kafka installation running in KRaft mode, first generate a cluster ID. If we move to the Kafka bin directory, we can find a file called kafka-storage.sh Shell ./kafka-storage.sh random-uuid Copy the generated UUID. Then format the Kafka storage: Shell ./kafka-storage.sh format \ -t <CLUSTER_ID> \ -c ../config/server.properties For example: Shell ./kafka-storage.sh format \ -t abc123xyz \ -c ../config/server.properties You normally perform this initialization only once for a fresh Kafka storage directory. Don't repeatedly format an existing cluster unless you intentionally want to initialize/reset it. 2. Start the Kafka Server Because we are now inside the bin, specify the configuration file relative to the parent directory: Shell ./kafka-server-start.sh ../config/server.properties Or we can run it in the background: Shell ./kafka-server-start.sh -daemon ../config/server.properties The important relationship is: Plain Text bin/ └── kafka-server-start.sh config/ └── server.properties So the command is Shell ./kafka-server-start.sh ../config/server.properties 3. Verify That Kafka Is Running We can also check the Kafka port: Plain Text lsof -i :9092 4. Create a Topic Still inside the bin directory: Shell ./kafka-topics.sh \ --create \ --topic test-topic \ --bootstrap-server localhost:9092 \ --partitions 1 \ --replication-factor 1 List topics: Shell ./kafka-topics.sh \ --list \ --bootstrap-server localhost:9092 5. Start a Consumer Open another terminal and go to Kafka's bin directory: Shell ./kafka-console-consumer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 The consumer will wait for messages. 6. Start a Producer Open another terminal: Shell ./kafka-console-producer.sh \ --topic test-topic \ --bootstrap-server localhost:9092 Now type: Plain Text Hello Kafka The message should appear in the consumer terminal. This is the introduction to this series. We will discuss more details in the upcoming articles.

This is Part 3 of a series on running sentiment analysis inside Neo4j. Part 1, Running Sentiment Analysis Inside Neo4j With a Java Plugin, used the VADER lexicon via a pure Java UDF – rule-based, fully local, high throughput.Part 2, Wasm Inside Neo4j: Building the Example That Didn't Exist, embedded a Wasm runtime inside a Java UDF using wasmtime-java — sandboxed, still local, lower throughput. Part 3 takes a different approach. Instead of bundling a scoring model inside the plugin JAR, we call TypeSafe AI's Jev from inside a Java UDF. Jev is a System One model – it doesn't generate text. It takes a state (the text to evaluate) and a set of typed questions and returns structured decisions with explicit probability and confidence information. No prompt engineering, no JSON schema coercion, no parsing. The result is callable directly from Cypher: Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; The full source code is available on GitHub. What Jev Gives You That VADER Doesn't VADER gives you a compound score. For "The movie had some good moments but ultimately disappointed," it returns something around -0.3. That's a number, but it doesn't tell you how much to trust it. Jev separates the decision from the uncertainty. For the same sentence, it returns: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score — which is the result of the separate confidence question we asked Jev, rated on a scale from 0 to 2 — is 1.55, landing between "Somewhat confident" and "Very confident." The separate confidence score provides an additional signal about the classification and that information can be actionable in a way that a raw sentiment score isn't. For unambiguous text, Jev is equally unambiguous: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, ... } This structured uncertainty output is the key differentiator of Part 3. If you need to act differently on low-confidence classifications — routing them to human review, flagging them for follow-up, or simply surfacing the uncertainty in a UI — VADER's scores don't provide a model-derived estimate of uncertainty about the classification. Jev returns an explicit confidence assessment alongside the decision. Confidence information should not be confused with accuracy: a model can be highly confident and still be wrong. What We're Building A Neo4j Java UDF that calls the Jev API over HTTPS, passing the input text as the evaluation state and asking two typed questions: a sentiment classification (positive, negative, or neutral) and a confidence score. For non-blank input, the UDF returns a Map<String, Object> with six keys: choice, confidence, score, positive, negative and neutral. For blank input, it returns only choice and confidence as a guard response without calling the API. No Wasm runtime, no bundled lexicon, no native libraries. The plugin has no Wasm runtime, native library, or bundled sentiment lexicon. Its only runtime dependency beyond Neo4j itself is Jackson, which is relocated into a private package namespace to avoid classpath conflicts with Neo4j's own Jackson dependencies. Prerequisites Java 21. OpenJDK 21 or later. Confirm with java -version.Maven 3.9.6. Install manually rather than via Homebrew on Apple Silicon. Confirm with mvn -version.Neo4j Desktop. Download and install Neo4j for Desktop. We're using version 2026.07.0. Update the neo4j.version property in pom.xml to match your installation.A Jev API key. Sign up for access at TypeSafe AI. The key is passed to Neo4j via a JVM property in neo4j.conf — more on this in the deployment section. Getting the Code The full source is on GitHub. Clone the repo and navigate to the neo4j-jev-udf directory: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set jev-udf mv jev-udf ../jev-udf cd ../jev-udf/neo4j-jev-udf The Maven Project The pom.xml has two things worth noting. First, org.neo4j:neo4j is declared as provided scope — Neo4j is already present in the database JVM at runtime. Second, we use maven-shade-plugin to relocate Jackson into a private package namespace. Neo4j ships its own version of Jackson internally, and without relocation, the two can conflict at runtime in ways that produce silent failures. XML <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>jev-neo4j-poc</artifactId> <version>1.0-SNAPSHOT</version> <packaging>jar</packaging> <properties> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>2026.07.0</neo4j.version> </properties> <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.17.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.1</version> <executions> <execution> <phase>package</phase> <goals><goal>shade</goal></goals> <configuration> <artifactSet> <excludes> <exclude>org.neo4j:*</exclude> </excludes> </artifactSet> <relocations> <relocation> <pattern>com.fasterxml.jackson</pattern> <shadedPattern>com.example.jev.shaded.jackson</shadedPattern> </relocation> </relocations> <shadedArtifactAttached>false</shadedArtifactAttached> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> Update the neo4j.version property to match your own Neo4j Desktop installation. The Java UDF The UDF uses Java's built-in HttpClient — no third-party HTTP library needed. The Jev API key is read first from the environment variable TYPESAFE_API_KEY, then from a JVM system property of the same name as a fallback. The fallback matters because Neo4j Desktop runs as a separate process and doesn't inherit your shell's environment variables; we'll set the JVM property in neo4j.conf in the deployment section. Java package com.example; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ArrayNode; import com.fasterxml.jackson.databind.node.ObjectNode; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.UserFunction; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.util.HashMap; import java.util.Map; public class JevUDF { private static final String API_URL = "https://api.typesafe.ai/v1/systemone"; private static final ObjectMapper mapper = new ObjectMapper(); private static final HttpClient client = HttpClient.newHttpClient(); private static String getApiKey() { String key = System.getenv("TYPESAFE_API_KEY"); if (key == null || key.isBlank()) { key = System.getProperty("TYPESAFE_API_KEY"); } if (key == null || key.isBlank()) { throw new RuntimeException("TYPESAFE_API_KEY not set as environment variable or JVM property"); } return key; } @UserFunction("com.example.jev.sentiment") @Description("Classifies text sentiment using Jev (TypeSafe AI). Returns choice, probabilities and confidence.") public Map<String, Object> sentiment(@Name("text") String text) throws Exception { if (text == null || text.isBlank()) { Map<String, Object> empty = new HashMap<>(); empty.put("choice", "neutral"); empty.put("confidence", 0.0); return empty; } String apiKey = getApiKey(); ObjectNode body = mapper.createObjectNode(); body.put("model", "jev-latest"); body.put("state", text); ObjectNode questions = mapper.createObjectNode(); ObjectNode sentiment = mapper.createObjectNode(); sentiment.put("type", "choice"); sentiment.put("instructions", "What is the overall sentiment of this text?"); ObjectNode criteria = mapper.createObjectNode(); criteria.put("positive", "Text expresses positive sentiment"); criteria.put("negative", "Text expresses negative sentiment"); criteria.put("neutral", "Text is neutral or factual"); sentiment.set("criteria", criteria); questions.set("sentiment", sentiment); ObjectNode confidence = mapper.createObjectNode(); confidence.put("type", "score"); confidence.put("instructions", "How confident are you in the sentiment classification?"); ArrayNode levels = mapper.createArrayNode(); levels.add("Not confident"); levels.add("Somewhat confident"); levels.add("Very confident"); confidence.set("criteria", levels); questions.set("confidence", confidence); body.set("questions", questions); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Content-Type", "application/json") .header("Authorization", "Bearer " + apiKey) .POST(HttpRequest.BodyPublishers.ofString(mapper.writeValueAsString(body))) .build(); HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) { throw new RuntimeException("Jev API error " + response.statusCode() + ": " + response.body()); } JsonNode result = mapper.readTree(response.body()); JsonNode answers = result.get("answers"); JsonNode sentimentAnswer = answers.get("sentiment"); JsonNode confidenceAnswer = answers.get("confidence"); Map<String, Object> output = new HashMap<>(); output.put("choice", sentimentAnswer.get("choice").asText()); output.put("confidence", sentimentAnswer.get("confidence").asDouble()); output.put("score", confidenceAnswer.get("score").asDouble()); output.put("positive", sentimentAnswer.get("probabilities").get("positive").asDouble()); output.put("negative", sentimentAnswer.get("probabilities").get("negative").asDouble()); output.put("neutral", sentimentAnswer.get("probabilities").get("neutral").asDouble()); return output; } } A few things worth noting in the Jev request body. The choice question uses a criteria object — a map of option names to descriptions. The score question uses a criteria array — an ordered list of level descriptions. These are different shapes, and mixing them up produces a 422 error from the API. Both questions are answered in a single API call; Jev evaluates them in parallel against the same state. Deploying to Neo4j Build the JAR: Shell mvn -q clean package Stop Neo4j in Desktop, then copy the JAR to your plugins folder: Shell cp target/jev-neo4j-poc-1.0-SNAPSHOT.jar \ ~/Library/Application\ Support/neo4j-desktop/Application/Data/dbmss/<your-dbms-id>/plugins/ On Linux, the plugins folder is typically under $NEO4J_HOME/plugins/. On Windows, the plugins folder is typically under %NEO4J_HOME%\plugins\. Edit neo4j.conf in the conf/ folder of the same dbms directory and add three lines: Plain Text dbms.security.procedures.unrestricted=com.example.jev.* dbms.security.procedures.allowlist=com.example.jev.* server.jvm.additional=-DTYPESAFE_API_KEY=<your-api-key> Two lines are shown in the security configuration. allowlist controls which extension functions are loaded and makes the function callable from Cypher. unrestricted gives matching extensions full database access, bypassing Neo4j's security restrictions and should be used only where necessary. For this example, both settings are configured for the com.example.jev.* namespace; use the narrowest configuration that your Neo4j version and deployment require. The server.jvm.additional line passes the API key to Neo4j's JVM as a system property. For this local Neo4j Desktop example, that's a practical way to make the key available without modifying the UDF code. The Neo4j Desktop process does not automatically receive environment variables exported in your terminal session, so the JVM property is the reliable alternative. Do not commit this value to source control. For production, use your deployment environment's secret-management mechanism and inject the value into the Neo4j process without storing the secret in a tracked configuration file. Restart Neo4j in Desktop, then select Query from the left-hand navigation pane, connect to your instance, and confirm the function is registered: Cypher SHOW FUNCTIONS YIELD name WHERE name STARTS WITH 'com.example' RETURN name; Expected: "com.example.jev.sentiment" Results Positive Sentence Cypher RETURN com.example.jev.sentiment('The movie was great!') AS result; Expected: JSON { "choice": "positive", "confidence": 1.0, "score": 2.0, "positive": 1.0, "negative": 0.0, "neutral": 0.0 } Negative Sentence Cypher RETURN com.example.jev.sentiment('The movie was absolutely terrible.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 2.0, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } Mixed Sentiment Cypher RETURN com.example.jev.sentiment('The movie had some good moments but ultimately disappointed.') AS result; Expected: JSON { "choice": "negative", "confidence": 1.0, "score": 1.55, "positive": 0.0, "negative": 1.0, "neutral": 0.0 } The winning sentiment option has a confidence value of 1.0. But the score drops to 1.55 out of 2 – somewhere between "Somewhat confident" and "Very confident." This is structured uncertainty output, and it's what distinguishes Jev from a rule-based scorer. There are two different quantities here. confidence belongs to the sentiment choice itself: it describes the probability concentration of the classification. score comes from a separate question that explicitly asks Jev how confident it is in that classification. They answer different questions and should not be interpreted as two versions of the same confidence value. Blank-Input Guard Cypher RETURN com.example.jev.sentiment('') AS result; Expected: JSON { "choice": "neutral", "confidence": 0.0 } The blank-input guard returns immediately without making an API call. Here neutral is a guard value rather than a Jev classification: no sentiment analysis is performed for blank or whitespace-only input. Scalability and Limitations Every call to com.example.jev.sentiment() makes an outbound HTTPS request to the Jev API. That shapes where this approach fits and where it doesn't. Latency. Jev responds in 70-500ms per call. For an occasional interactive request — a customer submitting a review, a user looking up a specific node — that latency may be acceptable. For a batch pipeline, if 1,000 calls are made sequentially at a 200ms average, that's over three minutes of API time; concurrent execution changes the wall-clock time but introduces its own rate-limit and resource considerations. Cost. At the pricing available when this article was written, input tokens cost approximately $0.042 per million and output tokens were free. The test requests in this article averaged approximately 381 input tokens per call, giving a per-call cost of around $0.000016 (381 ÷ 1,000,000 × $0.042). Pricing is subject to change; check TypeSafe AI's current pricing before estimating production cost. At this workload size, cost is unlikely to be the primary constraint; latency, throughput, network dependency, and data residency are more significant considerations. At that rate, 1,000 calls would cost roughly $0.016, while one million calls would cost roughly $16, assuming the same average input size. Network dependency. Parts 1 and 2 of this article series are fully local – no network, no external dependency, no data leaving the database. Part 3 introduces all three. The UDF itself runs inside Neo4j, but the inference does not: the text crosses the network boundary to TypeSafe AI, and only the structured result returns to Neo4j. If the Jev API is unreachable, the UDF throws. In a production deployment, that's an operational risk worth planning for: the HttpClient in this example has no explicit timeout configured; in production, set a connect timeout and a per-request timeout to prevent a slow or unresponsive API from blocking Neo4j worker threads indefinitely. A fallback return value and monitoring on the call path are also worth considering. Note also that the UDF makes a synchronous blocking HTTP call inside database query execution; concurrent Cypher invocations increase pressure on Neo4j worker threads, the API's rate limits, and the network. Data residency. Text passed to com.example.jev.sentiment() leaves Neo4j and is sent to TypeSafe AI's API. If your data residency requirements prohibit that, this approach isn't suitable. Review TypeSafe AI's data handling policies before deploying in a regulated environment. Caching as a mitigation. For workloads where the same text is scored repeatedly, results can be stored as node properties on first call and read directly on subsequent queries — skipping the API entirely. That's a natural pattern in Neo4j and largely eliminates the latency and cost concerns for stable datasets. The right use case. This approach is best suited to low-frequency, interactive workloads where explicit uncertainty information matters more than throughput and where data leaving the database is acceptable. For high-volume batch ingestion, Part 1 (the Java VADER UDF) remains a strong fit — local, fast, no network dependency. Part 3 trades local throughput and offline operation for a richer decision output and explicit uncertainty information. Comparing the Three Approaches Part 1: Java VADERPart 2: Wasm/VADERPart 3: JevOutputPolarity scores (float)Polarity scores (float)Typed decision + explicit uncertainty informationLocationFully localFully localNetwork (TypeSafe AI API)Latency< 1ms50-200ms (init)70-500msCostFreeFree~$0.000016/callOfflineYesYesNoExecution boundaryJava UDFWasm runtimeExternal APIUncertainty outputNoNoYesTypical fitHigh-volume batchSandboxed local inferenceInteractive, uncertainty-aware decisions Summary We added a third option to the series: calling TypeSafe AI's Jev from inside a Neo4j Java UDF, returning a typed sentiment decision with explicit uncertainty information directly from Cypher. The implementation has the smallest inference footprint of the three – no Wasm runtime, no bundled lexicon, no native libraries. The JAR contains only the UDF class, a relocated Jackson, and Java's built-in HttpClient. The main configuration consideration is passing the API key to Neo4j's JVM via server.jvm.additional in neo4j.conf, since the Neo4j Desktop process does not automatically receive environment variables exported in your terminal session. The key contribution of Part 3 isn't the mechanics — it's the explicit uncertainty output. For clear-cut sentences, Jev is unambiguous. For genuinely mixed sentiment, it returns both the decision and a structured confidence measure that Cypher can act on directly. That's something neither VADER nor the Wasm implementation provides. A deeper discussion of putting Jev inside Neo4j can be found in the free online book Lessons Learned Putting Jev Inside Neo4j. The full source code is available on GitHub.

By Akmal Chaudhri CORE