DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Tools

Development and programming tools are used to build frameworks, and they can be used for creating, debugging, and maintaining programs — and much more. The resources in this Zone cover topics such as compilers, database management systems, code editors, and other software tools and can help ensure engineers are writing clean code.

icon
Latest Premium Content
Trend Report
Kubernetes in the Enterprise
Kubernetes in the Enterprise
Refcard #366
Advanced Jenkins
Advanced Jenkins
Refcard #378
Apache Kafka Patterns and Anti-Patterns
Apache Kafka Patterns and Anti-Patterns

DZone's Featured Tools Resources

Closing the Drift Loop: An AI Agent That Drafts Terraform Fixes

Closing the Drift Loop: An AI Agent That Drafts Terraform Fixes

By Sidharth Shah
Every Terraform team I’ve worked with has drift detection. A scheduled terraform plan -detailed-exitcode, a Terraform Cloud health assessment, a Spacelift or env0 monitor. Exit code 2 fires, a Slack message lands, and someone opens a ticket. Then the ticket sits. Drift found in minutes gets fixed in days, because the fix needs a human to read the diff, work out why the resource changed, decide whether to codify or revert, write the HCL, and push a PR. Detection is automated. Remediation is still artisanal. The Gap The tempting fix is to auto-apply on drift. Almost nobody does that in production, and they’re right not to. A blind apply on a security group that someone widened during an incident re-breaks the thing they just fixed. Drift means reality and code disagree, and “code wins” is not always the right answer. The middle path is an agent that reads the drift, drafts a remediation as a pull request with its reasoning and its prediction of the post-merge plan, and then stops. The PR is the gate. The real plan runs as a required check on that PR, the same one every human PR gets, and the agent is never in the apply path. I’ve been running this against production stacks in two regions. It turns a days-long ticket into a short review. I’ve published the workflows as a reference implementation in this repository. The linked repo keeps the same shape (detect → remediate → PR gate → apply) with the minimum needed to run. Firefly and env0 sell versions of this idea, and tfdrift’s remediate command and autoai-driftguard are open-source ones. The idea is the easy part. What I want to write down is the set of design decisions that make it safe to leave running, because that’s where the failures live. Three properties make the shape defensible. The agent’s blast radius is a branch: it can’t touch state, cloud resources, or the merge button, so the worst case is a PR a reviewer closes (for an honest agent, at least; more on that later). The existing change process is reused, not bypassed: CODEOWNERS, required reviews, and policy checks run on the agent’s PR exactly as they do on a human’s. And the judgment call stays with a person, but it arrives pre-researched. Two Kinds of Drift When plan says a resource differs from state, there are two directions to go: DirectionMeaningActionRevertThe out-of-band change was wrong; code is the truthNo HCL change; apply the existing configCodifyThe out-of-band change was right; code is staleUpdate HCL to match reality, so apply produces no diff A detector can’t tell them apart. A t3.medium that is now a t3.large is either someone’s forgotten experiment (revert) or last week’s capacity fix that never made it into code (codify). The evidence lives outside Terraform: CloudTrail, tickets, tags, Slack threads. Gathering that evidence and writing down a hypothesis a human can confirm or reject in a minute is exactly what an LLM is good at and a cron job is not. The Workflow Four stages. Only the second one is new. Stage 1: Detection A scheduled workflow runs a plan and captures a machine-readable output. YAML # .github/workflows/drift-detect.yml (abridged) on: schedule: [{ cron: "0 */6 * * *" }] jobs: plan: permissions: { id-token: write, contents: read, actions: write, issues: write } strategy: fail-fast: false matrix: include: # CloudTrail LookupEvents is regional; the agent runs where the resource lives - { stack: network/prod-use1, region: us-east-1 } - { stack: network/prod-use2, region: us-east-2 } steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.DRIFT_READONLY_ROLE } # reads + lock-table write; plan holds the lock aws-region: ${{ matrix.region } - name: Plan (refresh-only) id: plan run: | set +e terraform plan -refresh-only -detailed-exitcode -lock-timeout=5m \ -out=drift.tfplan -input=false code=$?; set -e echo "code=$code" >> "$GITHUB_OUTPUT" [ "$code" -eq 1 ] && exit 1 # plan error: fail loudly terraform show -json drift.tfplan > drift.json - name: Count drifted resources # refresh-only plans report under resource_drift id: count run: | n=$(python3 -c "import json; d=json.load(open('drift.json')); print(sum(1 for r in d.get('resource_drift', []) if r['change']['actions'] != ['no-op']))") echo "n=$n" >> "$GITHUB_OUTPUT" - name: Drift-storm guard # >20 drifted resources is a provider/state problem, not drift if: steps.plan.outputs.code == '2' && steps.count.outputs.n > 20 run: gh issue create --title "[drift] storm in ${{ matrix.stack }" --body "Paging a human." - name: Dispatch remediation if: steps.plan.outputs.code == '2' && steps.count.outputs.n <= 20 run: gh workflow run drift-remediate.yml -f stack=${{ matrix.stack } -f region=${{ matrix.region } -f run_id=${{ github.run_id } Three details do the work. -refresh-only separates real drift from HCL that merged but hasn’t been applied yet. The role is read-only in shape but needs lock-table writes because plan holds a state lock, which is also why -lock-timeout=5m is there: a scheduled plan that collides with an apply should wait, not fail. And the storm guard exists because a provider upgrade looks like drift on every resource; past 20 in one run, the workflow pages a human instead of dispatching the agent. Stage 2: The Remediation Agent The agent has three capabilities: read the plan JSON, read evidence, open a PR. No cloud writes, no terraform apply. YAML # .github/workflows/drift-remediate.yml (abridged) on: workflow_dispatch: inputs: { stack: { required: true, type: string }, region: { required: true, type: string }, run_id: { required: true, type: string } } jobs: remediate: permissions: { contents: write, pull-requests: write, id-token: write, actions: read } steps: - uses: actions/download-artifact@v4 # real workflows sanitize '/' -> '-' in the stack name so this matches the upload step with: { name: drift-${{ inputs.stack }, run-id: ${{ inputs.run_id }, github-token: ${{ secrets.GITHUB_TOKEN } } - uses: aws-actions/configure-aws-credentials@v4 with: { role-to-assume: ${{ vars.DRIFT_READONLY_ROLE }, aws-region: ${{ inputs.region } } - name: Analyze drift # model + AWS read access; no GitHub write token here run: python agent/remediate.py analyze --stack "${{ inputs.stack }" --region "${{ inputs.region }" --plan drift.json --out proposal/ env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }, ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL } } - name: Open PR # bot token only; see below for why not GITHUB_TOKEN run: python agent/remediate.py propose --stack "${{ inputs.stack }" --proposal proposal/ env: { GH_TOKEN: ${{ secrets.DRIFT_BOT_TOKEN } } The token in that last step matters more than it looks. GitHub limits workflow runs for events created with GITHUB_TOKEN, so a PR opened with it gets a pull_request run parked in an approval-required state: no jobs, no checks on the PR, no error. If the plan and policy checks are required, the PR can’t merge until someone with write access clicks “Approve workflows to run.” The tool surface is small, and every tool is read-only except the last: Python TOOLS = [ read_plan_json, # resource_drift entries from drift.json read_hcl, # repo files, capped per file read_tags, # tags on the drifted ARN cloudtrail_lookup, # LookupEvents in the stack's region, last 14 days describe_resource, # aws ec2/rds/iam describe-* search_issues, # GitHub issues/PRs mentioning the resource ID propose_fix, # the ONLY write: stages a branch + PR body under proposal/ ] Two limits on the CloudTrail tool. LookupEvents returns management events only, for one region, for the last 90 days, 50 per page, at two calls a second. That covers “who changed this security group” and not “who read this object”; for older questions, point the tool at CloudTrail Lake or the S3 bucket storing logs. Notice what’s missing: no apply, no cloud modify, no merge. A “revert” decision changes no HCL, because the HCL is already right. Instead, the agent commits a decision record, .drift-decisions/<address>-<fingerprint>.md, with the direction it chose, the evidence, and the plan it expects after merge. That gives the PR a real diff (GitHub refuses PRs with no commits) and puts the decision in git history next to the resource. A revert PR’s plan check exits 2 on purpose, and the gate compares the plan to the record’s claim instead of failing on the exit code. The system prompt sets the contract: You are a Terraform drift analyst. For each drifted resource you will (1) determine whether the out-of-band change should be codified or reverted, (2) cite the evidence that led you there, (3) produce the minimal HCL diff for the codify case, and (4) rate the strength of the evidence as strong, partial, or none. You cannot apply changes. If evidence is insufficient to choose a direction, say so and draft the codify variant anyway so a reviewer can see what accepting the change would look like. That last sentence matters. An agent that won’t act under uncertainty is the same as no agent. Drafting the codify variant costs cents against an engineer’s hour, and it’s reversible. Stage 3: The PR as the Gate Here is the agent’s output on a real drift, a security group that grew an ingress rule outside Terraform: [drift] network/prod-use1: aws_security_group.app_alb drifted out-of-band Recommendation: Codify (evidence: strong) Drift: ingress tcp/8443 from 10.40.0.0/16 not present in state. Evidence: CloudTrail AuthorizeSecurityGroupIngress at 2026-09-08T03:12Z by role IncidentResponder-oncall. Matches issue #431 (“ALB health checks failing from new monitoring subnet”), closed the same day. The CIDR is aws_subnet.monitoring, added in PR #412, which never updated the SG. Change: adds the ingress block to modules/alb/sg.tf, parameterized on var.monitoring_cidr. Expected post-merge plan: no changes. The plan check on this PR confirms or refutes that. If you disagree: close this PR; the next scheduled apply removes the rule. #431 suggests that re-breaks monitoring. The reviewer’s job goes from “investigate” to “verify.” Every line is checkable in under a minute, and the plan check proves the fix converges. The gate is the same set of required checks human PRs pass, plus one step that judges the agent’s claim: YAML - name: Plan on PR run: | set +e; terraform plan -detailed-exitcode -out=pr.tfplan -input=false echo "code=$?" >> "$GITHUB_OUTPUT"; set -e terraform show -json pr.tfplan > pr.json - name: Judge only the claimed resource # codify: the claimed resource is no-op/absent in pr.json # revert: the claimed fingerprint still appears under resource_drift run: python agent/gate.py --plan pr.json - name: Policy check run: conftest test pr.json -p policy/ # Branch protection: 1 approval + Code Owners review. The bot can't approve # its own PR, and "Allow GitHub Actions reviews to count" stays off. Branch protection does the real safety work. The bot can’t approve its own PR, and because its PRs touch platform-owned paths, the platform team’s CODEOWNERS entry has to. Stage 4: Apply (Unchanged) On merge, your existing apply pipeline runs. If the agent was wrong and the reviewer missed it, you have git history, a plan artifact, and a normal revert PR. Same failure mode as any bad merge, which is the point. Guardrails One PR per resource. A stack with six drifted resources gets six PRs, each with its own codify/revert decision. One PR for all six is how a reviewer approves five good changes and one bad one. That rule was the first thing I learned from one of the projects. I staged three drifts at once: a tag added to an S3 bucket (codify), a security-group rule opened to the world (revert), and a subnet deleted out from under state (needs-human). My first gate deadlocked on the first two. The codify check wanted a fully clean plan, the revert check wanted exactly one drifted resource, and each PR’s plan showed the other PR’s drift. Neither could go green. The fix is the “judge only the claimed resource” step above. With it, both PRs passed while the deleted subnet stayed open as an issue. Per-resource independence has to live in the gate, not just in the agent. Idempotent PRs. Before opening a PR, the agent looks for an open one with the same resource address and drift fingerprint, a hash of the before/after attributes with keys sorted so ordering noise doesn’t create a new fingerprint. If the resource drifts again differently, the old PR is stale: the agent closes it with a pointer to the new one. Never auto-remediate delete drift. If a resource exists in state and not in reality, the codify path is deleting HCL, which is the easiest change to get catastrophically wrong. The agent files its evidence as a drift/needs-human issue and stops. Deny-list sensitive types. IAM policies, KMS key policies, and anything under module.security. are evidence-only; the agent files an issue and a human opens the PR. It’s a deny-list on purpose. The human gate already makes the default safe; the list is about blast radius on the day the reviewer is tired, and IAM is where a plausible one-line change grants access to everything. An allow-list would be safer on paper and would spend its first month growing one type at a time while drift queued up. Cap the context. The agent gets the drifted resource’s resource_drift entry, the HCL file that declares it, and evidence for that ARN. Not the repo. Less context means fewer chances to “improve” adjacent code. Log the reasoning. Every PR body carries the evidence trail. When a codify decision turns out wrong three weeks later, you want to know whether the agent misread CloudTrail or CloudTrail was silent. What This Doesn’t Solve It doesn’t prevent drift. If people can write to prod from the console, they will; this makes the consequence cheap, not the cause. It doesn’t handle state corruption or provider upgrades that look like drift on every resource. And it doesn’t remove the reviewer. If your team approves agent PRs without reading them, you’ve built auto-apply with extra steps. Conclusion The industry spent the last few years making drift visible. The next step isn’t making it disappear on its own. It’s making the fix arrive already written, already justified, and already inside the review process you trust. An agent that can only open pull requests is an agent whose worst day is a closed PR. Nothing here is specific to GitHub Actions or AWS. Read-only agent, PR as the only write, existing gates unchanged: that ports to GitLab, Azure DevOps, or any Terraform platform with a VCS-driven apply. More
Replay, Scheduling, and Throttling: Engineering a Self-Service Kafka Streaming Tool at Scale

Replay, Scheduling, and Throttling: Engineering a Self-Service Kafka Streaming Tool at Scale

By Rajesh Gomathinayagam
What started as a recovery tool for streaming pipelines ended up as a full-fledged self-service platform, with scheduled watermark-based incremental ingestion, ad-hoc/one-time data replay, and throughput control built in along the way. The origin was narrow: streaming pipelines fail. A downstream consumer has an outage, a schema changes, a job crashes partway through, and, let's say, Kafka partitions are already gone due to max retention; when that happens, someone needs a way to re-publish a window of historical data without working with the source system to republish. We built a small tool to do exactly that: point it at a backup table (that held raw events), give it a date range, and it would replay that data back to a Kafka topic. As more teams started using this, they began asking questions like: can this be scheduled for a regular stream? Can we configure the TPS (Events Published/Sec)? Those questions are what turned a one-off recovery script into shared infrastructure. This article covers that evolution: the two design problems that mattered most in getting there — turning replay into a first-class, self-service capability, and making throughput control work correctly across a distributed Spark job, the harder of the two — plus the operational lessons that came from running it as infrastructure other teams now depend on. High-Level Architecture At its core, a user calls the Replay Service with a source table, a target Kafka topic, required TPS, and other optional parameters. The Replay Service kicks off a Spark job on serverless EMR through a Scheduler Service (Internal), which hands back a pipeline ID. The user uses that for a status request from the Replay Service, which parses the underlying job's logs and returns status - running, completed with a record count, or failed with a reason. Replay Service – Sequence Diagram Once the job is running, it reads from the source table and streams records to the target Kafka topic, where any number of downstream consumers can subscribe. The Spark job runs under its own AWS IAM role, separate from any team's own credentials, so onboarding a new table or topic is a matter of granting that role read access to the source table and write access to the target topic, not provisioning per-team infrastructure. Replay as a Self-Service Capability Replay ended up covering a few distinct shapes of the same underlying need: One-time replay: An ad hoc, manually triggered backfill for a specific date range, used for onboarding, backfill, or incident recovery.Scheduled replay: A recurring stream from a data lake table that's typically populated in batch, where the data needs to be shared with other systems, customer support, marketing, etc. The capability also allows customers to define a watermark (CDC) column, so subsequent pushes send only incremental data to the topic, improving performance and reducing cost.Auto topic management: Replay can provision its own topic to avoid colliding with the live feed, or, if needed, publish into the same live feed (a user-provided topic) so the data flows through the usual downstream setup. Keeping Throughput Steady Across a Distributed Job Often teams want to dictate TPS (Events Published/Sec) because the downstream system may or may not handle the volume the replay pipeline produces. Also, TPS needs to be deterministic to reliably estimate how long the backfill will take, or simply how long the job will run. The core challenge: a distributed job, a single target rate. A Spark job doesn't execute as one process. It runs as many parallel tasks spread across executor cores, each independently reading and publishing its own slice of data. If you want the whole job to publish at, say, 10,000 events per second in aggregate, you can't just tell every task "publish as fast as you can"; you have to divide that target across however many parallel task slots are actually running, and get every one of them pacing independently toward its share of the total. Here's the shape of the approach: Scala // NOTE: // sourceDataFrame is repartitioned to a multiple of available workers. For e.g. if total // number of table partition is 100 and available workers are 56, the data will be repartitioned // to 112 (RoundingWithCeil(100/56) ) so that every node gets fair share of records to publish val totalExecutorCount = spark.sparkContext.getExecutorMemoryStatus.size - 1 val coresPerExecutorCount = spark.conf.get("spark.executor.cores", "1").toInt val totalWorkers = totalExecutorCount * coresPerExecutorCount val tps = jobConfig.argumentConfig.tps val batchSize = math.max(1, tps / totalWorkers) val eventsPublished: LongAccumulator = spark.sparkContext.longAccumulator("eventsPublished") sourceDataFrame.foreachPartition((partition: Iterator[Row]) => { var partitionCount = 0L partition.grouped(batchSize).foreach { batch => val startTime = System.currentTimeMillis() batch.foreach { record => kafkaSink.value.send(targetTopic, headers, value) partitionCount += 1 } // Pace each batch to a 1-second window: if sending the batch took less // than a second, sleep out the remainder before starting the next one. val elapsed = System.currentTimeMillis() - startTime val sleepTime = 1000 - elapsed if (sleepTime > 0) Thread.sleep(sleepTime) } // flush() blocks until Kafka acknowledges all buffered records for this partition. // Only count after confirmed delivery, so the metric reflects events that // actually landed on the topic, not events that were merely handed to the producer. kafkaSink.value.flush() eventsPublished.add(partitionCount) }) A few things worth unpacking here: totalWorkers counts task slots, not executors. In Spark terms, it's executor count multiplied by spark.executor.cores (set at spark-submit time), the actual number of parallel task slots available, since each core runs its own independent task. Dividing tps by executor count alone would under-throttle any executor running more than one core.The rate limiter is per-task, not global. There's no central coordinator throttling the job as a whole. Each partition's task independently batches its records and sleeps out the remainder of each second. The aggregate rate emerges from every task pacing itself to the same per-slot target simultaneously.This only holds on a stable, dedicated cluster. totalWorkers is calculated once at the start of the job, so the whole scheme assumes the cluster isn't dynamically scaling mid-run. If executors are added or removed while the job is running, the fixed batchSize no longer matches the actual number of active task slots, and the real aggregate rate drifts from the configured target. It also assumes the job has the cluster's resources to itself: running on a shared, transient cluster where other jobs are competing for the same executors introduces exactly the kind of resource contention this mechanism is trying to prevent in the first place. In practice, this means running on a dedicated, statically-sized cluster per job rather than a shared or autoscaling one.This only produces a uniform aggregate rate if the data is repartitioned into a multiple of totalWorkers first. If partitions aren't evenly distributed across the available slots, some cores end up handling more partitions than others. Every task still paces to the same batchSize, but a core doing more partition-passes ends up contributing more to the real aggregate throughput than a core doing fewer. The per-task math is correct, but the cluster-wide result skews unless the upstream repartitioning step guarantees an even split. Getting this repartitioning right was the actual key to making the aggregate number line up with the configured target. The batching-and-sleep loop is straightforward once the data is distributed evenly going in.The count only advances on confirmed delivery. flush() blocks until Kafka acknowledges every buffered record in that partition, and only then does the accumulator increment. Without this, a job that crashes mid-run could report a published count higher than what Kafka actually has, which quietly breaks trust in the metric during incident investigations.Low TPS with many workers hits a floor. math.max(1, tps / totalWorkers) guarantees at least one message per task per second, so a target of, say, 10 TPS spread across 56 task slots still lets every task send 1 msg/sec, overshooting the aggregate target by up to 5.6x. This scheme is tuned for the large-backfill case it was built for; it under-throttles at very low target rates. At a high level: each partition publishes only its allocated number of events per second, then sleeps for the rest of the second. A fixed cluster size, repartitioning data to a multiple of total workers, and a batch size derived from the TPS target are what guarantee the aggregate throughput. Each streaming job also runs on its own dedicated serverless EMR instance, so there's no contention with other jobs for the same resources. A Few Supporting Capabilities A couple of other features round out the platform, worth a brief mention even though they weren't the hardest problems to solve: Custom headers on published messages, letting downstream consumers route or filter messages without inspecting payloads.On-the-fly decryption of encrypted (PII) columns, so sensitive source data can be safely streamed without requiring every downstream consumer to independently implement decryption. Running It as Shared Infrastructure Building the capability was one problem. Running it as infrastructure other teams depend on surfaced a couple of costs that don't show up until the platform has real usage. Dedicated clusters aren't free. The TPS guarantee described above only holds because every job gets its own statically-sized, serverless EMR cluster, with no sharing and no autoscaling mid-run. That isolation is what makes the throughput math work, but it also means the platform's infrastructure cost scales roughly linearly with the number of concurrent jobs, not with total data volume. A dozen small scheduled jobs cost as much in cluster overhead as a few large ones. It's the direct tradeoff for the guarantee: correctness under load, in exchange for giving up the efficiency of a shared pool. Self-service cuts both ways at off-boarding. Making it easy for a team to schedule a recurring replay made it just as easy to forget it exists. A scheduled pipeline is driven by its upstream dependency, so when the upstream stops running, the pipeline stops too. It doesn't keep quietly publishing after a team moves on. But the pipeline's configuration doesn't go away on its own: teams that deprecated the downstream consumer rarely came back to tell us, so the definition just sits there, cluttering inventory and skewing usage metrics even though nothing was actually running. This demands auto de-provisioning based on its last run check (i.e., this pipeline didn't run in the last X days) and clean up periodically. Summary A recovery tool and a self-service streaming platform look similar on the surface: both move data from a table to Kafka. What separates them is whether replay works as a capability that teams can self-serve rather than something an engineer has to trigger by hand, whether throughput holds steady when a dozen teams are running jobs against the same infrastructure at once, and whether the platform can be operated (onboarded, monitored, and off-boarded) without someone having to remember every job by hand. More
AI Coding Agent Guardrails: Finding Gaps in Tool Permissions
AI Coding Agent Guardrails: Finding Gaps in Tool Permissions
By Markus Eisele
When a Coding Agent Fixes the Test Instead of the Bug
When a Coding Agent Fixes the Test Instead of the Bug
By Serhii Romanov
Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop
Searching Kafka Without an Index: Lessons From Building Message Search for Kafdrop
By Shreyamsh Kamate
MQTT, Kafka, NATS, or Zenoh? Choosing Edge-to-Cloud Messaging by Traffic Pattern
MQTT, Kafka, NATS, or Zenoh? Choosing Edge-to-Cloud Messaging by Traffic Pattern

Edge to cloud is not one integration problem. It is four: telemetry going up, control going down, sites syncing sideways, and data reaching people. Each has different physics. Most failed projects picked one technology for all four. Every enterprise with factories, stores, vehicles, hospitals, or devices faces the same architectural question. How does data move between the edge and the data center or cloud, in both directions, reliably, securely, and at acceptable cost? The vendor landscape answers with dozens of overlapping options: MQTT platforms, hyperscaler services, event streaming, event mesh, messaging fabrics, delivery platforms, and protocol proxies. This article structures the option space by traffic pattern instead of by product category, maps the vendor categories onto those patterns, and closes with a decision framework you can apply to your own architecture. The categories here sit alongside the wider integration landscape rather than inside it. For that broader map across event streaming, API, and batch, see the Data Integration Landscape 2026. What Does Edge-to-Cloud Communication Mean? Edge-to-cloud communication is the movement of data between distributed physical locations and centralized computing environments, in both directions, across networks the architect does not control. The word edge hides four very different tiers: The device edge means microcontrollers, sensors, and actuators with kilobytes of memory.The gateway edge means industrial PCs, vehicle computers, and protocol converters.The site edge means a factory, store, hospital, or ship with local compute and its own network boundary.The near edge means regional data centers close to many sites. A technology that fits one tier often fails at another. Edge to cloud splits into four traffic patterns: northbound telemetry, southbound control, site-to-site sync, and outbound delivery to people. The traffic crossing these tiers falls into four patterns: Northbound telemetry moves sensor readings, machine states, transactions, and logs from the edge toward central platforms for analytics, AI, and business systems.Southbound control moves commands, configuration, software updates, and increasingly AI models from central platforms down to machines and gateways.Site-to-site traffic synchronizes data between sites, between a site and a core data center, or between clouds.Outbound delivery pushes live data from central platforms to the applications people use: dashboards, mobile apps, and trading screens. Direction plus consumer type plus constraint determines the right technology. Vendor preference does not. Why Do Network Constraints Decide the Edge-to-Cloud Architecture? The network between edge and cloud is the one component you cannot refactor. Its properties dictate the architecture before any product evaluation starts. Connectivity is unreliable. Ships, vehicles, and remote sites go offline for hours or days. Any viable architecture is offline first: local operation continues, data buffers locally, and synchronization resumes when the link returns. This is why store and forward is the single most important capability on any edge evaluation checklist. Inbound connections are forbidden. Industrial sites, retail networks, and vehicle fleets sit behind NAT, firewalls, and cellular carriers. Security teams do not open inbound ports for messaging. Every serious edge technology therefore uses the same trick: the edge dials out. MQTT clients dial out to brokers. NATS leaf nodes dial out to hub clusters. Azure IoT Operations dials out from the factory to the cloud broker. Different stacks, identical answer, because the constraint is identical. Hardware is constrained and heterogeneous. An 8-bit microcontroller, a fanless gateway PC, and a site server need different footprints. Some technologies scale down to bare microcontrollers. Others assume a Kubernetes cluster. Matching footprint to tier is half the selection process. Footprint is not the only axis. In fixed installations like plants and warehouses, the device edge can stay thin, because the uplink is reliable and most intelligence lives upstream. In the field, agriculture, cranes, rural grid, remote or contested sites, the uplink is wireless and often degraded, saturated, expensive, or absent for days. There, the edge cannot just buffer and forward. It has to run enough local intelligence, pattern matching, image recognition, autonomous decisions, to operate correctly on its own and sync opportunistically when the link returns. That pushes compute onto the device tier for reasons that have nothing to do with protocol size, and it changes the pattern, not just the product. Data residency is now an architecture requirement, not a legal afterthought. European manufacturers increasingly must prove what data leaves the site, country, or jurisdiction. Topic design, routing rules, and retention policies become compliance controls. And one constraint is usually invented rather than measured: latency. Teams assert real-time requirements that the business case does not contain. A large share of edge traffic is perfectly fine as periodic batch over a metered link. Keep that thought. It returns in the site-to-site section. Northbound: How Does Telemetry Get from OT and Devices into the Core? Northbound telemetry is the pattern the industry knows best, and the one with the most settled consensus. At the machine level, OPC UA provides the semantic information model. It defines what a temperature, a batch, or a machine state means. MQTT provides the efficient transport, and Sparkplug adds topic structure, birth and death certificates, and state awareness on top of MQTT. The two standards stopped being rivals. OPC UA Pub/Sub over MQTT is the convergence point: OPC UA semantics carried over MQTT infrastructure. Edge gateways and industrial DataOps products, for example HighByte or Ignition with its MQTT modules, convert brownfield protocols like Modbus and S7 into that stack. The Unified Namespace (UNS) pattern then organizes all site data as a live, hierarchical topic structure on a central broker. I covered UNS design and its trade-offs in depth in Unified Namespace vs. Data Product in IT/OT for Industrial IoT and the broader MQTT market in MQTT Market Trends: Cloud, Unified Namespace, Sparkplug, Kafka Integration, so this section stays short. The real decision is the broker and its operating model. Self-managed platforms carry the northbound load for most industrial estates: HiveMQ and EMQX at the enterprise end, Mosquitto and NanoMQ where a lightweight or embedded footprint matters. They give you deployment control, protocol depth, and edge variants that run on gateways. Managed cloud services remove the operations burden and integrate natively with cloud analytics. The dividing line is usually sovereignty and site autonomy: if the broker must run inside the factory, on the ship, or in a specific jurisdiction, self-managed wins. If all sites have reliable connectivity and the data may leave, managed wins on cost of operations. The hyperscalers themselves tell an asymmetric story. Google retired Cloud IoT Core in 2023 and exited the category. AWS IoT Core remains the steady incumbent for managed device connectivity. Azure went furthest architecturally: Azure IoT Operations runs a distributed MQTT broker on Arc-enabled Kubernetes at the site edge and bridges it to the Event Grid MQTT broker in the cloud, which now supports Sparkplug natively. That is a symmetric edge plus cloud MQTT pair from a single vendor, and it is a new pattern. One hyperscaler retreated, one held, one doubled down. The lesson for architects: managed convenience is real, but so is the risk that a cloud provider deprecates your ingestion layer. The open standards underneath, MQTT and OPC UA, are the insurance policy. Choose products that speak them, and migration remains possible. Southbound and Outbound: How Do Data and Commands Flow Back to Machines and People? Almost all edge content covers ingestion. The return direction is harder, and it is where architectures break, because the return direction has two completely different consumers. Machines need commands and state. Request-response over pub/sub topics handles commands: reboot this gateway, close this valve, change this setpoint. Retained messages and device twins handle state: a device that reconnects must immediately learn its desired configuration without waiting for the next command. Fleet-wide rollout needs staged delivery, and the payloads keep growing. Configuration files became firmware images, and firmware images are becoming AI models. Shipping a quantized model to ten thousand gateways is a southbound data movement problem, not an MLOps footnote. The same fabric increasingly carries traffic for agents at the edge: AI agents that perceive local data, act locally, and coordinate through the messaging layer. NATS and the MQTT platforms are both positioning for exactly this workload. People need the current picture. A plant manager opening a dashboard, an operator reconnecting after a network drop, and a trader loading a screen mid-session all have the same requirement: current state on connect, then live updates. A log is not a user interface. Machines just consume the stream. People do not. This is the last-mile delivery problem, and it has its own vendor category: Lightstreamer, Ably, and PubNub sit between the central platform and many concurrent browsers and mobile apps, handling fan-out, per-client throttling, and entitlement checks. The category is maturing fast. For instance, Lightstreamer's Kafka Connector 2.0, released in July 2026, moved snapshot management into the connector itself, so every reconnecting client receives materialized state followed by the live stream. Delivery to humans deserves a dedicated article. Here it earns its place as the fourth traffic pattern that most edge architectures forget until the first user complains. Site-to-Site: How Do Sites, Data Centers, and Clouds Stay in Sync? The third direction is sideways: keeping data consistent between many sites and one core, or between multiple clouds. For durable event streams, Kafka-style replication is the default. Open source MirrorMaker 2 or vendor-specific cluster linking copies topics between an edge cluster and the central cluster, preserving ordering and history. Apache Kafka is the open-source option; Confluent, Amazon MSK, Aiven, and other solutions are the managed ones. This suits sites large enough to operate a streaming cluster, such as a factory with local analytics that also feeds the global platform. For thousands of small sites, the streaming cluster per site model collapses under its own operational weight. This is where NATS is architecturally distinct rather than just another broker. The entire server is a single small binary with no external dependencies. Each store, vehicle, or clinic runs a leaf node that dials out to the central cluster, keeps serving local applications when the WAN drops, and uses JetStream mirroring to synchronize persisted streams when connectivity returns. Retail chains run this pattern across thousands of locations. The same design serves both the intermittent case, hours or days offline with a bulk sync on reconnect, and the scheduled-batch case where sites replicate on a timer rather than continuously. Its own maintainers warn against one trap: do not stretch a single stream cluster across unstable edge links, because the cluster's consensus breaks every time the network does. Instead, let each site persist its data locally and keep serving through an outage, then mirror to the center asynchronously when the link returns. For enterprises whose sites run applications rather than device fleets, an event mesh covers the same ground at the application layer. Solace links its brokers across data centers and clouds so an event published anywhere reaches any interested subscriber, with topic routing and governance built in. It is the natural fit when the endpoints are SAP, mainframes, and microservices rather than sensors. And a large share of site-to-site traffic does not need streaming, even if streaming is often the stronger long-term architecture. Nightly file synchronization, batch uploads over metered links, and change data capture from edge databases remain perfectly valid when there is no real-time requirement. This is especially true when the existing OT and IT systems on both ends are batch-oriented anyway, and no business case justifies the cost and effort of making them real time. Real time is a requirement, not a default. Stream when you need fresh data, or when you build greenfield and carry no legacy batch systems to preserve. An architecture that streams everything pays streaming costs for batch problems. Which Category Serves Which Edge to Cloud Pattern? Seven vendor categories map onto the four patterns. No single category covers all of them, and messaging fabrics, event mesh, and protocol proxies deliberately span several. No single vendor covers every direction. Seven categories divide the field, and the real picture is that some sit in one pattern while others deliberately span several. MQTT platforms (EMQX, HiveMQ, Mosquitto, NanoMQ) serve both northbound telemetry and southbound control, because MQTT is bidirectional by design. Hyperscaler services (AWS IoT Core, Azure IoT Operations) do the same for the managed-cloud case, ingesting telemetry and pushing desired state through device twins. Event streaming (Apache Kafka, Confluent, Aiven, and more in the data streaming landscape) owns site-to-site replication. Last-mile delivery(Ably, Lightstreamer, PubNub) owns outbound delivery to people, and it is the only pattern with a category built solely for it. Three categories span multiple patterns. An event mesh (Solace) routes site-to-site and out to applications. Messaging fabrics (NATS, Zenoh) carry northbound, southbound, and site-to-site in a single fabric. Protocol proxies (Zilla) translate across all four, replacing several specialized components with one layer. The underlying distinction is worth stating plainly: a protocol standard is not a deployment architecture. MQTT standardizes the wire, which is why so many products speak it and your devices stay portable. NATS and Zenoh standardize the topology, which reduces moving parts and increases commitment. Zenoh, born from DDS research and adopted by automotive initiatives including General Motors' uProtocol and by the ROS 2 robotics ecosystem, goes furthest: it routes named data along the shortest path between producer and consumer, aiming to make the cloud versus edge placement debate irrelevant. These spanning options are not equivalent. A fabric like NATS or Zenoh reduces moving parts and unifies security by being one technology end to end, but that reach is also the lock-in: the more it carries, the more expensive it is to leave. A protocol proxy like Zilla does the opposite. It spans the patterns by translating between existing technologies, which keeps your exit options open at the cost of another component in the path. The right choice depends on how many patterns you actually need and how much lock-in you can accept, not on which vendor markets hardest. How to Choose: Five Questions That Pick Your Edge-to-Cloud Architecture Product evaluations start too late. Answer five questions first, and the shortlist mostly writes itself. Five-question decision framework for choosing an edge-to-cloud architecture, from consumer type to data residency Who consumes the data, machines or people? People need current state on connect, then live updates, which means a last-mile delivery layer over the backbone. Machines just consume the stream.Does this flow actually need real time? Measure before you architect. If a fifteen-minute batch satisfies the business case, use file sync or change data capture. Stream when you need fresh data, because streaming infrastructure you did not need is the most expensive kind.What is the most constrained node in the path? A microcontroller means MQTT or Zenoh, the two with clients small enough for the device itself, not a JVM. A gateway opens the field to lightweight NATS or full brokers, and a site server adds event meshes or a small streaming cluster.How many sites, and can each one operate a cluster? A few large sites with local analytics justify edge Kafka clusters with MirrorMaker or cluster linking. Hundreds or thousands of small sites demand hub-and-spoke: NATS leaf nodes or hierarchical MQTT bridges, with store and forward at every spoke. Enterprises whose sites run applications rather than devices fit an event mesh.Where must the data live? If residency or sovereignty rules apply, the broker runs on site or in jurisdiction, self-managed, and routing policy becomes a compliance control. If data may leave and connectivity is solid, managed cloud services win on operations cost. Two checks apply on almost every branch, at every step. The edge must dial out and never accept inbound connections to the site. And local operation must survive offline, with a buffer sized deliberately, not discovered in the first outage. Five questions, four patterns, one map. The technologies then sort themselves into it. Moving Data Is Not Automating the Business Every pattern and category in this article solves the same problem: moving data between edge, data center, and cloud. That movement is necessary but not sufficient. When telemetry arrives, something must decide what happens next: trigger the maintenance workflow, roll back the failed firmware deployment, replicate the dataset, open the ticket, retrain the model. Deciding what happens next is workflow orchestration. It spans the same edge-to-cloud boundary as the data movement beneath it. Data integration moves the events. Orchestration, with the process intelligence to see how those flows actually run, decides what the business does about them. And increasingly, agentic AI acts on those decisions at the edge. These are not three separate topics. They are one stack, and edge to cloud is where they meet the physical world. Architectures that move data and improvise everything above it end up with the automation logic scattered across broker plugins, cron jobs, and cloud functions. That is the integration spaghetti this whole discipline set out to remove. Edge to cloud is not one problem. It is four. Architect for the pattern, not the product.

By Kai Wähner DZone Core CORE
New York City Taxi Trips: Putting Neo4j and DuckDB to Work
New York City Taxi Trips: Putting Neo4j and DuckDB to Work

In the previous article, we built a custom Neo4j procedure that embeds DuckDB directly into the Neo4j JVM and makes it callable from Cypher. We showed how it works and ran a few examples to prove the concept. Now it's time to put it to work on something real. In this article, we'll use that procedure on the NYC Taxi and Limousine Commission (TLC) trip record data for January 2024. The dataset covers nearly three million yellow cab trips and more than fifty thousand green cab trips across 265 taxi zones. Our goal is to aggregate those trips into a traversable graph in Neo4j — zones as nodes, routes as relationships — and then query it in ways that neither engine could handle as naturally on its own. The full source code is available on GitHub. Getting the Code Before starting, ensure that you've worked through the previous article and have the duckdb.query procedure built and installed in Neo4j. Then get this project: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set neo4j-duckdb-taxi mv neo4j-duckdb-taxi ../neo4j-duckdb-taxi cd ../neo4j-duckdb-taxi The Dataset The NYC TLC publishes monthly trip records in Parquet format. Each row represents a single trip and includes the pickup and dropoff zone identifiers, fare amount, trip distance, and duration. Alongside the trip files, the TLC provides a zone lookup table that maps each of the 265 LocationID values to a borough and neighborhood name and a shapefile that defines the zone boundaries. We'll download all of this programmatically in a Jupyter notebook. Once the files are in place, DuckDB can query the Parquet files directly — no import step needed. Python RAW_DIR = os.path.expanduser("~/neo4j-duckdb-taxi/data/raw") for filename in ["yellow_tripdata_2024-01.parquet", "green_tripdata_2024-01.parquet"]: path = os.path.join(RAW_DIR, filename) size_mb = os.path.getsize(path) / 1024 / 1024 result = duckdb.sql(f"SELECT COUNT(*) AS row_count FROM read_parquet('{path}')").fetchone() print(f"{filename}") print(f" Size: {size_mb:.1f} MB") print(f" Row count: {result[0]:,}") print() DuckDB reads the Parquet file directly and returns the row count without requiring a separate import into a database. The same loop covers the green cab file. This is DuckDB in its natural habitat — fast analytical queries over columnar data without a server or a schema to maintain. Preparing the Zone Nodes Before loading any trips, we need the zone nodes in Neo4j. Each node represents one of the 265 taxi zones and carries the zone name, borough, and service zone. Separately, the first notebook computes a centroid for each zone from the shapefile, which uses a projected coordinate system (NY State Plane in feet). To get latitude and longitude values suitable for a web map, we reproject each centroid and save the results to a JSON file for the dashboard. Cypher CALL duckdb.query( 'SELECT LocationID, Zone, Borough, service_zone FROM read_csv_auto("{ZONE_CSV}") ORDER BY LocationID' ) YIELD row RETURN row.LocationID AS locationId, row.Zone AS zone, row.Borough AS borough, row.service_zone AS serviceZone Cypher MERGE (z:Zone {locationId: $locationId}) SET z.zone = $zone, z.borough = $borough, z.serviceZone = $serviceZone The first statement reads the zone lookup CSV through duckdb.query and the second merges one node per row. With all 265 zones in place, we can move on to the trips. Aggregating Trips With DuckDB The raw trip files are too granular for graph storage. We don't want three million individual trip nodes — we want one relationship per route that summarizes all the trips that ran that route in January. DuckDB handles the aggregation. For yellow trips, we work from a 100,000-row sample to keep the graph build fast; green trips are small enough to aggregate in full. SQL WITH sample AS ( SELECT * FROM read_parquet("{YELLOW_PARQUET}") USING SAMPLE 100000 ROWS ) SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes, SUM(passenger_count) AS total_passengers FROM sample WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID The sample gives us one row per route with counts and averages already computed. We run the same query for green cab trips, substituting the green Parquet file and the appropriate timestamp column names, without sampling since the green dataset is much smaller. Writing to Neo4j With duckdb.query Now comes the part the previous article set up: using duckdb.query to run the aggregation from Cypher and write the results into Neo4j as relationships. The procedure executes a DuckDB SQL query and streams the results back as maps, which Cypher can process directly. Cypher CALL duckdb.query( 'WITH sample AS ( SELECT * FROM read_parquet("{YELLOW_PARQUET}") USING SAMPLE 100000 ROWS ) SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes, SUM(passenger_count) AS total_passengers FROM sample WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID' ) YIELD row RETURN row.PULocationID AS pu, row.DOLocationID AS do, row.trip_count AS tripCount, row.avg_distance AS avgDistance, row.avg_fare AS avgFare, row.avg_duration_minutes AS avgDurationMinutes, row.total_passengers AS totalPassengers Cypher UNWIND $batch AS row MATCH (pu:Zone {locationId: row.puId}) MATCH (do:Zone {locationId: row.doId}) MERGE (pu)-[t:YELLOW_TRIP]->(do) SET t.count = row.count, t.avgDistance = row.avgDistance, t.avgFare = row.avgFare, t.avgDurationMinutes = row.avgDurationMinutes, t.totalPassengers = row.totalPassengers DuckDB reads the Parquet file, aggregates the rows, and returns them through a single CALL. The notebook maps the returned columns into the relationship properties, then passes the results to the second statement in batches of 500. Cypher matches the zone nodes and writes the relationships. The two engines are cooperating directly — DuckDB does what it is good at, Neo4j does what it is good at, and the boundary between them is the duckdb.query call. We run the same pattern for green cab trips, replacing the SQL source with the green Parquet file (using the lpep timestamp columns and no sampling) and the relationship type with GREEN_TRIP. The Graph Schema After both loads complete, the graph looks like this: 265 Zone nodes, each with locationId, zone, borough and serviceZoneYELLOW_TRIP relationships connecting pickup zones to dropoff zones, with count, avgDistance, avgFare, avgDurationMinutes and totalPassengersGREEN_TRIP relationships with the same properties Each route represented in the yellow sample or the full green dataset is now a first-class edge in the graph. We can traverse it, filter on it, and aggregate across it using Cypher. Querying the Graph The following examples are orchestrated from the accompanying notebook. Where a query produces zone IDs or route pairs, the notebook passes those results into the next statement. 1. Graph Filter -> DuckDB Aggregate Cypher first fetches all Manhattan zone IDs from the graph. DuckDB then reads the raw Parquet, filters to those zones, and aggregates trip stats. Cypher MATCH (z:Zone {borough: 'Manhattan'}) RETURN z.locationId AS locationId ORDER BY z.locationId Cypher CALL duckdb.query( 'SELECT PULocationID AS location_id, COUNT(*) AS trip_count, ROUND(AVG(trip_distance), 2) AS avg_distance_miles, ROUND(AVG(fare_amount), 2) AS avg_fare_usd FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IN ({id_list}) GROUP BY PULocationID ORDER BY trip_count DESC LIMIT 10' ) YIELD row RETURN row.location_id AS locationId, row.trip_count AS tripCount, row.avg_distance_miles AS avgDistanceMiles, row.avg_fare_usd AS avgFareUsd 2. Graph Routing -> DuckDB Enrichment Cypher finds the top 10 busiest routes from the aggregated relationships already in the graph. DuckDB then goes back to the complete raw Parquet to compute average trip duration for those exact zone pairs. Because the graph was built from a sample, the duration values here draw on the full dataset rather than the sampled one. Cypher MATCH (pu:Zone)-[t:YELLOW_TRIP]->(do:Zone) RETURN pu.locationId AS puId, pu.zone AS puZone, do.locationId AS doId, do.zone AS doZone, t.count AS tripCount ORDER BY t.count DESC LIMIT 10 Cypher CALL duckdb.query( 'WITH pairs(pu_id, do_id) AS ( VALUES {values} ) SELECT t.PULocationID AS pu_id, t.DOLocationID AS do_id, ROUND(AVG((epoch_ms(tpep_dropoff_datetime) - epoch_ms(tpep_pickup_datetime)) / 60000.0), 2) AS avg_duration_minutes FROM read_parquet("{YELLOW_PARQUET}") t JOIN pairs p ON t.PULocationID = p.pu_id AND t.DOLocationID = p.do_id GROUP BY t.PULocationID, t.DOLocationID ORDER BY avg_duration_minutes DESC' ) YIELD row RETURN row.pu_id AS puId, row.do_id AS doId, row.avg_duration_minutes AS avgDurationMinutes 3. DuckDB Scan -> Graph Context DuckDB reads the raw Parquet to find the top 5 pickup zones by volume. Cypher then pulls each zone's name, borough, and connectivity from the graph. Cypher CALL duckdb.query( 'SELECT PULocationID AS location_id, COUNT(*) AS trip_count FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IS NOT NULL GROUP BY PULocationID ORDER BY trip_count DESC LIMIT 5' ) YIELD row RETURN row.location_id AS locationId, row.trip_count AS tripCount Cypher UNWIND $ids AS locId MATCH (z:Zone {locationId: locId}) OPTIONAL MATCH (z)-[:YELLOW_TRIP|GREEN_TRIP]->(neighbor:Zone) RETURN z.locationId AS locationId, z.zone AS zone, z.borough AS borough, count(DISTINCT neighbor) AS connectedZones ORDER BY connectedZones DESC These queries traverse the graph rather than scanning rows. The aggregation is already baked into each relationship, so the work Neo4j does here is pattern matching and property lookup — exactly what a graph engine is built for. 4. Yellow vs. Green Comparison Across Shared Routes Cypher CALL duckdb.query( 'WITH yellow AS ( SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG(trip_distance), 2) AS avg_distance FROM read_parquet("{YELLOW_PARQUET}") WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID ), green AS ( SELECT PULocationID, DOLocationID, COUNT(*) AS trip_count, ROUND(AVG(fare_amount), 2) AS avg_fare, ROUND(AVG(trip_distance), 2) AS avg_distance FROM read_parquet("{GREEN_PARQUET}") WHERE PULocationID IS NOT NULL AND DOLocationID IS NOT NULL GROUP BY PULocationID, DOLocationID ) SELECT y.PULocationID AS pu_id, y.DOLocationID AS do_id, y.trip_count AS yellow_trips, y.avg_fare AS yellow_avg_fare, y.avg_distance AS yellow_avg_distance, g.trip_count AS green_trips, g.avg_fare AS green_avg_fare, g.avg_distance AS green_avg_distance FROM yellow y JOIN green g ON y.PULocationID = g.PULocationID AND y.DOLocationID = g.DOLocationID ORDER BY y.trip_count DESC LIMIT 10' ) YIELD row RETURN row.pu_id AS puId, row.do_id AS doId, row.yellow_trips AS yellowTrips, row.yellow_avg_fare AS yellowAvgFare, row.yellow_avg_distance AS yellowAvgDistance, row.green_trips AS greenTrips, row.green_avg_fare AS greenAvgFare, row.green_avg_distance AS greenAvgDistance This is the cross-file join that sets query 4 apart: a single CALL reads both Parquet files inside DuckDB, joins them on shared zone pairs, and returns a side-by-side fare and distance comparison for the top ten routes. The two files never leave DuckDB — Neo4j just receives the result rows. 5. Graph and Raw File Counts Side by Side Cypher MATCH (z:Zone) WITH count(z) AS zones MATCH ()-[y:YELLOW_TRIP]->() WITH zones, count(y) AS yellow MATCH ()-[g:GREEN_TRIP]->() RETURN zones, yellow, count(g) AS green Counts what's in the graph: zone nodes and aggregated route relationships. Cypher CALL duckdb.query( 'SELECT (SELECT COUNT(*) FROM read_parquet("{YELLOW_PARQUET}")) AS yellow_rows, (SELECT COUNT(*) FROM read_parquet("{GREEN_PARQUET}")) AS green_rows' ) YIELD row RETURN row.yellow_rows AS yellowRows, row.green_rows AS greenRows This uses duckdb.query to count the raw rows in the Parquet files. Running both of these queries together shows the relationship between the summarized graph and the source data it was built from. The graph holds 265 zones, 7,018 yellow route relationships and 5,314 green route relationships, while the underlying Parquet files contain nearly three million yellow trips and more than fifty thousand green trips. What the Data Shows A few things stand out from the query results. The busiest pickup zone in the full yellow dataset is JFK Airport (zone 132) with over 145,000 departures, and it connects to 236 distinct drop-off zones — a wider reach than Midtown Center (zone 161), which connects to 120 zones despite nearly as many departures. The Upper East Side dominates the top routes: the four busiest zone-to-zone pairs are all within or between Upper East Side North and South, with the short Upper East Side South to Upper East Side North run clocking the highest volume at 779 sampled trips. Among the shared routes examined, green cabs generally show higher average fares than yellow cabs for comparable trip distances — the Upper East Side North to Upper East Side North intra-zone route, for example, shows a green average fare of $8.51 versus $6.86 for yellow over a nearly identical distance. The Dashboard We built a small interactive dashboard to explore the graph visually, as shown in Figure 1. Figure 1. Dashboard It connects to Neo4j and runs Cypher queries live against the data we loaded. The page opens with three summary tiles showing the zone, yellow route, and green route counts. Below them are a map of pickup zone activity, a bar chart of the top pickup zones, a scatter plot of average fare against average distance, a grouped bar chart comparing yellow and green average fares on shared routes, and a table of the busiest routes. A sidebar lets us choose Yellow, Green, or Both and set how many routes and zones to show. Changing either control re-runs the relevant Cypher queries and re-renders the views in place. The map plots each zone as a circle marker, sized by trip volume and colored by borough. Hovering a marker shows the zone name, borough, and trip count. Because the zone centroids were computed and saved during the data preparation step, no additional spatial processing is needed at query time. The dashboard code is included in the GitHub repository alongside the notebooks. What Each Engine Contributes It's worth being explicit about the division of labor, because it illustrates why the two-engine approach is worth the setup cost. TaskEngineRead and aggregate Parquet filesDuckDBColumnar filtering and GROUP BYDuckDBStore zone nodes and route relationshipsNeo4jRoute traversal and relationship pattern matchingNeo4jBridging the two from Cypherduckdb.query procedure DuckDB never touches the graph. Neo4j never touches the Parquet files. The procedure is the seam between them, and it keeps the aggregation inside Cypher rather than in a separate ETL script. Summary We started with raw trip records in Parquet format and ended with a traversable graph that answers routing and fare questions in Cypher. DuckDB handled the heavy lifting on the file side — reading columnar data, filtering and aggregating millions of rows. Neo4j handled the graph side — storing zones and routes, serving pattern-match queries, feeding a live dashboard. The duckdb.query procedure connected the two engines without a separate ETL service. One CALL aggregated the data, and Cypher wrote the relationships. The full source code is available on GitHub.

By Akmal Chaudhri DZone Core CORE
AI Agents Need an Identity Plane: Rethinking Enterprise Authorization for Autonomous Software
AI Agents Need an Identity Plane: Rethinking Enterprise Authorization for Autonomous Software

Here's the security problem in enterprise AI I'd lose sleep over, and it isn't a model giving a wrong answer. It's a model connected to an identity with enough authority to act on that answer. Think about what an agent really is once it can query a database, call an infrastructure API, update a ticket, and kick off a workflow. It isn't a chat window anymore. It's software that executes things, with credentials, on its own. Most companies still secure it like a web app with a fancy text box. I think that gap is where a lot of future incidents will start. Who Actually Did That? Picture an incident response agent. A pager goes off. The agent pulls telemetry, queries a database, opens a ticket, calls a cloud API, and suggests a fix. Nobody clicked anything between the second step and the fifth. Now try to answer the question an investigator will ask later. Which principal did each of those things? Not who typed the prompt. Whose credential touched the database, with what scope, and who signed off? Microsoft's own guidance is refreshingly blunt about this. It calls out "identity ambiguity": agents running on shared secrets, service principals, or mixed "on behalf of user" setups, so nobody can say for certain which principal is accountable. It also points out that logs tend to record the chat response and miss the tool actions and downstream authorization decisions that matter. You can read it in Least privilege for AI agents with Microsoft Entra Agent ID, updated July 15, 2026. Authentication tells you the agent is who it says it is. It doesn't tell you whether a given action should be allowed. Those are different questions, and agents force us to ask the second one constantly. An App Has Paths. An Agent Makes Them Up. With a traditional app, the logic is fixed when you design it. A user acts, the app calls an API, the API touches data. You can map the paths on a whiteboard. An agent picks its path at runtime. It chooses tools, chains calls, reads results that shape the next call, and sometimes crosses system boundaries while doing it. Microsoft describes agents as planning and executing multistep workflows across systems, sometimes with little or no human involvement. Software like that needs what every other actor in the enterprise already has: an identity of its own, an accountable owner, scoped permissions, a lifecycle, and an off switch. That's the direction Microsoft's Entra Agent ID is heading, treating agents as a distinct identity type instead of recycling ordinary application identities. Here's an overview of agent identities in Microsoft Entra. I'll say the quiet part loudly: the LLM should never be your security boundary. A model can be talked into things, confused, or hijacked by a sentence buried in a retrieved document. Put the boundary where you can enforce it deterministically, around the identity, the tools, and the systems downstream. Authenticated and Still Dangerous An agent can log in flawlessly and still be wildly overprivileged. Take the incident agent again. It needs to read logs, metrics, tickets, and configuration, and it needs to write exactly one thing: an incident ticket. It has no business deleting infrastructure, editing IAM roles, creating admins, exporting customer records, or touching security policy. Microsoft's pattern makes that concrete. Keep the read role for gathering evidence separate from the write role for tickets. Allowlist only "create or update ticket." Block delete and admin actions. Ask a human before any bulk update. The Number Nobody Calculates Permission creep isn't new. Agents just make it sneakier. Microsoft names it "permission creep and excessive aggregate privileges." A team grants a broad role to get a pilot moving and never narrows it later. Or they stack lots of small roles without separating tasks, and the sum ends up far too powerful. Nobody adds it up, so everybody underestimates what the agent can really do. Every grant looks fine on its own. CRM read. Mailbox access. Ticketing. A cloud role. A deployment pipeline. Together they describe a high-privilege operator who never sat through onboarding. So stop asking "is this permission safe?" and start asking "what can this agent do, end to end, with everything combined?" Microsoft's first implementation step is exactly that: inventory your agents and tool integrations and work out their aggregate effective permissions. Every Tool Is a Door The dangerous failures here are boring ones. Microsoft warns that if an agent can call any tool it can find, one prompt injection or workflow bug can set off export, delete, or privilege changes, or chain actions across services. The industry is starting to name this properly. The OWASP GenAI Security Project announced its Top 10 for Agentic Applications on December 9, 2025, covering risks like goal hijack, tool misuse, and identity and privilege abuse. Its ASI03 entry is about agents running on borrowed, overscoped, or long-lived credentials, so a compromised agent inherits far more access than its job needs. Teleport has a readable breakdown. The rule I'd tattoo on every design review: a tool isn't trusted just because an agent asked for it. Microsoft's advice is to deny unreviewed tools, plugins, integrations, and cross-tenant or guest paths by default, and to check authorization again at the downstream system on every call instead of leaning on the orchestrator. Revalidate identity, role, and scope at each hop so one weak integration can't talk its way around the rest. There's a catch, and Microsoft admits it. All of this depends on downstream systems actually rechecking authorization. Where they don't, the risk walks right back in. "On Behalf Of" Is a Boundary, Not a Label When an agent acts as itself, the audit trail is simple. When it acts for Alice, things get interesting. Does it inherit everything Alice can do? It shouldn't. Alice may hold broad rights that one task will never need. The answer is explicit "on behalf of" context, narrower scopes, and a named owner for the agent. Microsoft recommends logging the "on behalf of" user wherever it applies. OWASP's ASI03 describes the flip side: delegated credentials, cached sessions, and implicit identity let an attacker use the agent to do something its owner never intended. Login Isn't the Finish Line Classic IAM asks, "can this identity reach this resource?" Agents need a second question: "should this exact action happen right now?" Microsoft's recommended pattern pairs a stable, lifecycle-managed agent identity with time-limited privileges. That means just-in-time entitlements: temporary role activation, short-lived tokens or approvals, so elevated access exists only for the length of one workflow. For remediation agents, it suggests JIT elevation or approval gates, scoping to specific resource groups, denying subscription-wide rights, keeping rollbacks ready, and requiring step-up controls for destructive actions. Here's the shape of the decision I'd want in front of every tool call: Python def authorize(agent_id, on_behalf_of, tool, action, resource, context): scope = effective_scope(agent_id, on_behalf_of) # computed, never assumed if tool not in allowlist(agent_id): return deny("tool not allowlisted") if action not in scope.allowed_actions(resource): return deny("action outside effective scope") if is_high_impact(action): # delete, export, privilege change return require_approval_or_jit(agent_id, action, resource) return allow(ttl_seconds=300, correlation_id=context.correlation_id) Don't focus on the syntax. Notice where the decision lives: outside the model, per action, with a short lifetime on whatever it grants. A Transcript Isn't an Audit Trail Microsoft spells out what a useful record holds: agent identity, role, effective scope, action, resource, correlation ID, and the "on behalf of" user when there is one. Its suggested success metrics include the share of agent actions that emit those fields and the share of workflows you can trace end to end across the orchestrator, the tools, and the downstream systems. If your SIEM can tell you an agent "answered a question" but can't tell you which API it called under which scope, you have chat history. You don't have telemetry. How Fast Can You Pull the Plug? Most agent programs I read about spend pages on capabilities and one sentence on shutdown. Microsoft treats the stop path as a first-class concern. Test revocation by disabling the agent, rotating credentials, invalidating tokens, and removing stale permissions. Its success factors include mean time to revoke or disable an agent identity, token invalidation included, and the share of environments with a tested kill switch and recovery procedure. Put mean time to revoke on the same dashboard as mean time to detect. If shutting down an agent takes a ticket, an approval, and two days, its autonomy was never really bounded. It was just unobserved. Legacy Makes Everything Harder Agents can't safely automate what an enterprise doesn't understand. Legacy estates love to hide undocumented business logic, creaky APIs, fragmented data, inconsistent identity models, and thin observability. Point an agent at that, and you're handing a fast, tireless actor a map nobody ever finished drawing. That's why identity, integration, observability and governance now sit beside model choice in modernization work, and why vendors in that space are repositioning around agents. On March 3, 2026, Sonata Software said it had been recognized as a Microsoft Frontier Partner, describing an approach that pairs AI agents with human engineering across cloud and AI platforms, AI business solutions and security. Sonata Software announced that it had earned the 2026–2027 Microsoft AI Business Solutions Inner Circle award, marking the company’s sixth Inner Circle recognition, with Microsoft's Niels Jensen saying in the release, "Inner Circle partners represent our top selling partners worldwide" (Sonata's press release). Both items are vendor announcements, so read them as a signal about where the services market is going, not proof of any particular control. The shift is what matters here: modernization is being pitched as preparation for governed autonomy, and governance starts with identity. What I'd Do on Monday Inventory every agent and tool integration, guest and cross-tenant paths included, and work out each agent's aggregate effective permissions.Give every production agent its own identity and a named owner. Retire shared credentials.Swap broad roles for task-scoped ones, and split read from write.Allowlist tools and actions. Put delete, export, and privilege changes behind approval or JIT elevation.Enforce authorization at the downstream system, not just the orchestrator.Emit identity, role, scope, action, resource, and correlation ID for every action, and send it to your SIEM.Run a revocation drill and write down how long it took. The Short Version Nobody's trying to stop companies from giving agents autonomy. The point is to make that autonomy bounded, attributable, and revocable. If an agent can behave like an employee, an application, and an automation system all at once, the real question is why we still secure it as if it were only an application.

By Igboanugo David Ugochukwu DZone Core CORE
Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense
Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense

Every Kafka tutorial covers idempotency the same way: "Store the event ID. Before processing, check if it exists. If it does, skip. Done." That advice is correct. It is also incomplete in a way that will surface only under load — and only in production, at the moment you least want a surprise. Here is what actually happens inside a multi-replica consumer group when two delivery attempts arrive at the same instant. The Pattern You Know A Kafka topic configured with acks=all and enable.idempotence=true on the producer side still delivers messages at least once. Network partitions, broker leader elections, retry storms — any of these can cause a consumer to receive the same message twice. The standard fix is to deduplicate on the consumer side using a stable event ID included in every message payload. The implementation looks like this: Java // Application-layer idempotency check if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event — skipping | eventId: {}", eventId); return false; } // Process and record processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) // ... .build()); Under normal operation, this is correct. One consumer thread, one message, one check, one save. The second delivery hits the existsByEventId guard and is dropped cleanly. The problem is that a Kafka consumer group under load does not serialize delivery at the event level. The Race Condition You Don't See in the Tutorial Kafka's partition assignment guarantees that only one consumer in a group receives messages from a given partition at a given time. What it does not guarantee is that the same event ID cannot arrive via two different partitions — for example, during a producer retry that sends to a different partition — or that two messages with the same semantics cannot appear if upstream logic emits duplicates before the broker sees them. More concretely: if you run two replicas of your consumer service (and in Kubernetes, you will), two pods can receive the same event in the same wall-clock window. The sequence looks like this: Plain Text T=0 Pod A receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=0 Pod B receives event e1, calls existsByEventId("e1") → false (not yet in DB) T=1 Pod A passes the guard, begins processing T=1 Pod B passes the guard, begins processing T=2 Pod A saves ProcessedEvent("e1") → INSERT succeeds T=2 Pod B saves ProcessedEvent("e1") → ??? Both pods read false from the application-layer check before either has written. Both proceed. Without a database-level guardrail, both complete, and you have sent the welcome email — or charged the card, or incremented the counter — twice. The Fix: The Database as the Serialization Point The application-layer check is an optimization — it avoids a write attempt for the obvious duplicate case. It is not, and cannot be, a correctness guarantee, because it is not atomic with the write. The correctness guarantee has to live at the layer that is atomic: the database. The schema includes a UNIQUE constraint on event_id: SQL CREATE TABLE processed_events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), event_id VARCHAR(255) NOT NULL UNIQUE, -- the guarantee is here event_type VARCHAR(100) NOT NULL, processed_at TIMESTAMP NOT NULL DEFAULT now(), topic VARCHAR(255) NOT NULL, partition_number INT NOT NULL, offset_number BIGINT NOT NULL ); That UNIQUE constraint is enforced at the storage engine level — not at the application level, not at the ORM level. No two rows with the same event_id can be committed, regardless of how many concurrent writers attempt it. The database serializes the race. What changes in the application is how you handle the loser of that race: Java @Transactional public boolean sendWelcomeNotification(UserCreatedEvent event, ...) { String eventId = event.getEventId().toString(); // Fast path: skip if already recorded (optimization, not guarantee) if (processedEventRepository.existsByEventId(eventId)) { log.warn("Duplicate event detected — skipping | eventId: {}", eventId); notificationMetricsService.incrementDuplicate(); return false; } try { // ... process the event ... processedEventRepository.save(ProcessedEvent.builder() .eventId(eventId) .eventType("USER_CREATED") .processedAt(LocalDateTime.now()) .topic(topic) .partitionNumber(partition) .offsetNumber(offset) .build()); return true; } catch (DataIntegrityViolationException ex) { // Two threads passed the application check simultaneously. // The DB constraint caught one of them. This is correct behavior. log.warn("Duplicate event caught at DB level — skipping | eventId: {} | userId: {}", eventId, event.getId()); notificationMetricsService.incrementDuplicate(); return false; } } The DataIntegrityViolationException is Spring's translation of a constraint violation from any supported database (PostgreSQL, MySQL, Oracle). Catching it and suppressing it is not an error condition — it is the correct outcome of a race that the DB resolved correctly. Note that the exception is caught in the service layer, not in the Kafka listener. If it propagated up to the listener, Spring Kafka's retry mechanism would re-deliver the event, triggering more unnecessary attempts. Catching it at the service boundary terminates the race cleanly. Two Layers, One Correct Answer The full pattern has two distinct roles: LayerRoleWhat it stopsApplication check (existsByEventId)OptimizationThe non-concurrent duplicate: the obvious second delivery hours or days laterDB UNIQUE constraint + DataIntegrityViolationExceptionCorrectness guaranteeThe concurrent duplicate: two threads, two replicas, same event, same instant Removing either layer has a cost. Without the application check, every retry hits a write attempt that will fail, generating unnecessary DB load and constraint violations in your logs. Without the DB constraint, the application check is a promise with no enforcement — and distributed systems do not honor promises. The right posture is to treat the application check as a performance optimization and the database constraint as the correctness invariant. What This Looks Like Under Fault Injection I verified this pattern against deliberate duplicate delivery in a Kafka consumer running two Spring Boot replicas. Injecting two simultaneous deliveries of the same eventId produced exactly one SENT notification log and one SKIPPED log (the DataIntegrityViolationException path). No double side effects, no exceptions visible to the retry mechanism, no compensating logic required. The database caught the race that the application check could not. Key Takeaways Application-layer idempotency checks are not atomic with the writes they guard. Under concurrent delivery, both threads can pass the check before either writes.The database's UNIQUE constraint is the actual serialization point. It is enforced at a level that the application cannot race.DataIntegrityViolationException is a success path, not an error path. It means the DB caught a concurrent duplicate — suppress it, do not propagate it.You need both layers. The application check eliminates unnecessary write attempts. The DB constraint enforces the invariant. Neither is sufficient alone. If your Kafka consumer only has the application check, you have an idempotency aspiration, not an idempotency guarantee. The race condition described here may never surface on a single-replica dev environment. It will surface on a multi-replica production deployment, under the load conditions that make that second replica necessary in the first place. The implementation described in this article is part of a production-grade event-driven microservices platform built with Spring Boot 4, Java 21, and Kafka. The full source is available at github.com/Rummy43/ai-microservices-platform.

By Ramesh Yara
Google Cloud KMS vs. AWS KMS: Java Encryption Guide
Google Cloud KMS vs. AWS KMS: Java Encryption Guide

Every system that stores sensitive data needs keys that never leave a hardened boundary, an audit trail of every use, and rotation that doesn't mean re-encrypting petabytes. On Google Cloud, that's Cloud KMS; on AWS, it's AWS KMS. Both store symmetric and asymmetric keys, both use HSMs, and both are built for envelope encryption. But AWS KMS vs. Google Cloud KMS comes down to the choices that show up in your code: how keys are named, what rotation means, where authorization lives, and whether the service hands you data keys. This article walks through Cloud KMS in Java and then maps it onto AWS KMS. The Cloud KMS Resource Model Cloud KMS organizes keys in a four-level hierarchy under a project: Location. Where the key material lives: a region such as us-east1, a multi-region such as us, or global. Key material stays in the location you pick for its whole lifecycle.Key ring. A named group of keys in one location. It exists for organization and IAM inheritance.CryptoKey. The key you reference in code. It has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, and a few more) and a rotation schedule.CryptoKeyVersion. The actual key material. A key has one or more versions, and for symmetric keys one of them is the primary, the version that new encryptions use. A symmetric key with three versions: new data uses version 3, data written under version 2 still decrypts, and version 1 is on its way out. A key is identified only by its full resource name, projects/my-project/locations/us-east1/keyRings/payments/cryptoKeys/card-data, optionally followed by /cryptoKeyVersions/2. There are no aliases. Nothing can move after it's created: the location is fixed when you create the key ring, and key rings themselves can't be deleted. Each key version also has a protection level: SOFTWARE: keys are protected by Google's software crypto (BoringCrypto, FIPS 140-3 Level 1).HSM: keys live in Cloud HSM, which is FIPS 140-2 Level 3 validated. HSM_SINGLE_TENANT gives you a dedicated HSM.EXTERNAL and EXTERNAL_VPC: Cloud EKM keeps the key material in a supported external key manager that you operate, reached over the internet or a VPC. Google never holds the key. Creating a Key That Rotates Itself The Java client is KeyManagementServiceClient. It uses Application Default Credentials, so on GKE, Cloud Run, or GCE, the workload's service account authenticates with no keys in the code. This creates a key ring and an HSM-backed symmetric key that rotates every 90 days: Java try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { KeyRing ring = kms.createKeyRing( LocationName.of("my-project", "us-east1"), "payments", KeyRing.getDefaultInstance()); CryptoKey key = CryptoKey.newBuilder() .setPurpose(CryptoKeyPurpose.ENCRYPT_DECRYPT) .setVersionTemplate(CryptoKeyVersionTemplate.newBuilder() .setAlgorithm(CryptoKeyVersionAlgorithm.GOOGLE_SYMMETRIC_ENCRYPTION) .setProtectionLevel(ProtectionLevel.HSM)) .setRotationPeriod(Duration.newBuilder().setSeconds(90L * 24 * 3600)) // every 90 days .setNextRotationTime(Timestamp.newBuilder() .setSeconds(Instant.now().plusSeconds(90L * 24 * 3600).getEpochSecond())) .build(); kms.createCryptoKey(KeyRingName.parse(ring.getName()), "card-data", key); } The rotation period can be anywhere from one day to 100 years. When it fires, Cloud KMS creates a new version and makes it primary. Earlier versions stay enabled, so old ciphertext still decrypts, and nothing is re-encrypted. To rotate by hand, create a version with CreateCryptoKeyVersion and promote it with UpdateCryptoKeyPrimaryVersion. Automatic rotation applies only to symmetric keys; asymmetric keys are rotated manually. The AWS KMS key rotation process differs in how key material is managed, as described below. Envelope Encryption KMS keys aren't meant to encrypt your data directly. A single Encrypt call accepts at most 64 KiB of plaintext (8 KiB of plaintext plus AAD for HSM keys), and every call is a network round trip. The standard pattern is envelope encryption, also used with AWS KMS envelope encryption: encrypt the data locally with a data encryption key (DEK), then have KMS encrypt (wrap) the DEK with a key encryption key (KEK) that never leaves the service. You store the wrapped DEK next to the ciphertext. Envelope encryption: The payload never goes to KMS; only the 32-byte DEK does. Cloud KMS has no data-key API, so you generate the DEK yourself. Google's guidance is to generate a new DEK every time you write data, never share one across users, and prefer Tink, which packages this whole flow. Doing it by hand shows what's happening: Java CryptoKeyName kek = CryptoKeyName.of("my-project", "us-east1", "payments", "card-data"); ByteString aad = ByteString.copyFromUtf8("tenant=acme"); // must be supplied again to decrypt try (KeyManagementServiceClient kms = KeyManagementServiceClient.create()) { // 1. Generate a fresh DEK locally. Cloud KMS has no GenerateDataKey call. KeyGenerator gen = KeyGenerator.getInstance("AES"); gen.init(256); SecretKey dek = gen.generateKey(); // ... encrypt the payload locally with dek (AES-GCM), then discard dek ... // 2. Wrap the DEK with the KEK, with CRC32C checks against corruption in transit. ByteString dekBytes = ByteString.copyFrom(dek.getEncoded()); EncryptResponse wrapped = kms.encrypt(EncryptRequest.newBuilder() .setName(kek.toString()) .setPlaintext(dekBytes) .setPlaintextCrc32C(crc32c(dekBytes)) .setAdditionalAuthenticatedData(aad) .setAdditionalAuthenticatedDataCrc32C(crc32c(aad)) .build()); if (!wrapped.getVerifiedPlaintextCrc32C() || wrapped.getCiphertextCrc32C().getValue() != crc32c(wrapped.getCiphertext()).getValue()) { throw new IllegalStateException("integrity check failed"); } ByteString wrappedDek = wrapped.getCiphertext(); // store next to the encrypted payload String usedVersion = wrapped.getName(); // the primary version that wrapped it // 3. Later: unwrap. Pass the key, not the version; Cloud KMS picks the version. ByteString unwrapped = kms.decrypt(DecryptRequest.newBuilder() .setName(kek.toString()) .setCiphertext(wrappedDek) .setAdditionalAuthenticatedData(aad) .build()) .getPlaintext(); SecretKey restored = new SecretKeySpec(unwrapped.toByteArray(), "AES"); } static Int64Value crc32c(ByteString data) { return Int64Value.of(Hashing.crc32c().hashBytes(data.toByteArray()).padToLong()); } Encrypt uses the key's current primary version. Decrypt also takes the key, not a version: the ciphertext records which version produced it, so rotation is invisible to readers. The optional CRC32C fields let the server confirm it received your plaintext intact, and let you confirm the ciphertext you got back. The AAD (tenant=acme) is bound to the ciphertext, so one tenant's wrapped DEK can't be swapped into another tenant's record. Disabling, destroying, and deleting Lifecycle operations act on versions, not keys. You disable a version (ENABLED to DISABLED) to stop it from being used, and re-enable it later. There's no switch that disables a whole key; you disable each version. Destroying a version is deliberately slow. DestroyCryptoKeyVersion moves it to DESTROY_SCHEDULED for 30 days by default. The duration is set when the key is created, and an organization policy can enforce a minimum. During that window, RestoreCryptoKeyVersion brings it back in the DISABLED state. After the window, the material is gone and data encrypted by that version can't be recovered. Destroyed versions are no longer billed. Cloud KMS has also added permanent deletion: you can delete key versions and then the key itself, though a deleted key's name can't be reused. Access control is just IAM Cloud KMS has no policy language of its own. Access is ordinary IAM, granted at the organization, folder, project, key ring, or key level and inherited downward. The predefined roles split duties cleanly: roles/cloudkms.admin manages keys but can't encrypt or decrypt.roles/cloudkms.cryptoKeyEncrypterDecrypter, cryptoKeyEncrypter, and cryptoKeyDecrypter can use keys but not manage them.roles/cloudkms.signerVerifier and publicKeyViewer cover asymmetric keys. The operator who rotates keys never sees plaintext, and the service that decrypts can't destroy its key. Grant usage roles on the specific key, not the project. Two more controls round this out. Cloud Audit Logs record admin operations (create, rotate, destroy) automatically, but Encrypt and Decrypt are Data Access logs, which are off by default. If you want a record of every key use, enable them for Cloud KMS. Key Access Justifications, used with Cloud EKM keys, attaches a reason code to every Google request for your key, and your external key manager can refuse it. Bringing your own key To import key material, you create an import job with a wrapping method, such as RSA_OAEP_3072_SHA256_AES_256 or one of the newer post-quantum HPKE methods. The job gives you a public key, and it expires after three days. You wrap your key material with that public key and import it as a new version of a key. For a key that must only ever hold imported material, set importOnly, which means rotation is manual: import new material and make it primary. If you destroy an imported version, you can re-import the same material later. AWS KMS for comparison AWS KMS covers the same ground with a flatter, version-less model: Keys, not versions. A KMS key has a key ID (a UUID), an ARN, and optional aliases such as alias/card-data. There's no hierarchy above it beyond the account and Region.Rotation is hidden. Automatic rotation is off by default for customer managed keys. When enabled, it runs every 365 days by default, configurable from 90 to 2,560 days, and RotateKeyOnDemand allows up to 25 extra rotations per key. The key ID never changes; AWS keeps old material internally and picks it on decrypt.All keys in HSMs. Keys in the default key store live in HSMs validated at FIPS 140-3 Level 3. There's no software tier. CloudHSM key stores and external key stores (XKS) cover the dedicated and hold-your-own-key cases.A data-key API. GenerateDataKey returns a fresh DEK in plaintext along with a copy wrapped under your key, so step 1 and step 2 above are one call.Encryption context. The AWS equivalent of AAD is a map of key-value pairs. Unlike Cloud KMS AAD, it's recorded in CloudTrail and can be used in policy conditions such as kms:EncryptionContext:tenant.Three authorization layers. Every key has exactly one key policy, a resource policy that supports Deny and conditions. IAM policies work only if the key policy enables them, and grants add temporary, programmatic permissions on top.Whole-key lifecycle. DisableKey and ScheduleKeyDeletion act on the whole key. The deletion window is 7 to 30 days, and CancelKeyDeletion backs it out.Replication. Multi-Region keys share the same key ID and material across Regions, and ReplicateKey copies one to a new Region. You choose this at creation; a single-Region key can't be converted. Here's the same envelope flow on AWS: Java Map<String, String> context = Map.of("tenant", "acme"); // logged in CloudTrail, usable in policy try (KmsClient kms = KmsClient.create()) { GenerateDataKeyResponse dataKey = kms.generateDataKey(GenerateDataKeyRequest.builder() .keyId("alias/card-data") .keySpec(DataKeySpec.AES_256) .encryptionContext(context) .build()); byte[] dek = dataKey.plaintext().asByteArray(); // encrypt locally, then discard SdkBytes wrappedDek = dataKey.ciphertextBlob(); // store next to the payload byte[] unwrapped = kms.decrypt(DecryptRequest.builder() .keyId("alias/card-data") // optional for symmetric keys .ciphertextBlob(wrappedDek) .encryptionContext(context) .build()) .plaintext().asByteArray(); } KeyId is optional on Decrypt for symmetric keys because the ciphertext blob identifies the key, but AWS recommends passing it so you don't decrypt with a key you didn't expect. Cloud KMS requires the key name on every Decrypt. Side by side ConceptGoogle Cloud KMSAWS KMSKey identifierResource name projects/…/cryptoKeys/k, no aliasesKey ID, ARN, aliasesVersionsFirst-class CryptoKeyVersion with its own stateHidden behind the key IDProtectionSOFTWARE, HSM (FIPS 140-2 L3), single-tenant HSM, EXTERNAL, EXTERNAL_VPCAll HSM-backed (FIPS 140-3 L3), plus CloudHSM and external key storesMax plaintext per call64 KiB (8 KiB with AAD for HSM keys)4,096 bytesData keysGenerate locally (or with Tink)GenerateDataKeyAuthenticated dataAAD bytes, not loggedEncryption context map, logged and usable in policiesAuto-rotation1 day to 100 years90 to 2,560 days, off by default, plus 25 on-demandDisablePer versionWhole keyDestroyPer version, 30 days scheduled by default, restorableWhole key, 7–30 days, cancellableAccess controlIAM roles on the hierarchy, admin and usage splitKey policy (with Deny) + IAM + grantsUsage loggingData Access audit logs, off by defaultCloudTrail management eventsMulti-regionChoose a multi-region location for the key ringMulti-Region keys, ReplicateKeyBYOKImport job, expires in 3 daysGetParametersForImport, token valid 24 hoursThroughput quota (default)Token-based per region; about 60,000 software operations a minute10,000 to 100,000 symmetric operations a second, by Region What the AWS KMS vs. Google Cloud KMS encryption comparison means in practice: Cloud KMS is simpler to reason about for access. It's one IAM model with roles that separate managing keys from using them. AWS is more expressive: a key policy can Deny, and conditions on encryption context can tie a key's use to a tenant. That power comes with the risk of a key policy that locks out its own administrators.Versions are explicit on Google Cloud. You can disable or destroy just the key material that was compromised and leave the rest working. On AWS you disable or delete the whole key, or rotate and keep the old material.AWS logs every use by default. On Google Cloud, enable Data Access logs for Cloud KMS on day one, or you won't have a record of who decrypted what.Plan for quotas. AWS gives each account and Region a large shared pool of cryptographic operations. Cloud KMS's default is far smaller per project and region, so cache unwrapped DEKs for hot reads where your threat model allows, and request a quota increase before launch. Both quotas are adjustable.Both want envelope encryption. Neither service is built to encrypt bulk data directly. On Google Cloud, Tink makes the local half trivial. On AWS, the AWS Encryption SDK plays the same role. Conclusion Google Cloud KMS treats keys as resources in your project hierarchy: versions are visible, protection level is chosen per key, and IAM is the whole authorization story. That's easy to audit and gives fine-grained control over key material, as long as you turn on Data Access logs and size your quotas. AWS KMS hides versions and puts authorization in a per-key policy language that's more expressive, especially with encryption context, and its data-key API saves a step. These differences are worth considering when reviewing AWS KMS best practices alongside Google Cloud KMS key management. On both clouds, the envelope pattern is the common ground: generate a DEK, wrap it, store it next to the data, and let KMS guard only the key that guards everything else.

By Sandeep Pal
How to Keep AI Agents From Going Rogue Using Docker Sandbox
How to Keep AI Agents From Going Rogue Using Docker Sandbox

An AI agent is an "autonomous software program that uses artificial intelligence to make a plan, execute tools, and complete an action." Autonomy (through AI) plus tools is what makes an agent powerful, and these tools can range from a simple calculator to querying a sophisticated database. While autonomy makes things easier for humans, it only opens a big security hole. Because an agent can do almost anything, without proper guardrails, it can cause production outages. For example, in June 2025, a Replit agent deleted an entire database belonging to a SaaStr founder even though it was explicitly told to request permissions. This is why it is important to put proper guardrails around your agent. Guardrails can take the form of rules that prevent destructive tool calls or allow calls only to a specific list of domains. Another name for such a thing is "sandbox." A sandbox is a "security mechanism for separating running programs, usually in an effort to mitigate system failures and/or software vulnerabilities from spreading." There are multiple agent sandbox offerings. For example: Docker SandboxLangsmith SandboxOpen Sandbox Docker Sandbox Today, we will discuss Docker Sandbox. It has gained a lot of traction recently and plays an important role in Docker's AI strategy. Simply put, Docker Sandbox helps you run agents in an isolated environment. These sandboxes can run on your laptop or be hosted in the cloud. To understand how these sandboxes restrict agents, let's look at the architecture. First, there is a host machine. As we discussed earlier, this can be either your host or cloud infrastructure. When you launch a Docker Sandbox, it is primarily made up of two components. An AI agentDocker engine private to its own Sandbox. Each sandbox has its own Docker engine. There are multiple ways to implement a Sandbox architecture, but the industry is pivoting towards microVMs. This is primarily because of: Low resource overheadFaster bootup timesStrong isolationsReduced attack surfaceHyper-scaling A popular microVM tech stack involves Firecracker, Cloud hypervisors, and Kata containers. Now, coming to the communications, let's take a simple example. Imagine there is a hotel booking agent that searches and reserves hotels for customers. When a request comes in: Step 1: The agent makes a tool call to search for hotels. Step 2: The tool call leaves the micro VM to be verified against the Network policy. PowerShell sbx policy allow network api.search.foo.com sbx policy allow network api.search.bar.com Step 3: Above is an example of how you can define network policies for a Sandbox. In this case, we are allowing the agent to search under the foo domain and restrict its access to bar. Step 4: If the request comes for the foo domain, the network policy allows it to be forwarded to the particular website that serves the request. Let's look at how to create these Sandboxes on your laptop. Step 1: Installation PowerShell brew trust docker/tap brew install docker/tap/sbx Step 2: Sign in to your Docker account. PowerShell sbx login If you don't have one, you can create a free Docker account. Step 3: Run the sbx command. PowerShell sbx The first time you run it, you will see the following text: PowerShell Waiting for authentication... Signed in as sirivarma. Daemon started (PID: 58801, socket: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/sandboxd.sock) Logs: /Users/svegiraju/Library/Application Support/com.docker.sandboxes/sandboxes/sandboxd/daemon.log Select a default network policy for your sandboxes: 1. Open — All network traffic allowed, no restrictions. ❯ 2. Balanced — Default deny, with common dev sites allowed. 3. Locked Down — All network traffic blocked unless you allow it. Use ↑/↓ or 1–3 to navigate, Enter to confirm, Esc to cancel. Network policy set to "Balanced". Default deny, with common dev sites allowed. I picked balanced so things like GitHub are on the allow list and everything else is denied by default. When you run sbx from a project folder, the sandbox has read/write permissions to everything in that folder. Step 4: Running sbx ls in a different shell will give the following output. PowerShell sbx ls PowerShell SANDBOX AGENT STATUS PORTS WORKSPACE docker-agent-svegiraju docker-agent running /Users/svegiraju Step 5: Running sbx policy ls will show the current list of allow and deny rules with the "Balanced" path we have chosen. PowerShell sbx policy ls PowerShell NAME TYPE DECISION RESOURCES default-ai-services network allow **.chatgpt.com:443 **.oaistatic.com:443 **.oaiusercontent.com:443 **.openai.com:443 api.anthropic.com:443 api.perplexity.ai:443 cdn.openaimerge.com:443 chatgpt.com:443 gemini.google.com:443 generativelanguage.googleapis.com:443 models.dev:443 nanoclaw.dev:443 platform.claude.com:443 play.googleapis.com:443 statsig.anthropic.com:443 default-package-managers network allow **.bun.sh:443 **.gradle.org:443 **.packagist.org:443 **.yarnpkg.com:443 apache.org:443 Step 6: In order to stop and remove the sandbox, issue the following commands PowerShell sbx stop my-sandbox sbx rm my-sandbox Step 7: Of course, you may want to diagnose issues within the sandbox or while running it. This is when we will use the "diagnose" command. PowerShell sbx diagnose Conclusion What we discussed so far shows why sandboxes matter and how to create one with Docker Sandbox. It is a free, open-source offering from Docker for individual use. In the upcoming blogs, we will discuss Cloud Sandboxes (running sandboxes in the cloud infrastructure instead of locally) and more complex configurations like Credentials, GPU passthrough, and environment files. All in all, Sandbox is an interesting project from Docker. Go try it out.

By Siri Varma Vegiraju DZone Core CORE
Claude Code Superpowers: The 5-Tool Stack That Ends Limits and Context Amnesia
Claude Code Superpowers: The 5-Tool Stack That Ends Limits and Context Amnesia

Claude Code has changed how many people write software. You type in the terminal, ask for a change, and the tool reads files, edits code, runs commands, and reports back. For a lot of daily work, it feels fast and useful. Still, 2 problems show up again and again. First, you hit usage limits. A long coding session can stop in the middle of a task. You wait, switch accounts, or break the flow. Second, every new session starts blank. The tool does not remember the bug you fixed yesterday, the library versions you prefer, or the style rules you care about. You spend the first part of the day explaining the same project again. Those two issues waste time. They also make the tool feel less like a partner and more like a one-shot helper. A small set of real tools fixes most of that pain. This article walks through 5 of them in plain language. OmniRoute helps when you hit limits by switching to other models. claude-mem keeps project memory across sessions. Headroom shrinks big tool outputs so you use fewer tokens. Claude Code Setup (from Anthropic) looks at your repo and suggests useful helpers. Task Observer watches how you correct the AI and slowly learns your style. These tools do not give you free top models forever. Free options help as a backup. Paid models stay the better choice for hard or important work. The goal is simple: fewer stops, less repeated explanation, and more time spent shipping code. 5 Tools (Along With Links) OmniRoute – runs on your computer and sends requests to many AI providers. When one is full or slow, it can switch to another (including free ones). Link: https://github.com/diegosouzapw/OmniRouteclaude-mem – remembers what happened in past sessions and brings the useful parts back next time. It keeps data on your machine by default. Link: https://github.com/thedotmack/claude-memHeadroom – cuts down big tool outputs and repeated text before they go to the model. The full text stays on your computer if needed later. Link: https://github.com/headroomlabs-ai/headroomClaude Code Setup – official tool from Anthropic. It looks at your code and suggests the best hooks, skills, and helpers for your project. It only suggests; it does not change files by itself. Link: https://claude.com/plugins/claude-code-setup (also in the official marketplace claude-plugins-official)Task Observer – watches when you fix the AI's code or give better instructions. Later, it can suggest updates so the AI matches your style more. Link: https://github.com/rebelytics/one-skill-to-rule-them-all Start With This First Put a clear CLAUDE.md file in your project. Claude Code already uses project notes and some memory. The tools below work better when that file is good. 1. Handle Limits With OmniRoute OmniRoute sits on your computer. Claude Code talks to it instead of only to one company. OmniRoute can try a paid model first, then switch to free ones if needed. It does not give you free Claude models. It just lets you keep going when the main path is blocked. Free models change, have limits, and are not always as good. Use them when you are stuck. Use paid models for important work. 2. Remember the Project With claude-mem Normal sessions forget everything when they end. claude-mem notes useful things while you work, shortens them, saves them on your computer, and brings the right parts back next time. You can also search past notes. 3. Use Fewer Tokens With Headroom Long sessions and memory make prompts bigger. Tool outputs and logs eat tokens fast. Headroom shrinks the noisy parts before they reach the model. The full text stays on your computer if the model needs it later. How much you save depends on the work, but coding sessions usually get a clear drop. 4. Set Up the Project With Claude Code Setup A messy project makes the AI guess wrong more often. Missing helpers, unclear structure, or no useful hooks lead to extra steps and more mistakes. Claude Code Setup is the official Anthropic plugin that helps fix that. It does not rewrite your code. It only looks and suggests. What It Does Claude Code Setup scans your repository. It checks things like: File and folder layoutLanguage and framework signals (for example, package.json, language files, common config files)Dependencies and patterns already in the project From that scan, it recommends the most useful automations in five areas: MCP servers – connections to outside tools, docs, browsers, databases, and similar servicesSkills – ready-made instruction packs for common tasksHooks - actions that run at key moments (for example, after an edit, before a commit, or when a file changes)Subagents – smaller specialist agents (for example, security review or performance review)Slash commands – short commands you can type for repeated workflows It usually surfaces the top one or two ideas in each area. If you ask for one type only (for example, "what MCP servers should I use?"), it can give a longer list. How to Use It Install it from the official marketplace inside Claude Code: Plain Text /plugin install claude-code-setup@claude-plugins-official Or open the plugin browser and pick it from the official list. Link: https://claude.com/plugins/claude-code-setup. Then talk to it in plain words, for example: "recommend automations for this project""help me set up Claude Code""what hooks should I use?""what MCP servers fit this repo?" It stays in read-only mode. It does not change files by itself. You review the suggestions and decide what to add. Why It Helps Without this step, the agent often works with a half-ready environment. It may miss useful tools, skip helpful hooks, or invent steps that a good setup would handle automatically. After you apply the best suggestions, the same agent makes fewer wrong guesses and needs less hand-holding. Simple Example You open a React + Node project. Claude Code Setup might suggest: A browser or Playwright MCP for frontend checksA docs or package MCP for library lookupHooks that format or lint after editsA security-focused subagent if it sees auth codeSlash commands for common test or PR steps You pick the ones that fit, install them, and keep going. Where It Sits in the Stack Run it early. A clear CLAUDE.md plus a tuned project setup gives the other tools (memory, compression, routing, style learning) a cleaner base to work on. In short: Claude Code Setup is the official way to audit your repo and get practical, project-specific recommendations. Use it once on a new or large project, apply the highest-value ideas, and the rest of the stack works with less friction. 5. Learn Your Style With Task Observer You fix the same style issues again and again. Task Observer notes those fixes and can suggest skill updates. You check the ideas first. Over time, the AI starts closer to how you write. How the Pieces Fit Good Order to Add Them Write or improve CLAUDE.md.Run Claude Code Setup and add the best ideas.Add claude-mem so sessions remember the project.Add Headroom if tokens or context size become a problem.Add OmniRoute when you hit limits often.Add Task Observer once the rest is stable and you want better style match. Safety Notes OmniRoute, Headroom, and claude-mem (default) run on your computer.Free providers and any cloud sync send data outside. Check what they get.Only install tools you trust. Official Anthropic tools and well-known open projects are safer, but you still choose what runs.The extra layers add a little delay. On most computers, it is small next to model time. Keep Expectations Real Free models are not unlimited top quality. Limits and quality change. Paid models are still best for hard or important work. This stack mainly stops the daily restart and limit pain so you can code more. What About GitHub Copilot? Copilot has its own tools. This stack is made for Claude Code. Some parts can be reused: OmniRoute can work with tools that accept a custom address.Headroom works with several coding tools.claude-mem works with some other tools, but not the same way. Do not expect the same Claude Code Setup tips or Task Observer skill inside Copilot. Treat Copilot as a different setup that can share some routing or shrink tools, not the full stack. Closing Use a clear CLAUDE.md, OmniRoute for model switching, claude-mem for memory, Headroom for smaller prompts, Claude Code Setup for project fit, and Task Observer for style. You still face real limits. You just fight them less and ship more code.

By Ram Ghadiyaram DZone Core CORE
Two Engines, One Query: DuckDB Inside Neo4j
Two Engines, One Query: DuckDB Inside Neo4j

If you've been following the database world lately, you'll have noticed DuckDB turning up in some unexpected places. One of the more interesting examples is pg_duckdb, a Postgres extension that embeds DuckDB directly into the Postgres process, letting analytical queries run through DuckDB's engine without leaving the database. It's a clever idea: keep your data where it is, but bring a faster analytical engine. That got us wondering: could we do something similar with Neo4j? The answer is not in the same way. Postgres has a mature extension architecture that allows foreign query engines to be embedded at a low level. Neo4j doesn't expose that kind of hook. But Neo4j does let you write custom procedures in Java that run inside the Neo4j JVM and are callable directly from Cypher. Also, DuckDB ships an official JDBC driver that bundles its native binary and loads it via JNI. Put those two things together and you've got something that feels surprisingly close to the pg_duckdb experience: DuckDB running in-process, queries fired from Cypher, results flowing back into a graph. That's what we built. In this article, we'll walk through the whole thing from scratch. The full source code is available on GitHub. Prerequisites Before starting, we'll need the following: JDK 21 – check with java -versionMaven – check with mvn -versionNeo4j Desktop – with a local database instance created and ready to start Note that this approach requires a self-managed Neo4j install. AuraDB doesn't support custom plugins, so Neo4j Desktop is the right choice here. Version Summary For reference, here are all the component versions used in this article: ComponentVersionOpenJDK21.0.12.1Maven3.9.6Neo4j Desktop2.2.1Neo4j2026.07.0Neo4j Procedure API5.26.0DuckDB JDBC1.1.3JUnit Jupiter5.10.2maven-compiler-plugin3.11.0maven-shade-plugin3.5.2maven-surefire-plugin3.5.4 Getting the Code The full source code is on GitHub. To get just this project: Shell cd ~ git clone --filter=blob:none --sparse https://github.com/VeryFatBoy/neo4j.git cd neo4j git sparse-checkout set duckdb-procedure mv duckdb-procedure ../duckdb-procedure cd ../duckdb-procedure/neo4j-duckdb-procedure Or follow along from scratch using the steps below. How It Works The architecture is straightforward. We write two Java classes and package them as a fat jar: DuckDBConnection manages a singleton DuckDB connection that lives for the lifetime of the Neo4j process. It uses double-checked locking to keep things thread-safe and reconnects automatically if the connection is ever closed.DuckDBProcedure is the Neo4j procedure itself. It accepts a SQL string from Cypher, runs it against the shared DuckDB connection and streams the results back as rows, each one a Map<String, Object> that Cypher can destructure with dot notation. DuckDB runs in-memory by default, which means it starts clean every time Neo4j restarts. That's fine for analytics against external files, and we'll discuss file-backed persistence in the article Summary section. Project Setup Start by creating the project directory: Shell mkdir -p neo4j-duckdb-procedure/src/main/java/com/example/duckdb cd neo4j-duckdb-procedure Verify the structure: Shell find . -type d Now create pom.xml in the project root. XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>neo4j-duckdb-procedure</artifactId> <version>1.0.0</version> <packaging>jar</packaging> <properties> <java.version>21</java.version> <maven.compiler.source>21</maven.compiler.source> <maven.compiler.target>21</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <neo4j.version>5.26.0</neo4j.version> <duckdb.version>1.1.3</duckdb.version> </properties> <dependencies> <!-- Neo4j procedure API, provided by Neo4j at runtime --> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>${neo4j.version}</version> <scope>provided</scope> </dependency> <!-- DuckDB JDBC driver, bundled with native binary --> <dependency> <groupId>org.duckdb</groupId> <artifactId>duckdb_jdbc</artifactId> <version>${duckdb.version}</version> </dependency> <!-- Test dependencies --> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>5.10.2</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <source>21</source> <target>21</target> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>3.5.4</version> </plugin> <!-- Fat jar: bundles DuckDB JDBC into the plugin jar --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.5.2</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> The Neo4j dependency is provided scope — it won't be bundled into the jar since Neo4j supplies it at runtime. The DuckDB JDBC driver is included at compile scope so it gets bundled along with its native binary. The JUnit dependency is test scope so it's excluded from the fat jar. The maven-shade-plugin builds the fat jar at package time. Confirm Maven can resolve the dependencies: Shell mvn dependency:resolve The Java Classes Create DuckDBConnection.java at src/main/java/com/example/duckdb/: Java package com.example.duckdb; import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; public class DuckDBConnection { private static volatile Connection instance; private DuckDBConnection() {} public static Connection getInstance() throws SQLException { if (instance == null || instance.isClosed()) { synchronized (DuckDBConnection.class) { if (instance == null || instance.isClosed()) { instance = DriverManager.getConnection("jdbc:duckdb:"); } } } return instance; } } The volatile keyword ensures the instance is visible across threads without full synchronization on every read. The jdbc:duckdb: connection string with no path gives us an in-memory DuckDB instance. Now create DuckDBProcedure.java in the same package: Java package com.example.duckdb; import org.neo4j.procedure.Description; import org.neo4j.procedure.Name; import org.neo4j.procedure.Procedure; import java.sql.Connection; import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.Statement; import java.sql.Types; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Stream; public class DuckDBProcedure { public static class QueryResult { public Map<String, Object> row; public QueryResult(Map<String, Object> row) { this.row = row; } } @Procedure(name = "duckdb.query") @Description("Run a DuckDB SQL query and return results as rows") public Stream<QueryResult> query(@Name("sql") String sql) throws Exception { List<QueryResult> results = new ArrayList<>(); synchronized (DuckDBConnection.class) { Connection conn = DuckDBConnection.getInstance(); try (Statement stmt = conn.createStatement()) { boolean hasResultSet = stmt.execute(sql); if (hasResultSet) { try (ResultSet rs = stmt.getResultSet()) { ResultSetMetaData meta = rs.getMetaData(); int columnCount = meta.getColumnCount(); while (rs.next()) { Map<String, Object> row = new HashMap<>(); for (int i = 1; i <= columnCount; i++) { String colName = meta.getColumnName(i); int colType = meta.getColumnType(i); String typeName = meta.getColumnTypeName(i); Object value = mapValue(rs, i, colType, typeName); row.put(colName, value); } results.add(new QueryResult(row)); } } } else { Map<String, Object> status = new HashMap<>(); status.put("status", "OK"); status.put("updateCount", (long) stmt.getUpdateCount()); results.add(new QueryResult(status)); } } } return results.stream(); } private Object mapValue(ResultSet rs, int index, int sqlType, String typeName) throws Exception { if (typeName != null && typeName.equalsIgnoreCase("HUGEINT")) { long val = rs.getLong(index); return rs.wasNull() ? null : val; } switch (sqlType) { case Types.TINYINT: case Types.SMALLINT: case Types.INTEGER: { long val = (long) rs.getInt(index); return rs.wasNull() ? null : val; } case Types.BIGINT: { long val = rs.getLong(index); return rs.wasNull() ? null : val; } case Types.FLOAT: case Types.REAL: case Types.DOUBLE: case Types.NUMERIC: case Types.DECIMAL: { double val = rs.getDouble(index); return rs.wasNull() ? null : val; } case Types.BOOLEAN: case Types.BIT: { boolean val = rs.getBoolean(index); return rs.wasNull() ? null : val; } case Types.NULL: return null; default: return rs.getString(index); } } } A few things worth noting about DuckDBProcedure: We use stmt.execute(sql) rather than executeQuery() because DuckDB write operations like COPY don't return a ResultSet. The boolean return value tells us which path to take.The mapValue method handles type mapping explicitly. DuckDB's HUGEINT type – used for aggregated sums — has no standard java.sql.Types mapping, so we check the type name directly. Every numeric read is followed by rs.wasNull() to avoid returning 0 for a null column.We synchronize on DuckDBConnection.class to keep concurrent Cypher calls thread-safe against the shared connection. Writing the Tests We tried neo4j-harness first, which spins up a lightweight embedded Neo4j instance in JUnit — the same approach used in the Neo4j Procedure Template. In our environment, we found that the embedded server took longer than expected to become ready during the test run, which led us to a simpler alternative. The fix is to skip neo4j-harness entirely and test the DuckDB logic directly via JDBC. Since DuckDBProcedure is a thin wrapper around a JDBC connection, we can exercise the same type mapping, null handling, and file reading logic without Neo4j in the loop at all. The tests run in under 4 seconds and are fully self-contained. First, add the test directory structure and a src/test/resources folder to hold the test data files: Shell mkdir -p src/test/java/com/example/duckdb mkdir -p src/test/resources The test data files cover all the types and edge cases we want to verify: Plain Text src/test/resources/ ├── test.csv ├── test_nulls.csv ├── test_floats.csv ├── test_booleans.csv ├── test_timezones.csv └── test.json Maven copies everything in src/test/resources to target/test-classes automatically, so the test class can resolve file paths via the classloader without hardcoding any absolute paths. The test class at src/test/java/com/example/duckdb/DuckDBProcedureTest.java covers 12 cases: Range queriesCSV readingParquet write and readJSON readingAggregation returning numeric values rather than stringsNull handlingFloatsBooleansEmpty result setsBad SQL error surfacingIn-memory table persistence across callsDuckDB-side JOIN between two files Each test mirrors something we verified manually in Neo4j. File paths are resolved portably so the tests work regardless of where the project is cloned: Java private String resourcePath(String filename) { return getClass().getClassLoader().getResource(filename).getPath(); } The Parquet test is the one exception — since DuckDB generates the Parquet file rather than reading a pre-existing one, it writes to a JUnit @TempDir that's cleaned up automatically after the test run. Run the full suite with: Shell mvn test All 12 tests should pass in a few seconds. Running mvn package will also run the tests before building the fat jar, so any regression in the DuckDB logic is caught before the jar is deployed. Building and Installing Build the fat jar: Shell mvn package -DskipTests Any warning about MANIFEST.MF from the shade plugin is harmless. The jar lands at target/neo4j-duckdb-procedure-1.0.0.jar. Copy it to your Neo4j Desktop plugins folder. You can find the path in Neo4j Desktop under the three-dot menu for your database instance, then Open Folder > Plugins. Then copy the jar: Shell cp target/neo4j-duckdb-procedure-1.0.0.jar "/path/to/your/neo4j/plugins/" Open neo4j.conf in the conf folder alongside plugins and add: Plain Text dbms.security.procedures.unrestricted=duckdb.* Restart the database. Confirm the procedure is registered using Query from the left-hand navigation pane: Cypher SHOW PROCEDURES YIELD name WHERE name STARTS WITH 'duckdb' RETURN name You should see duckdb.query. If the procedure doesn't appear after restarting, also add the following in neo4j.conf: Plain Text dbms.security.procedures.allowlist=duckdb.* Some Neo4j configurations require procedures to be explicitly allowlisted before they're loaded at startup. Putting It to Work The test data files used here are also available in src/test/resources in the project repository. Let's start simple. DuckDB has a built-in range function we can use without any external data: Cypher CALL duckdb.query("SELECT range AS num FROM range(1, 6)") YIELD row RETURN row.num AS num Five rows should be returned, confirming that DuckDB is running inside Neo4j. Note that DuckDB's range() function is exclusive of the upper bound, so range(1, 6) returns 1 through 5. Now let's read a CSV file. Create a test file at /tmp/test.csv: Plain Text city,country,population London,UK,9648110 Paris,France,2161000 Berlin,Germany,3645000 Tokyo,Japan,13960000 New York,USA,8336817 Query it from Cypher: Cypher CALL duckdb.query("SELECT city, country, population FROM read_csv_auto('/tmp/test.csv') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB reads and parses the CSV; Neo4j receives the result. We can also write the CSV out as a Parquet file: Cypher CALL duckdb.query("COPY (SELECT * FROM read_csv_auto('/tmp/test.csv')) TO '/tmp/test.parquet' (FORMAT PARQUET)") YIELD row RETURN row And read it back: Cypher CALL duckdb.query("SELECT city, country, population FROM read_parquet('/tmp/test.parquet') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population DuckDB handles JSON just as easily. Create /tmp/test.json: JSON [ {"city": "London", "country": "UK", "population": 9648110}, {"city": "Paris", "country": "France", "population": 2161000}, {"city": "Berlin", "country": "Germany", "population": 3645000}, {"city": "Tokyo", "country": "Japan", "population": 13960000}, {"city": "New York", "country": "USA", "population": 8336817} ] And read it: Cypher CALL duckdb.query("SELECT city, country, population FROM read_json_auto('/tmp/test.json') ORDER BY population DESC") YIELD row RETURN row.city AS city, row.country AS country, row.population AS population We can also join two files entirely inside DuckDB before the result reaches Cypher. Create /tmp/test_timezones.csv: Plain Text city,timezone London,GMT Paris,CET Berlin,CET Tokyo,JST New York,EST Then join it against the population data: Cypher CALL duckdb.query("SELECT p.city, p.population, t.timezone FROM read_csv_auto('/tmp/test.csv') p JOIN read_csv_auto('/tmp/test_timezones.csv') t ON p.city = t.city ORDER BY p.population DESC") YIELD row RETURN row.city AS city, row.population AS population, row.timezone AS timezone The join happens entirely inside DuckDB. Neo4j just receives the final result set. Now for the most interesting demo: combining DuckDB analytics with Neo4j graph data in a single query. First, create some graph nodes: Cypher UNWIND [ {city: "London", continent: "Europe"}, {city: "Paris", continent: "Europe"}, {city: "Berlin", continent: "Europe"}, {city: "Tokyo", continent: "Asia"}, {city: "New York", continent: "North America"} ] AS data MERGE (c:City {name: data.city}) SET c.continent = data.continent Now join DuckDB file data with Neo4j graph data by city name: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.name AS city, c.continent AS continent, row.population AS population ORDER BY row.population DESC And aggregate by continent — population figures from DuckDB, continent groupings from the graph: Cypher CALL duckdb.query("SELECT city, population FROM read_parquet('/tmp/test.parquet')") YIELD row MATCH (c:City {name: row.city}) RETURN c.continent AS continent, count(c) AS city_count, sum(row.population) AS total_population ORDER BY total_population DESC That's two engines contributing to a single result: DuckDB handles the file analytics, Neo4j handles the graph traversal and aggregation. What We Learned Building this wasn't entirely smooth, and the bumps are worth documenting. XML comments in a Maven POM can't contain double dashes anywhere except at the closing -->. That's a quirk of XML parsing that's easy to miss.DuckDB's HUGEINT type – used for the result of aggregation functions like SUM – has no corresponding entry in the standard java.sql.Types constants. Our initial type mapping switch statement silently fell through to the string default, returning numeric values as strings. The fix is to check ResultSetMetaData.getColumnTypeName() by name before entering the switch.Write operations like COPY can't go through executeQuery() since they don't return a ResultSet. Switching to stmt.execute() and checking the boolean return value handles both read and write paths cleanly.Null numeric values in JDBC return 0 from getInt() and getLong(), not null. You must call rs.wasNull() immediately after each numeric read to distinguish a genuine zero from a missing value. Comparing Approaches It's also worth noting that APOC (Awesome Procedures on Cypher) Extended's apoc.load.jdbc procedure supports DuckDB via its JDBC driver and apoc.jdbc.analytics goes further by letting you export graph data into a temporary DuckDB table for analytical queries. These are complementary approaches worth exploring alongside the custom procedure built here. The following table compares the three ways to use DuckDB with Neo4j, each with a different integration style. ApproachHow It WorksDirectionPersistent StateRequiresCustom procedure (this article)DuckDB runs in-process inside the Neo4j JVM as a singleton connectionFiles and SQL into Neo4jYes, across Cypher callsCustom Java pluginAPOC apoc.load.jdbcDuckDB treated as an external JDBC data source; driver jar dropped into plugins folderFiles and SQL into Neo4jNoAPOC Extended + DuckDB JDBC jarAPOC apoc.jdbc.analyticsGraph data exported into a temporary DuckDB table; analytical SQL runs against itGraph data into DuckDB for analyticsNoAPOC Extended + DuckDB JDBC jar Summary We started with a simple question: could DuckDB be embedded in Neo4j the way pg_duckdb embeds it in Postgres? The answer is not natively, but closer than you might expect. With two Java classes, a fat jar, and a few lines of Neo4j configuration, we built a working duckdb.query procedure that runs DuckDB SQL directly from Cypher. Along the way we handled CSV, Parquet and JSON files, DuckDB-side joins, cross-engine aggregations with Neo4j graph data and a handful of type mapping edge cases that the 12-test suite now catches automatically. There's plenty of room to grow this further. File-backed DuckDB, a proper connection pool, cleaner error handling, and possibly a maintained open-source plugin are all natural next steps. But the foundation is there. What we've built here is a proof of concept — it works, and the tests confirm it, but anyone taking it further should treat robustness, connection pooling, and error handling as open items before putting it near production data. The full source code is available on GitHub.

By Akmal Chaudhri DZone Core CORE
Genkit Go vs ADK Go 2.0: A Hands-On Look at Boilerplate and Low-Level Control
Genkit Go vs ADK Go 2.0: A Hands-On Look at Boilerplate and Low-Level Control

Genkit and ADK (Agent Development Kit) are both Google-backed Gen AI frameworks with first-class Go SDKs, and I keep getting the same question: what is actually different when you sit down and write the code? This is not a "which one is better" article. They point at different problems: ADK is an agent orchestration framework (agents, sessions, multi-agent transfer, and since 2.0 a graph-based workflow engine), while Genkit is a GenAI application framework (model calls, flows, structured output, middleware). But they overlap on the everyday tasks every Go developer does (call a model, get typed output, stream, add tools, intercept requests), and on those tasks the developer experience diverges a lot. So instead of comparing docs, I wrote the same eight programs twice and ran every single one against a live Gemini backend: One prompt → one answerA raw model call with no framework machineryStructured output into a Go structStreamingTool callingIntercepting/wrapping the model callA custom model backendMulti-turn conversation and HTTP serving Everything below compiled and ran on Genkit Go v1.11.0 and ADK Go v2.1.0 (the google.golang.org/adk/v2 module, GA since June 30, 2026) with gemini-3-flash-preview, on 2026-08-05. Where I quote framework internals, they come from those released versions, not from the main branch. Nothing in this article is speculated from documentation. TL;DR TaskGenkit GoADK Go 2.0One prompt → textgenkit.GenerateText(...): 21 lines, 2 error pathsmodel + agent + runner + session + event loop: 49 lines, 8 error pathsStructured outputgenkit.GenerateData[Recipe](...), schema inferred from struct tags, typed resulthand-written genai.Schema tree, raw JSON text back, unmarshal it yourselfStreamingcallback receives deltas, final response is ready-assembledSSE partial events; the final event repeats the full text, you deduplicate by handToolstyped Go func, loop runs inside Generatetyped Go func (genuinely elegant), but agent/runner/session around itWrap the model callWithUse middleware with a real next(), applied per callfixed-signature callbacks (no wrapping) or decorate the model.LLM interface, per agentLocal models (Ollama)official plugin, first-class providerno native support; OpenAI-compatible adapter with BaseURLCustom model backendone function, works instantly with Generatetwo-method interface (the smallest contract in either SDK), but invoking it needs the full stackMulti-turnautomatic via agents + session stores (experimental track); resp.History() for plain generate callsautomatic via the GA session servicesMulti-agentdelegation via the Agents middleware (experimental track)first-class: sub-agents, transfer, workflow graph engineServe over HTTPgenkit.Handler(flow) on plain net/httpfull REST platform API (sessions, SSE, events) + embedded Web UIEcosystem16 Go plugins: multi-vendor models, vector stores, telemetry, MCPGoogle Cloud integrations (Vertex, GCS, BigQuery, Apigee, Cloud Run) + OpenAI-compatible + MCP The pattern that emerged: Genkit's primitive is the model call; ADK's primitive is the agent. When the thing you need is a model call, Genkit hands it to you directly, and ADK makes you rent the whole agent apparatus. When the thing you need is a long-lived, multi-agent system, ADK's apparatus stops being boilerplate and starts being the product. The Mental Model Genkit: The Model Call Is the Unit Genkit Go has exactly one setup object (*genkit.Genkit) and everything else is a package-level function call: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{})) text, err := genkit.GenerateText(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) That is the complete program (plus imports and an error check). Init doesn't even return an error: configuration problems panic at startup, and from there on there is a single err to handle per generation. My runnable version is 21 non-blank lines with 2 error-handling lines. ADK: The Agent Is the Unit ADK has no "just generate" entry point at the framework level. The unit of execution is an agent, and an agent runs inside a runner, and a runner requires a session service. This is the minimal equivalent program: Go model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{ APIKey: os.Getenv("GOOGLE_API_KEY"), }) // if err != nil ... a, err := llmagent.New(llmagent.Config{ Name: "assistant", Model: model, Instruction: "You are a helpful assistant.", }) // if err != nil ... r, err := runner.NewInMemory("hello-app", a) // in-memory session/artifact/memory services // if err != nil ... msg := genai.NewContentFromText("Why is Go a great language for building AI agents? One short sentence.", genai.RoleUser) var text string for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { if err != nil { /* ... */ } if event.LLMResponse.Content == nil { continue } if event.IsFinalResponse() { for _, part := range event.LLMResponse.Content.Parts { text += part.Text } } } 49 non-blank lines, 8 error-handling lines for the same output. And to be fair, runner.NewInMemory is already ADK being nice: the explicit form makes you wire runner.Config{AppName, Agent, SessionService, AutoCreateSession} yourself. Count the concepts you must hold for one prompt: a model, an agent with a Name (mandatory), an app name, a user ID, a session ID (both meaningless for a one-shot call, both mandatory positional arguments), a RunConfig, an event iterator, an IsFinalResponse() predicate, and manual concatenation of Content.Parts. There is no .Text() helper on events. Even ADK's own console launcher spends ~45 lines turning events into printable text (cmd/launcher/console/console.go), because every consumer re-implements this loop. None of this is bad design: sessions, users and apps are exactly the right vocabulary for a deployed multi-agent platform. It is simply the wrong altitude when what you wanted was an answer to a prompt, and there is no lower altitude offered (almost; see the escape hatch below). Structured Output: Where the Gap Is Widest This was the experiment with the biggest difference, and the one where I initially got ADK wrong in an instructive way. Genkit You describe the shape as a Go type. The JSON Schema is inferred from the struct (via reflection over json/jsonschema tags), constrained decoding is enabled by default, and you get the struct back, parsed and typed: Go type Recipe struct { Name string `json:"name"` PrepMinutes int `json:"prep_minutes"` Ingredients []string `json:"ingredients"` } recipe, _, err := genkit.GenerateData[Recipe](ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You generate recipes."), ai.WithPrompt("A simple tortilla de patatas recipe"), ) fmt.Println(recipe.Name, recipe.PrepMinutes, recipe.Ingredients) Output from my run: Plain Text Classic Spanish Tortilla de Patatas (15 min): 5 ingredients 28 lines total. The schema definition is the Go type. ADK ADK's llmagent.Config.OutputSchema takes a *genai.Schema: a hand-built tree, field by field, with a hand-maintained Required list. There is no "infer from struct" path for agent output (curiously, ADK does have schema inference, but only for tool inputs): Go recipeSchema := &genai.Schema{ Type: genai.TypeObject, Properties: map[string]*genai.Schema{ "name": {Type: genai.TypeString}, "prep_minutes": {Type: genai.TypeInteger}, "ingredients": {Type: genai.TypeArray, Items: &genai.Schema{Type: genai.TypeString}, }, Required: []string{"name", "prep_minutes", "ingredients"}, } a, err := llmagent.New(llmagent.Config{ Name: "chef", Model: model, Instruction: "You generate recipes.", OutputSchema: recipeSchema, }) The schema enforcement worked perfectly in my runs: Gemini returned valid JSON every time. The interesting part is what happens next. My first attempt read the event's Output field, which looks exactly like what you want. It was always nil. Digging into the released v2.1.0 source: the LlmAgent does parse the structured output internally, and does attach it to the event, but the runner deliberately strips it before the event reaches you. In runner/runner.go, every non-partial agent reply gets cloned with clone.Output = nil. The parsed value exists to feed ADK 2.0's graph workflow engine (typed routing between nodes), not for you. As a consumer of Runner.Run, you receive raw JSON as text and parse it yourself: Go var jsonText string for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) { // ... error handling, nil checks, IsFinalResponse, Parts loop ... jsonText += part.Text } var recipe Recipe if err := json.Unmarshal([]byte(jsonText), &recipe); err != nil { /* ... */ } (If you set OutputKey, the same raw string is also copied into session state, still not parsed for you.) Two more sharp edges I hit in the source while verifying this: The OutputSchema doc comment in v2.1.0 still warns: "when this is set, agent can only reply and cannot use any tools, such as function tools, RAGs, agent transfer, etc." The implementation has grown workarounds (on some backends it injects a synthetic set_model_response tool; on Vertex AI with Gemini ≥2.0 it can use native constrained output alongside tools), which means combining tools and output schema behaves differently depending on backend and model name, silently.Final tally: 68 lines vs Genkit's 28, and the typed half of "typed output" is entirely yours to write. Streaming: Deltas vs. Partial Events Both frameworks stream. The difference is who assembles the result. Genkit gives you a callback that receives deltas, and the final response arrives already assembled: Go resp, err := genkit.Generate(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("Count from one to ten in words, one per line."), ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error { fmt.Print(chunk.Text()) return nil }), ) My run printed streamed chars: 48, final response chars: 48. Chunks and final response account for the same text exactly once each. ADK streams by setting agent.RunConfig{StreamingMode: agent.StreamingModeSSE} and then flags chunk events with LLMResponse.Partial. The gotcha: after all the partial events, ADK emits a final non-partial event that repeats the entire text. My run: partial (streamed) chars: 48, final event chars: 48. Print events naively and you output everything twice. Every ADK consumer must carry this filter: Go for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{ StreamingMode: agent.StreamingModeSSE, }) { // ... if event.LLMResponse.Partial { fmt.Print(part.Text) // the streamed deltas } // the final event repeats the full text: skip it, or you print it twice } This isn't hypothetical: ADK's own example code and console launcher both contain hand-rolled deduplication (if text != prevText in cmd/launcher/console/console.go). It's a small thing, but it's a small thing every single ADK streaming consumer writes, and no Genkit consumer does. As a bonus, Genkit's GenerateDataStream[T] even streams partially-parsed typed structs (your Recipe filling in field by field), which has no ADK equivalent. Tool Calling: Closer Than I Expected Here I expected a bigger gap and didn't find one: ADK's function tools are genuinely well designed. Both frameworks let a tool be an ordinary typed Go function, with the JSON Schema inferred from the input struct: Go // Genkit weather := genkit.DefineTool(g, "getWeather", "Gets the current weather for a city", func(ctx *ai.ToolContext, in WeatherInput) (string, error) { return fmt.Sprintf("22C and sunny in %s", in.City), nil }) resp, err := genkit.Generate(ctx, g, ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithPrompt("What is the weather in Valencia right now?"), ai.WithTools(weather), ) Go // ADK: also nice! Generics, schema inferred, type params never spelled out. weatherTool, err := functiontool.New(functiontool.Config{ Name: "get_weather", Description: "Gets the current weather for a city", }, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) { return WeatherOutput{Report: fmt.Sprintf("22C and sunny in %s", in.City)}, nil }) Both ran the tool and produced the right answer. The differences are around the edges: Ceremony: the Genkit program is 29 lines; the ADK one is 66, because the tool still needs the agent/runner/session stack around it.Where the loop lives: Genkit's tool loop runs inside Generate (with parallel tool execution and a WithMaxTurns bound), so tools work in a plain one-shot call. In ADK the loop belongs to the agent flow, which is consistent with its philosophy, but it means there is no way to use ADK tools without an agent.A public-API gotcha: ADK's public tool.Tool interface is only Name()/Description()/IsLongRunning(). The executable contract (Declaration(), Run()) lives in an internal/ package you cannot reference, so you can't hand-implement a runnable tool against a public interface; you go through functiontool.New. Genkit's equivalent (ai.ToolFunc + DefineTool) is fully public.Composability limit: ADK documents (in its own examples/tools/multipletools) that you cannot mix built-in Gemini tools like geminitool.GoogleSearch{} with custom function tools on one agent; the workaround is wrapping each in a sub-agent.Genkit also has tool interrupts/resume as a stable API (pause generation from inside a tool, get an Interrupts() list, resume with RespondWith/RestartWith). ADK's equivalents are tool confirmation flows wired through its HITL machinery. Getting Low-Level You can go deeper in Genkit than in ADK, and its middleware system is the reason: Genkit exposes the model call, the tool executions, and the whole generation loop as wrappable layers, while ADK exposes fixed callback points and one wrappable interface. Here is what that means in code. Wrapping the Model Call Genkit middleware is a real onion: you get the request and a next() you control. Timing, retry, caching, and request rewriting are all one closure, applied per call: Go timing := ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) { return &ai.Hooks{ WrapModel: func(ctx context.Context, params *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) { start := time.Now() resp, err := next(ctx, params) log.Printf("model call: %d messages in, took %s", len(params.Request.Messages), time.Since(start)) return resp, err }, }, nil }) resp, err := genkit.Generate(ctx, g, /* ... */, ai.WithUse(timing)) Ran, printed model call: 1 messages in, took 3.323s. There are WrapGenerate and WrapTool hooks at the other two altitudes (the whole tool loop, and each tool execution), middleware can inject tools, and the built-ins (Retry, Fallback, tool approval, filesystem sandbox) compose the same way. ADK has callbacks, not middleware: BeforeModelCallback and AfterModelCallback with fixed signatures. They can short-circuit (return a response from Before and the model is never called, the documented caching hook) and they can replace the response, but they cannot wrap. There is no next(). To time a call, I had to smuggle state between two separate functions: Go var start time.Time before := func(ctx agent.Context, req *model.LLMRequest) (*model.LLMResponse, error) { start = time.Now() return nil, nil // nil means "continue to the model" } after := func(ctx agent.Context, resp *model.LLMResponse, respErr error) (*model.LLMResponse, error) { log.Printf("took %s", time.Since(start)) return nil, nil } This works (ran, took 1.631s) but it's a workaround; it's attached per-agent rather than per-call, and anything genuinely wrap-shaped (hedged requests, per-attempt timeouts, fallbacks across models) has to move to a different mechanism entirely: decorating the model. Which brings me to the thing ADK gets really right: Custom Model Backends: Local Gemma on Ollama Nobody plugs in an echo model, so no toy examples here. The realistic scenario is a backend the framework didn't bless for you: a local model server, an internal gateway, a provider without an official plugin. I tested two levels of this: running Google's Gemma locally on Ollama, and writing a backend from scratch. Genkit ships an official Ollama plugin, so the local model is a first-class provider: Go oll := &ollama.Ollama{ServerAddress: "http://localhost:11434"} g := genkit.Init(ctx, genkit.WithPlugins(oll)) gemma := oll.DefineModel(g, ollama.ModelDefinition{Name: "gemma3n:e4b", Type: "chat"}, nil) resp, err := genkit.Generate(ctx, g, ai.WithModel(gemma), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) 23 lines, ran on the first try against gemma3n:e4b on my Mac, and the local model gets the same Generate, flows, middleware, and structured output as any cloud model. The plugin covers embedders too. ADK has no Ollama integration of its own. The path that works is the OpenAI-compatible adapter pointed at Ollama's OpenAI endpoint: Go gemma, err := openaimodel.NewModel(ctx, "gemma3n:e4b", &openaimodel.ClientConfig{ BaseURL: "http://localhost:11434/v1", APIKey: "ollama", // any non-empty value; Ollama ignores it }) This also ran (47 lines with the usual agent, runner, and session around it), and it is a documented path: the package README lists Ollama, LM Studio, and vLLM. The difference is the shape of the support. In Genkit, the local model is a provider with its own plugin; in ADK, it rides on OpenAI compatibility, so anything Ollama offers outside that API surface is out of reach, and then you are writing the backend yourself, which is the next scenario. And When There Is No Plugin at All When the backend has no plugin and no OpenAI-compatible endpoint (an internal LLM gateway, a niche provider), you write the integration yourself. I did, in both frameworks: the same minimal OpenAI chat/completions client, roughly 45 lines of identical plain-net/http glue, run against the real OpenAI API. ADK's model.LLM interface is the smallest custom-backend contract I've seen in any framework: Go type LLM interface { Name() string GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error] } Two methods, and this interface is also ADK's genuine wrapping seam: ADK's own examples ship a resilientModel decorator doing retries and per-attempt timeouts this way. My OpenAI-backed model.LLM worked, with two costs on top of the shared HTTP glue. First, the vocabulary: requests and responses are google.golang.org/genai types, so you translate both ways, and some of it is non-obvious. The agent's Instruction arrives in Config.SystemInstruction, not in Contents; miss that and your backend silently drops the system prompt. (ADK's own OpenAI adapter is four files of translation code, which tells you how deep that rabbit hole goes. Also, parts of ADK sniff behavior from llm.Name(), so a renaming wrapper can silently change how output schemas are handled.) Second, the invocation: those two beautiful methods still need the agent, runner, and session stack around them to run. Total: 116 lines. The Genkit version of the same backend is the HTTP glue, a capability declaration, and nothing else: Go func callOpenAI(ctx context.Context, req *ai.ModelRequest, cb func(context.Context, *ai.ModelResponseChunk) error) (*ai.ModelResponse, error) { body := oaRequest{Model: "gpt-5-mini"} for _, m := range req.Messages { role := string(m.Role) // "system" | "user" | "model" if role == "model" { role = "assistant" } body.Messages = append(body.Messages, oaMessage{Role: role, Content: m.Text()}) } // ... plain net/http POST to api.openai.com/v1/chat/completions ... return &ai.ModelResponse{ FinishReason: ai.FinishReasonStop, Message: ai.NewModelTextMessage(out.Choices[0].Message.Content), }, nil } gpt := genkit.DefineModel(g, "myco/gpt-5-mini", &ai.ModelOptions{ Supports: &ai.ModelSupports{Multiturn: true, SystemRole: true}, }, callOpenAI) resp, err := genkit.Generate(ctx, g, ai.WithModel(gpt), ai.WithSystem("You are a helpful assistant."), ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."), ) Total: 79 lines, and Generate calls it directly; the model immediately works with structured output, streaming, middleware, and flows. The messages arrive as provider-neutral structs (ai.Message, with system prompt as a regular system-role message), so the translation is one role-mapping loop instead of a genai round-trip. One honest note: my first attempt failed at runtime because I passed nil options, since a Genkit model must declare what it Supports. That cost me one extra struct literal, and it's the flip side of a real feature: the framework validates every request against the declared capabilities and even simulates a system prompt for models that don't support one. The Raw Call Escape Hatch I didn't want to write "you can't go low-level in ADK" without trying it myself, so I tried. model.LLM is public, and this compiles and runs (33 lines in my experiment): Go llm, _ := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{APIKey: key}) for resp, err := range llm.GenerateContent(ctx, &model.LLMRequest{ Contents: []*genai.Content{genai.NewContentFromText(prompt, genai.RoleUser)}, }, false) { // resp.Content.Parts ... } But notice what you're holding: this is the bare genai transport with an ADK-shaped envelope. No instruction templating, no tool loop, no callbacks, no structured output processing, no telemetry: all of that lives inside the agent flow you just bypassed. There are no helpers (Text(), message builders) at this layer, and no example in the entire ADK repo uses it; only tests do. It's an escape hatch, not a supported altitude. That's the crux of the low-level story: in Genkit, dropping down doesn't cost you the framework. Generate is the low-level call: raw messages in, with middleware, tools, schemas, and tracing still active. In ADK, the abstraction is a cliff: you're either at agent altitude with everything, or at transport altitude with nothing. Where ADK's Ceremony Pays Off I also ran the experiments where I expected ADK's extra structure to earn its keep. It mostly did. Multi-turn conversation. Both frameworks carry the conversation for you; neither makes you thread history by hand. In ADK, I called r.Run twice with the same session ID and the second turn knew my name and city. The session service is the conversation, and swapping InMemoryService for the database or Vertex AI implementation brings persistence with it. In Genkit, the agents API does the same job with a session store. I ran the equivalent two turns on v1.11.0: Go g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithExperimental(), ) assistant := genkitx.DefineAgent(g, "assistant", aix.InlinePrompt{ ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You are a helpful assistant."), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()), ) first, err := assistant.RunText(ctx, "My name is Xavi and I live in Valencia. Say hi in one sentence.") // ... second, err := assistant.RunText(ctx, "Where do I live? Answer with just the city name.", aix.WithSessionID[any](first.SessionID)) Both frameworks answered turn 2: Valencia in my runs, with zero history plumbing on either side. And Genkit's session layer goes further than a message log: session snapshots, detach/resume (the client disconnects, the server keeps working, you resume from a snapshot) and artifacts, with in-memory and file stores shipping today plus a SessionStore interface for your own persistence. For conversation state, the two frameworks are at parity. (If you are outside an agent, composing plain Generate calls, ai.WithMessages(resp.History()...) threads history in one line, but that is a choice, not a limitation.) The one difference I would plan around is the maturity label. ADK's session services are GA and are the default path; Genkit's agents API is on the experimental track behind genkit.WithExperimental() and can change between minor releases, which is also why the rest of this article compares the stable surfaces. If you are comfortable riding the experimental track, Genkit gives you all of it today; if you need the stability guarantee, ADK has it. Multi-agent. ADK's sub-agents get an auto-injected transfer_to_agent tool, there are sequential/parallel/loop workflow agents, and 2.0's headline feature is a full graph workflow engine with typed routing between nodes, retries, and human-in-the-loop pauses. Genkit delegates too. The Agents middleware injects one delegate_to_<name> tool per sub-agent, discovers each sub-agent's description automatically for the orchestrator's prompt, forwards recent conversation history, and can merge sub-agent artifacts back into the orchestrator's session. I ran an orchestrator delegating to a poet sub-agent: Go orchestrator := genkitx.DefineAgent(g, "orchestrator", aix.InlinePrompt{ ai.WithModelName("googleai/gemini-3-flash-preview"), ai.WithSystem("You are a coordinator. Delegate writing tasks to the right sub-agent using its delegation tool, then return its result verbatim."), ai.WithUse(&middlewarex.Agents{ Agents: []aix.AgentRef{poet.Ref()}, MaxDelegations: 2, }), }, aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()), ) 43 lines, worked on the first run (I got my haiku back through the orchestrator), and the full researcher-plus-engineer version is in the official basic-agents sample. Same maturity note as sessions: this is the experimental track. And ADK's workflow layer, the sequential/parallel/loop agents and the 2.0 graph engine? Genkit's answer is that the orchestration language is Go itself. Flows compose: a flow calls other flows, sequence is code order, parallel is a goroutine group, a loop is a loop, and every flow stays traced and individually servable. I ran a brief-writing flow that fans out to two sub-flows in parallel and then synthesizes: Go brief := genkit.DefineFlow(g, "briefFlow", func(ctx context.Context, topic string) (string, error) { // Parallel: plain Go concurrency. var facts, structure string eg, egCtx := errgroup.WithContext(ctx) eg.Go(func() error { var err error facts, err = research.Run(egCtx, topic) return err }) eg.Go(func() error { var err error structure, err = outline.Run(egCtx, topic) return err }) if err := eg.Wait(); err != nil { return "", err } // Sequential: the next line of code. return genkit.GenerateText(ctx, g, ai.WithModelName(model), ai.WithPrompt("Write a two-sentence brief about %s using these facts:\n%s\nand this outline:\n%s", topic, facts, structure)) }) 53 lines including both sub-flows, ran on stable APIs, and genkit.Handler(brief) serves it over HTTP like any other flow. So the difference between the two workflow stories is representation, not capability. ADK reifies the pipeline as a data structure (edges, typed routes) executed by an engine that owns retries, pauses, and resumability at the graph level; Genkit keeps the pipeline as ordinary Go and covers those concerns with its own pieces (tool interrupts for human-in-the-loop, the retry middleware, and durable execution through integrations like Temporal). Pick by taste: an inspectable graph object, or code you read top to bottom. The serving platform. Genkit's HTTP story is its philosophy in miniature. A flow becomes a handler on plain net/http: Go mux := http.NewServeMux() mux.HandleFunc("POST /greetingFlow", genkit.Handler(greetingFlow)) server.Start(ctx, "127.0.0.1:9090", mux) Shell $ curl -X POST localhost:9090/greetingFlow -d '{"data":"Xavi"}' {"result":"Hello, Xavi!"} ADK's adkrest.NewServer is also a plain http.Handler (nice), but what it mounts is a platform: app listing, session CRUD, SSE runtime, artifacts, debug/trace endpoints. My curl transcript needed two calls (create the session, then run) and got back the full event envelope with usage metadata, state deltas, and node info. On top of that, ADK's launcher embeds the entire ADK Web UI into your binary via go:embed: go run . web webui gives you a chat UI with no npm and no separate process, which is honestly great. (Genkit's Developer UI is a separate CLI process, though it's also more than a chat UI: it traces every generate call, middleware hop, and tool execution.) These aren't contradictions of the boilerplate findings; they're the same design measured on its own terrain. ADK's five constructors are the price of a system where conversations persist, agents transfer to each other, and a REST platform falls out of a config struct. The Plugin Ecosystems One more thing I checked in the released modules, because it changes the decision for real projects: what each framework integrates with out of the box. Genkit Go v1.11.0 ships sixteen plugins in the module: model providers for Google (Gemini API and Vertex AI), Anthropic, OpenAI plus any OpenAI-compatible endpoint, and Ollama; vector stores for AlloyDB, PostgreSQL/pgvector, Pinecone, Weaviate and a local dev store; Firebase and Google Cloud telemetry; MCP; evaluators; and the middleware pack. And that is the Go module alone, before you count the wider JS ecosystem the framework shares its design with. ADK Go v2.1.0's integrations are a coherent Google Cloud story: Gemini and Vertex models, the OpenAI-compatible adapter, Apigee; sessions in memory, in a SQL database or on Vertex AI; GCS artifacts; Vertex AI memory; BigQuery agent analytics; MCP toolsets; and deploy tooling for Cloud Run and Agent Engine. What it does not have is a provider ecosystem beyond that: no Anthropic plugin, no vector store integrations, nothing aimed at other clouds. If your stack is pure Google Cloud, ADK's set covers it end to end, deploy included. The moment you need Anthropic next to Gemini, a Postgres vector store, or observability outside Google Cloud, Genkit has the plugin, and ADK expects you to build it. The Scorecard Numbers from my runnable programs (non-blank lines including imports; error-handling lines in parentheses): ExperimentGenkit GoADK Go 2.0Hello world21 (2)49 (8)Raw model callsame as hello33 (4), undocumented pathStructured output → typed struct28 (2)68 (9)Streaming27 (2)54 (8)One tool29 (2)66 (10)Time the model call34 (2)61 (8)Local Gemma via Ollama, invoked23 (2)47 (8)Custom model backend (real OpenAI), invoked79 (5)116 (13)Multi-turn (2 turns)37 (4) with agents + sessions; 30 (4) with plain Generate51 (8) Line counts are a blunt instrument; the interesting part is what the extra lines are. In every ADK program, they are the same lines: construct model, construct agent, construct runner, invent user/session IDs, iterate events, filter partials, concatenate parts. It's a fixed tax, roughly 25-30 lines, charged on every program regardless of whether sessions and events are relevant to it. A Concrete Decision Guide I would reach for Genkit Go when: The unit of my product is a model call or a flow: an API endpoint, a pipeline step, a CLI, a backend feature.I want typed structured output without maintaining schema trees by hand.I want to wrap, retry, fall back, cache, or rewrite model calls with composable per-call middleware.I'm integrating a custom or non-Google model backend and want the framework features to keep working on top of it.I value the Dev UI's per-call tracing during development.I want sessions and agents too, and I'm fine with the experimental track while that API stabilizes.My stack mixes vendors: Anthropic or OpenAI next to Gemini, Ollama locally, or Postgres/Pinecone/Weaviate for retrieval. The plugin shelf already covers them. I would reach for ADK Go 2.0 when: The unit of my product is a persistent, session-based agent: a chat product, a support agent, anything where conversation state is the point, and I want it on GA APIs today.I need multi-agent coordination: sub-agent transfer, workflow agents, or 2.0's graph engine with typed routing and HITL pauses.I want the serving platform for free: session REST API, SSE, the embedded Web UI, A2A, Cloud Run / Agent Engine deploy tooling.My stack is Gemini/Vertex-first and the genai-typed vocabulary is a feature, not a translation burden. And these compose better than you'd think: nothing stops you from building your model-centric services with Genkit flows and standing up an ADK agent where you genuinely need the agent platform. They're both idiomatic modern Go (both lean on iter.Seq2 and generics nicely), and they're solving different layers of the same problem. Conclusion After running everything, the facts are these: Genkit lets you get lower-level, and ADK charges more boilerplate. In detail: The boilerplate is real and remarkably consistent: ADK charges a fixed ~25-30 line agent tax on every program, because there is no supported altitude between "full agent" and "bare transport". Genkit's Generate occupies exactly that missing middle: raw request access with the framework still on.The low-level gap is sharpest in structured output (typed structs vs hand-built schema trees plus DIY parsing, and a runner that deliberately strips the parsed value it already has) and in middleware (a real next() vs fixed-signature callbacks).But ADK's ceremony is not waste: it buys GA session persistence, multi-agent transfer, a graph engine and an embedded serving platform, and some of its individual designs (the two-method model.LLM, generic function tools) are excellent Go. Pick the altitude that matches your product, and don't confuse "more setup" with "more capable", or "less setup" with "less serious." They're different tools that happen to share a language and a model provider. Further reading: Genkit Go Get Started guideADK 2.0 release notesGenkit GitHub repositoryADK Go GitHub repositoryGenkit Go basic-agents sample (orchestrator, delegation, artifacts)Genkit Go flows documentationStop Using Python for Gen AI: Genkit GoVercel AI SDK Middleware vs. Genkit Middleware

By Xavier Portilla Edo DZone Core CORE
5 Identity Patterns Your AI Agents Need Before They Touch Production
5 Identity Patterns Your AI Agents Need Before They Touch Production

Here's a scenario that plays out more often than anyone wants to admit. A team ships an internal agent to production. Neat little thing. It reads tickets, calls a couple of internal APIs, sends a Slack notification, and marks the ticket resolved. The engineer who deployed it, in the interest of getting something out the door, gave the agent a copy of their own API token. The token has the permissions of a senior engineer, which is to say it can do almost anything. Three months in, someone crafts a support ticket with embedded instructions. The agent, being an agent, treats those instructions as guidance. The Slack notification it sends goes to the wrong channel. The API call it makes touches production data it shouldn't have touched. And the audit log, when someone finally goes to reconstruct what happened, says the actions were performed by the senior engineer who lent their token six months earlier. That engineer is going to have a really bad afternoon. This is not a hypothetical. It's a pattern. And it's happening across the industry right now because most teams shipped agent-shaped software before they thought carefully about identity, and the tooling for doing this right isn't as widely known as it should be. What follows are five identity patterns that any agent in production needs to have in place. None of them are exotic. All of them are borrowed from twenty years of hard-won distributed systems experience, applied to the specific shape of the agent problem. If any one of them is missing, you have an incident waiting to happen. Pattern 1: Give the Agent Its Own Identity The single most common failure I see, and the one that causes the most cleanup work when things go wrong, is the borrowed identity problem. Agent runs under an engineer's credentials. Agent runs under a shared service account nobody owns. Agent runs under whatever role the deploying pod happened to inherit. All of these are the same failure dressed up differently, and the failure is that when the agent does something, you can't tell what the agent did versus what a human did. The fix is unglamorous. Every agent in your production environment gets a distinct machine identity. In cloud terms, that's a service account or a workload identity that belongs specifically to that agent. In identity-provider terms, it's a separate principal that gets audited, rotated, and permissioned in isolation. The SPIFFE and SPIRE projects have been doing serious work on machine identity in cloud-native environments for years, and their model of workload attestation (the agent proves it is what it claims to be through the environment it runs in) maps cleanly onto the agent case. Cloud providers offer their own versions: AWS IAM Roles for Service Accounts, Azure Workload Identity, GCP Workload Identity Federation. Pick one. The point is not the vendor. The point is that when an audit log entry appears, you can tell whether it came from Sarah, from the payments-agent-v3, or from something impersonating either of them. Two easy tests to see if you have this pattern right. First, can you list every agent in your environment along with its distinct identity? Second, if you had to revoke one agent's access this afternoon without affecting anything else, could you? If both answers are yes, you're in good shape. If either is no, the borrowed identity problem is somewhere in your stack. Pattern 2: Scope the Authority the Agent Gets Delegated Once your agent has its own identity, the next question is what that identity is allowed to do. The lazy answer, which is where most first deployments land, is "everything the deploying team could do." The correct answer is "the narrowest possible set of actions required for the agent to do its specific job, and no more." This sounds obvious. In practice, it is where most teams collapse, because designing narrow permissions is annoying and shipping is fun. But it's exactly the pattern that turns a small incident into a large one when things go wrong. The relevant technical primitives here come from the OAuth 2.0 family, and they've matured a lot in the last two years specifically because of agent use cases. The OAuth 2.0 Token Exchange spec lets you swap a broadly-scoped token for a narrowly-scoped one before the agent uses it. OAuth 2.0 DPoP (Demonstrating Proof of Possession) binds tokens to specific clients so a stolen token can't be replayed elsewhere. The OAuth 2.1 draft consolidates these into a modern default. For agent workloads specifically, the pattern that works is: the agent gets its identity from your identity provider, exchanges it for a scoped downstream token before each action, and that downstream token can only do the specific thing the agent needs to do right now. The tokens are typed. The scopes are named. And the blast radius of any single compromised token is one action, not the entire agent's job. The uncomfortable question this pattern forces you to answer is exactly what actions your agent needs to take. Most teams have never enumerated this precisely. Doing so is uncomfortable and clarifying, in that order. Pattern 3: Never Give the Agent Long-Lived Credentials Long-lived credentials are the single most reliable source of production incidents in cloud environments, and they're worse for agents than they are for humans because agents don't rotate their tokens the way humans rotate their laptops. The pattern is credentials that live for minutes, not months. Get them just before you need them. Discard them immediately after. Never persist them anywhere the agent's memory extends beyond a single request cycle. The tooling for this exists and is production-mature. HashiCorp Vault has been doing dynamic secrets and short-lived tokens for years. Cloud provider IAM offers just-in-time credential issuance through workload identity federation. Kubernetes gives you projected service account tokens with configurable lifetimes measured in minutes. The AWS Signing Version 4 family, when paired with STS-issued session credentials, gives you the same shape. The mental model shift that matters is that credentials become a request-scoped resource, not an agent-scoped one. When the agent starts a task, it acquires the credentials it needs. When the task ends, those credentials expire whether you remember to revoke them or not. If the agent's environment gets compromised, the attacker gets whatever credentials happened to be resident in memory at that instant. Which is almost nothing. The specific failure mode this prevents is the one that keeps security teams up at night: a stolen credential file, a compromised container image with baked-in secrets, an agent that got prompt-injected into leaking its own environment variables. All of those attacks become dramatically less useful when the credentials the attacker steals expire in three minutes. Pattern 4: Authorize Per-Tool, Not Per-Agent The moment agents started calling tools, and especially since MCP became a standard for exposing tools to agents, the granularity of authorization became a live problem. The way most first-generation agent frameworks handle this is: the agent has an API key or an OAuth token, and any tool the agent can invoke uses that same credential. Blanket authorization. If the agent has permission for anything, it has permission for everything. The pattern that works is per-tool authorization boundaries. Each tool the agent can invoke has its own scoped credential, tied to that specific tool and no others. When the agent invokes tool A, the credential presented has authority for tool A only. When it invokes tool B, a different credential presents authority for tool B only. Compromise of one tool's credential does not cascade to the others. This maps cleanly onto MCP's tool schema model, where each tool is a named function with a specific parameter shape. The agent's authorization to invoke each function should be independently governed. In OAuth terms, each MCP tool becomes a resource with its own scope. In enterprise-security terms, each tool becomes an object in your authorization policy, and the agent is a subject that has been granted specific access to specific objects. The additional discipline this pattern enforces is that when you introduce a new tool to an agent, you have to make an explicit decision about whether the agent should be authorized to invoke it. That decision goes through your normal access-control review. It doesn't just inherit from whatever the agent already had access to. Which is annoying, and correct. The tools with write access, network egress, or access to sensitive data deserve especially close review. Those are the tools where a prompt-injected agent does the most damage. Treating each of them as a distinct authorization decision is the difference between a controlled system and a system that quietly grows lateral movement paths every time someone adds a tool. Pattern 5: Preserve the Provenance Chain Across Every Hop The final pattern is the one most teams don't realize they need until they're in the middle of an incident. Agents rarely act alone. Agent A calls Agent B, which invokes a tool that talks to a downstream service, which triggers a job that eventually writes to a database. When something goes wrong at the far end of that chain, the question everyone needs to answer is: who initiated this? Not the immediate caller. The original caller. The human, or the scheduled trigger, or the external event that started the whole cascade. Without an intentional provenance chain, that question is often unanswerable. Each hop drops the identity of the previous hop and substitutes its own. By the time you're at the database write, all you can see is that Agent D did it, and Agent D has no memory of why. The pattern is to propagate identity, not replace it. Every hop signs its action with its own identity and includes the identity of the caller that triggered it. The provenance chain accumulates, so at the end of the chain you can read the complete history: user submitted request, request routed to agent A, agent A invoked agent B, agent B called tool X, tool X wrote to database Y. Every step signed. Every step attributable. The specifications and tools for this vary by maturity. SPIFFE-based trust domains support this natively. OAuth token exchange with the act claim preserves acting-on-behalf-of relationships across service calls. In-house implementations often just require a well-designed context header propagated through every service call, containing signed identity claims from each hop. Whatever the mechanism, the invariant that matters is: any action anywhere in your system can be traced back to the human, agent, or event that ultimately initiated it. When a bad outcome shows up in the logs, you can reconstruct who is accountable in real time rather than in a two-week postmortem. That capability is the difference between a system you can operate and one you're afraid to touch. The Objections Worth Answering "This is overkill for our internal tools." Every mature security incident in cloud history started with "this is just an internal tool." Internal agents get compromised. Internal tools become external targets when adjacent systems get breached. The five patterns cost a small amount of engineering upfront and prevent a very expensive class of incident later. "Our IAM tooling doesn't support this natively." Most of it does, actually. Every major cloud provider offers workload identity federation, short-lived tokens, and per-resource scoping. HashiCorp Vault covers most gaps. What's usually missing is not the tooling. It's the discipline to use the tooling. "We'll add this after we have product-market fit." The teams that say this most often are the teams that end up doing an emergency retrofit six months into rapid growth, when adding identity discipline to systems already in production requires touching every deployment. Doing it now, when the system is small, is much cheaper than doing it later, when the system is big and full of assumptions that would need to change. "Compliance hasn't asked for this." They will. When AI-related incidents start appearing in enforcement actions, and they are already appearing, compliance teams get asked what identity boundaries exist around agent workloads. The teams with these patterns in place answer calmly. The teams without them scramble. What This Buys You Getting these five patterns right is not glamorous. It won't win you a conference talk. It won't get you featured on Product Hunt. What it will do is make the difference between an agent that misbehaves and gets caught immediately with a clear blast radius, versus an agent that misbehaves and blows up something you can't easily reconstruct or contain. The teams that internalize this early get to operate agent-shaped software with the same confidence they operate any other production system. The teams that skip it get to write postmortems that are much harder than they needed to be. None of these patterns are novel. They are the accumulated wisdom of two decades of distributed systems security, applied to a new class of workload. The tooling exists. The specifications are mature. What has been missing is teams treating agent workloads with the same identity discipline they'd apply to any other service in their environment. Start with pattern one. The other four follow naturally once the foundation is in place.

By Rodrigo Martinez Pinto

Monthly Top Tools Experts

expert thumbnail

Abhishek Gupta

Principal PM, Azure Cosmos DB,
Microsoft

I mostly work on open-source technologies including distributed data systems, Kubernetes and Go
expert thumbnail

Yitaek Hwang

Software Engineer,
NYDIG

The Latest Tools Topics

article thumbnail
PWA on iOS, Part 4: What Add to Home Screen Really Does
Adding a PWA to the iPhone Home Screen creates a Web Clip with its own isolated storage. Learn what iOS saves, why logins disappear, and how to avoid costly bugs.
November 18, 2026
by Konstantin Shkurko
· 866 Views
article thumbnail
When Jev Says It's Sure, Is It Right? A Fraud Case Study in Neo4j
Jev’s confidence exceeds its accuracy, and only its top range holds up, mostly on easy cases. Use confidence for triage and error rates for automation thresholds.
November 17, 2026
by Akmal Chaudhri DZone Core CORE
· 678 Views
article thumbnail
Using Middleware in LangChain: Recovering From Tool Failures
LangChain Middleware provides ways to recover from Tool Failures. This article helps in understanding how to use ToolRetryMiddleware and @wrap_tool_call to handle errors.
November 17, 2026
by Ruchi Saini
· 688 Views · 3 Likes
article thumbnail
7 Failure Modes Every Production AI Agent Should Be Tested Against
Learn seven AI agent failure modes and learn how to test prompt injection, tool failures, memory, authorization, and unsafe actions.
November 16, 2026
by Mihir Bhatt
· 680 Views · 2 Likes
article thumbnail
Microsoft Foundry Agent Debugging Dashboard with Microsoft Fabric Eventhouse, KQL, Docker and Grafana
Build a Microsoft Foundry agent debugging dashboard using OpenTelemetry, Aspire, Microsoft Fabric Eventhouse, KQL, and Grafana.
November 11, 2026
by Jubin Soni, FBCS DZone Core CORE
· 1,441 Views · 1 Like
article thumbnail
Kafka Consumers on EKS: Designing for a Platform That Restarts Your Consumer Group
On EKS, pod restarts look like Kafka failures. This tutorial gives a baseline config and eight failure modes, with fixes for timing, rebalances and shutdown.
November 11, 2026
by Charan Reddy Rachumalla
· 909 Views
article thumbnail
Gitember: Building a Git Client Beyond Commit, Push, and Pull
Explore Gitember, an open-source Git client with multi-repository workspaces, historical content search, visual diff and merge tools, and local AI features.
November 11, 2026
by Igor Azarny
· 1,111 Views
article thumbnail
Kubernetes Rolling Updates Are Lying to You — and Your Users Are Paying for It
Learn how to configure Kubernetes readiness probes to verify application integration readiness and prevent failed requests during rolling updates.
November 11, 2026
by Pallavi Priya Patharlagadda
· 1,132 Views
article thumbnail
When to Use AMQP, JMS, Kafka, or MQTT: Trade-Offs, Not a Winner
AMQP vs. JMS vs. Kafka vs. MQTT: message broker, log, API, or edge protocol? How they differ, when to use which, and why one is not enough.
November 10, 2026
by Kai Wähner DZone Core CORE
· 1,934 Views · 5 Likes
article thumbnail
Kafka vs Flink vs Spark: Do You Really Need Real-Time?
Compare Kafka, Flink, and Spark by business SLAs, data consistency, and latency requirements to choose the right data streaming architecture.
November 10, 2026
by Kai Wähner DZone Core CORE
· 1,151 Views · 2 Likes
article thumbnail
Building a Feature Store With Neo4j and Feast
Neo4j works well alongside Feast as a feature computation engine, adding graph-native features like customer community detection to the same pipeline as aggregations.
November 10, 2026
by Akmal Chaudhri DZone Core CORE
· 1,127 Views
article thumbnail
Using the Anthropic Evaluation Tool to Check the Claude Code Plugin Efficiency
Evaluate Claude Code plugins and skills to measure their impact on code generation, reduce token usage, and identify useful technical guidance.
November 9, 2026
by Nicolas Duminil DZone Core CORE
· 989 Views
article thumbnail
Three Measured Outcomes of Manually Scaling Fleet in Prod
Run kubectl scale on prod while Fleet manages it and the fix can vanish in 7 seconds, hide behind a green status, or block your next deploy. Three lab cases, measured.
November 9, 2026
by Victor Estival
· 843 Views
article thumbnail
Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check
Learn how to use Spring Boot, Kafka, and Toxiproxy to simulate network failures, test producer idempotence, and troubleshoot duplicate messages.
November 6, 2026
by Soham Sengupta
· 1,278 Views · 2 Likes
article thumbnail
Full-Text Search Meets the Graph: Apache Lucene Inside Neo4j
Neo4j's built-in Lucene provides full-text search without a separate service. Search finds the entry point; the graph takes you the rest of the way.
November 5, 2026
by Akmal Chaudhri DZone Core CORE
· 1,123 Views · 1 Like
article thumbnail
A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)
Build a Go software factory with Genkit, Kubernetes, and senro, using a decision model to review designs, validate code, and control releases.
November 5, 2026
by Xavier Portilla Edo DZone Core CORE
· 1,343 Views
article thumbnail
Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go
Learn to run Genkit Go AI flows on Kubernetes with senro, using Gemini for text generation and jev for routing and reply validation.
November 4, 2026
by Xavier Portilla Edo DZone Core CORE
· 1,414 Views
article thumbnail
Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade
If a container has 1 CPU or under 1,792 MB, Java 27 silently switches it from Serial GC to G1. Test those services before you upgrade.
November 3, 2026
by Garima Agarwal
· 1,753 Views
article thumbnail
A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces
Learn how to secure Kubernetes Secrets by replacing long-lived service account tokens, tightening RBAC permissions, and improving Azure Key Vault integration.
November 2, 2026
by Ahmed Tariq
· 1,060 Views · 1 Like
article thumbnail
Deploying Small Language Models on Sovereign Cloud
Learn how to satisfy enterprise data sovereignty by packaging a fine-tuned Gemma SLM with Docker Hardened Images and deploying it securely to an air-gapped GDC cluster.
November 2, 2026
by Anuj Ashok Potdar
· 1,149 Views
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×