DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Software Design and Architecture

Software design and architecture focus on the development decisions made to improve a system's overall structure and behavior in order to achieve essential qualities such as modifiability, availability, and security. The Zones in this category are available to help developers stay up to date on the latest software design and architecture trends and techniques.

Functions of Software Design and Architecture

Cloud Architecture

Cloud Architecture

Cloud architecture refers to how technologies and components are built in a cloud environment. A cloud environment comprises a network of servers that are located in various places globally, and each serves a specific purpose. With the growth of cloud computing and cloud-native development, modern development practices are constantly changing to adapt to this rapid evolution. This Zone offers the latest information on cloud architecture, covering topics such as builds and deployments to cloud-native environments, Kubernetes practices, cloud databases, hybrid and multi-cloud environments, cloud computing, and more!

Containers

Containers

Containers allow applications to run quicker across many different development environments, and a single container encapsulates everything needed to run an application. Container technologies have exploded in popularity in recent years, leading to diverse use cases as well as new and unexpected challenges. This Zone offers insights into how teams can solve these challenges through its coverage of container performance, Kubernetes, testing, container orchestration, microservices usage to build and deploy containers, and more.

Integration

Integration

Integration refers to the process of combining software parts (or subsystems) into one system. An integration framework is a lightweight utility that provides libraries and standardized methods to coordinate messaging among different technologies. As software connects the world in increasingly more complex ways, integration makes it all possible facilitating app-to-app communication. Learn more about this necessity for modern software development by keeping a pulse on the industry topics such as integrated development environments, API best practices, service-oriented architecture, enterprise service buses, communication architectures, integration testing, and more.

Microservices

Microservices

A microservices architecture is a development method for designing applications as modular services that seamlessly adapt to a highly scalable and dynamic environment. Microservices help solve complex issues such as speed and scalability, while also supporting continuous testing and delivery. This Zone will take you through breaking down the monolith step by step and designing a microservices architecture from scratch. Stay up to date on the industry's changes with topics such as container deployment, architectural design patterns, event-driven architecture, service meshes, and more.

Performance

Performance

Performance refers to how well an application conducts itself compared to an expected level of service. Today's environments are increasingly complex and typically involve loosely coupled architectures, making it difficult to pinpoint bottlenecks in your system. Whatever your performance troubles, this Zone has you covered with everything from root cause analysis, application monitoring, and log management to anomaly detection, observability, and performance testing.

Security

Security

The topic of security covers many different facets within the SDLC. From focusing on secure application design to designing systems to protect computers, data, and networks against potential attacks, it is clear that security should be top of mind for all developers. This Zone provides the latest information on application vulnerabilities, how to incorporate security earlier in your SDLC practices, data governance, and more.

Latest Premium Content
Trend Report
Cloud-Native Foundations
Cloud-Native Foundations
Trend Report
Security by Design
Security by Design
Refcard #291
Code Review Core Practices
Code Review Core Practices
Refcard #392
Software Supply Chain Security
Software Supply Chain Security

DZone's Featured Software Design and Architecture Resources

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check

By Soham Sengupta
Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article. More
Building a Windows App for a Resilient SSH SOCKS Proxy and Selective Browser Routing

Building a Windows App for a Resilient SSH SOCKS Proxy and Selective Browser Routing

By Denis Tiumentsev
A full VPN is not always necessary. In addition, VPN connections may be unavailable because of restrictions imposed by a local or national internet provider. For the same reasons, a direct connection to an external SOCKS proxy may not work. Sometimes an alternative network route is needed not for the entire system, or even the entire browser, but only for specific addresses: an external API, an admin panel, documentation, a registry, a test environment, or another resource that is unavailable through the usual route. In this article, we’ll look at a narrower approach to solving this problem: To route traffic at the Chrome level, we use an extension suited to the task. In this case, we use FoxyProxy;Chrome sends only traffic for domains specified in FoxyProxy rules through the proxy;A local SOCKS5 endpoint runs on 127.0.0.1:1080;The endpoint is created using SSH dynamic forwarding;A PowerShell controller starts ssh.exe, monitors its state, and reconnects after a disconnection;An icon in the Windows notification area shows whether a local listener exists at the configured address and port;Launching the launcher again does not create a second controller or a second tray icon. I could not find an available tool that met my needs: one that provided this functionality, started quickly, was simple to use, and made its status easy to see. Existing solutions either have unnecessary settings or have complicated, unfriendly interfaces that require lengthy configuration before they can be used for this task. So, to create a local SSH tunnel on Windows, I developed my own PowerShell controller, launched through a VBS script. It displays a system tray icon, lets the user restart the tunnel, and automatically restores the connection when it drops. The complete source code for the PowerShell controller and VBS launcher, along with a sample configuration and installation instructions, is available in the repository: https://github.com/dtiumentsev/socks-via-ssh-tunnel-tray The controller is free to use under the MIT license. When Selective Routing Beats a Full VPN Suppose a developer needs access to a single API: api.example.dev Other websites, corporate services, local resources, and applications should continue to use the usual route. We also assume that outbound SOCKS traffic or VPN protocols may be unavailable. In this scenario, FoxyProxy matches a URL or domain to a proxy profile defined in its settings, while the proxy itself runs only locally: 127.0.0.1:1080 The goal is to manage the local tunnel as a tray application and be able to check its status at any time without extra steps. A local SOCKS5 proxy created through SSH dynamic forwarding is available at this address. Chrome connects to the local SOCKS5 endpoint, SSH encrypts the connection and carries it to the SSH host, and the remote side establishes a connection to the requested web resource. The setup looks like this: Chrome │ ├── Regular domains ──────────────────────────────► Direct connection │ └── api.example.dev │ ▼ FoxyProxy rule │ ▼ SOCKS5: 127.0.0.1:1080 │ ▼ ssh.exe with dynamic forwarding (-D) │ ▼ VPS / SSH host │ ▼ api.example.dev The advantage of this approach is its limited scope. The tunnel does not change Windows’ system-wide routing and does not affect applications that are not configured to use this SOCKS5 proxy. Prerequisites This article assumes that a VPS or other SSH host already exists and that the reader has legitimate key-based access to it. Preparing a VPS, installing sshd, and configuring server-side authentication are outside the scope of this article. The minimum requirements are: Windows 10 or Windows 11;The OpenSSH Client component installed;PowerShell 5.1 or later;A VPS or another SSH host accessible with a key;A private key corresponding to a public key authorized on the server;Google Chrome, Chromium, or another compatible browser;The FoxyProxy extension;Permission to use the SSH host and access the target resources. To check whether the built-in OpenSSH Client is present, run: Test-Path "$env:WINDIR\System32\OpenSSH\ssh.exe" A result of True means the path to ssh.exe exists. To check whether the private key is available, run: Test-Path "$env:USERPROFILE\.ssh\id_rsa" All environment-specific parameters are kept in a separate config.ps1 file. Here is an example configuration: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The main script loads config.ps1 from its own directory using $PSScriptRoot. The repository contains config.example.ps1; the working config.ps1 contains the actual local settings. SSH Dynamic Forwarding as a Local SOCKS5 Proxy The solution is based on SSH dynamic forwarding, an OpenSSH client mode enabled with the -D option. A simplified command looks like this: ssh.exe ` -i "$env:USERPROFILE\.ssh\id_rsa" ` -p 22 ` -D 127.0.0.1:1080 ` -N ` [email protected] After a successful connection, the SSH client begins listening on a local TCP port 1080 at 127.0.0.1. Any application that supports SOCKS5 can use this endpoint. Importantly, the listener is bound to the loopback address: 127.0.0.1:1080 rather than: 0.0.0.0:1080 The first option is accessible only from the local machine. The second would make the SOCKS proxy accessible to other devices on the network, which is unnecessary for this scenario and increases the risk of unauthorized use. SSH Options The controller uses the following arguments: $arguments = "-4 -i `"$keyPath`" -p $remotePort -D ${bindAddress}:$port -N " + "-o ExitOnForwardFailure=yes " + "-o ServerAliveInterval=15 " + "-o ServerAliveCountMax=3 " + "-o ConnectTimeout=10 " + "-o StrictHostKeyChecking=accept-new " + "-o IdentitiesOnly=yes " + $sshHost Here is what the main options do: OptionPurpose-4Forces IPv4 for the connection to the SSH host-i <key>Specifies a particular private key-p <port>Sets the server’s SSH port-D 127.0.0.1:1080Creates a local SOCKS5 listener on the loopback interface-NDoes not open an interactive shell session or execute a remote commandExitOnForwardFailure=yesExits SSH if dynamic forwarding cannot be establishedServerAliveInterval=15Sends SSH-level keepalive packets every 15 seconds of inactivityServerAliveCountMax=3Closes SSH after three unanswered keepalive messagesConnectTimeout=10Limits the time allowed for a TCP connection attempt to the SSH hostStrictHostKeyChecking=accept-newAdds a new server’s key to known_hosts but does not silently accept a change to a previously known keyIdentitiesOnly=yesUses the specified key rather than trying every identity available through ssh-agent ExitOnForwardFailure=yes is especially important for automation. If the local listener cannot be created, SSH should not keep running as though the tunnel were available. This option makes the client exit when it cannot establish dynamic, local, remote, or tunnel forwarding. StrictHostKeyChecking=accept-new is a practical compromise for the first connection to a new SSH host. It does not disable checking a previously known host key: if the server’s key changes unexpectedly, the client should terminate the connection rather than silently trust the new key. Host key verification remains part of SSH’s security model. Why Starting SSH Once Is Not Enough Running ssh.exe -D solves only the initial problem: it creates a SOCKS listener. In practice, long-lived connections regularly encounter unstable conditions: A laptop goes to sleep and wakes up;The device switches between Wi-Fi, Ethernet, and mobile internet;The external IP address changes;A home router restarts;The SSH host becomes temporarily unavailable;The TCP session’s network state becomes stale;The ssh.exe process exits because of an authentication error, timeout, or network interruption. If SSH was started manually in a separate window, the user has to notice the interruption, determine that the listener is gone, and run the command again. That is inconvenient for a background tool. A small controller can handle four tasks: Start ssh.exe in the background.Check whether the local SOCKS port is listening.Wait for a specified interval after SSH exits, then try to reconnect.Show the current state through a system tray icon. The PowerShell Controller The complete script is in the repository, but its logic is fairly straightforward. The controller keeps a reference to the SSH process: $script:sshProcess = $null It also sets a reconnect delay: $reconnectDelaySeconds = 5 After starting ssh.exe, the controller does not immediately declare the tunnel operational. It checks for a local listener using Get-NetTCPConnection: Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue This is better than relying only on the PID returned by Start-Process. A process starting successfully does not mean SSH has authenticated, established a session, and opened the local SOCKS listener. A simplified version of the check looks like this: function Is-Port-Listening { $listener = Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue return ($null -ne $listener) } When the controller detects a TCP listener at the configured address and port, it shows a green icon. If the listener disappears, the icon turns red. Why Use a WinForms Timer? To keep the tray icon working, the PowerShell script runs a WinForms message loop: [System.Windows.Forms.Application]::Run() If the script exits, the icon disappears. The tray interface also stops responding to clicks and context-menu actions. The tunnel is checked periodically using System.Windows.Forms.Timer: $timer = New-Object System.Windows.Forms.Timer $timer.Interval = $checkIntervalMs In the current version, the check runs every five seconds. This reduces how often the port is polled; the exact time of the next connection attempt depends on when the timer fires after the reconnect delay expires. NotifyIcon is the standard Windows Forms component for displaying an icon in the Windows notification area. It works well for a small background controller with a minimal interface. How Reconnection Works The recovery logic is intentionally simple: ssh.exe exits ↓ The controller clears the saved PID ↓ It waits five seconds ↓ It starts another attempt if the port is free Here is the part that detects when SSH has exited: if ($script:sshProcess -and $script:sshProcess.HasExited) { $script:sshProcess = $null $script:nextStartTime = (Get-Date).AddSeconds($reconnectDelaySeconds) } Then, once the wait is over: if ( -not $script:sshProcess -and -not (Is-Port-Occupied) -and (Get-Date) -ge $script:nextStartTime ) { Start-Ssh } Before starting a new process, the controller checks whether the configured local port is free. It does not terminate a process that may have occupied the port: in the event of a conflict, the user must free the port manually or choose another LocalPort in config.ps1. Reconnection cannot begin until the SSH client realizes that the connection is no longer working. The following SSH keepalive settings help with that: ServerAliveInterval=15 ServerAliveCountMax=3 With ServerAliveInterval=15 and ServerAliveCountMax=3, SSH may detect an unresponsive server after approximately 45 seconds (15 × 3). The controller then waits another five seconds before attempting to reconnect. This is not a strict recovery-time guarantee. The actual result depends on the TCP connection’s state, the type of interruption, network latency, and the behavior of the SSH host. Still, this mechanism is much more reliable than passively waiting for the system’s TCP timeout. The System Tray Icon Keeping a PowerShell window open for a background tunnel is inconvenient. An icon in the notification area provides a simple interface: A green icon means a local TCP listener was detected at the configured ${BindAddress}:${LocalPort}; in the example configuration, that is 127.0.0.1:1080; A red icon means the listener is absent or the tunnel has been stopped; A context menu provides manual control over the connection lifecycle. Version 1 provides the following commands: Menu itemActionReconnect nowTerminates the current managed SSH process and initiates another connection attemptStop tunnelStops SSH and disables automatic reconnectionStart tunnelResumes the controller and starts another connection attemptExitStops SSH, removes the tray icon, and exits the PowerShell controller A manual stop is fundamentally different from a network failure. After Stop tunnel, the controller sets an internal stopped state and does not start SSH again. Otherwise, the script would restore the tunnel five seconds after the user stopped it. Why the Icon Is Not Updated on Every Timer Tick I do not consider reassigning NotifyIcon.Icon on every timer tick a good approach. Even when the state has not changed, Windows has to redraw the icon, which may look like flickering. In version 1, the port state is stored: $script:lastPortState = $null The icon changes only when the state changes: if ($alive -eq $script:lastPortState) { return } This is a small detail, but it matters for the normal behavior of a background tool. Preventing Duplicate Launches It is easy to launch a tray application again—for example, by double-clicking the launcher a second time. Without protection, this could produce two tray icons and multiple PowerShell controllers trying to manage the same local port. A named mutex prevents that: $mutexName = "Local\Denis_SocksTunnelTray_v1" $createdNew = $false $mutex = New-Object System.Threading.Mutex( $true, $mutexName, [ref]$createdNew ) if (-not $createdNew) { $mutex.Dispose() exit 0 } The first instance creates the mutex and owns it for as long as it runs. When launched again, the second instance detects the existing mutex and exits immediately. It does not create a second SSH process or add another tray icon. The Local\ prefix limits the mutex’s scope to the current Windows user session. That is sufficient for a desktop tool running in a normal interactive session. Launching Without a Black Console Window Launching a .ps1 file through a .cmd file opens a cmd.exe window. Even if the PowerShell process is minimized or hidden, the console launcher can still be visible briefly. A minimal VBS launcher is more convenient for users. It starts PowerShell in hidden mode and then exits. The Start-SocksTunnelTray.vbs file sits next to the PowerShell script: Option Explicit Dim fso, shell, folder, ps1, command Set fso = CreateObject("Scripting.FileSystemObject") Set shell = CreateObject("WScript.Shell") folder = fso.GetParentFolderName(WScript.ScriptFullName) ps1 = folder & "\Start-SocksTunnelTray.ps1" command = "powershell.exe -NoLogo -NoProfile -STA -WindowStyle Hidden -ExecutionPolicy Bypass -File """ & ps1 & """" shell.Run command, 0, False Here: -NoLogo removes the PowerShell startup banner;-NoProfile prevents the user’s PowerShell profile from affecting the process;-STA starts PowerShell in a single-threaded apartment model suitable for Windows Forms;-WindowStyle Hidden hides the PowerShell window;-ExecutionPolicy Bypass applies only to this process and does not change the system’s execution policy;shell.Run command, 0, False starts the process without displaying a window and does not wait for it to exit. The user double-clicks the VBS file and sees only the tray icon. Configuring FoxyProxy Once the tunnel is running, the browser needs to know which requests to send through the local SOCKS5 proxy. In FoxyProxy, create a proxy profile with these settings: Proxy type: SOCKS5 Host/IP address: 127.0.0.1 Port: 1080 Next, add a rule that applies the profile only to the required domain. For example: https://api.example.dev/* Or, if subdomains are needed: https://*.example.dev/* The key is not to enable proxying for all URLs when the goal is selective routing. The narrower the rule, the easier it is to control its effects and troubleshoot problems. DNS Through SOCKS If the extension offers a “DNS resolution through SOCKS5” setting, it makes sense to enable it for domains that should use the proxy route. The practical reason is simple: a standard SOCKS5 profile may forward the TCP connection, but the browser or operating system may resolve the hostname locally before opening that connection. Depending on the configuration, this means some network information does not follow the selected route. To test from the command line, you can use: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.example.dev/ The h in socks5h indicates that the proxy, rather than the local system, should resolve the hostname. Verifying the Tunnel First, make sure Windows sees the listener: Get-NetTCPConnection -State Listen -LocalPort 1080 | Select-Object LocalAddress, LocalPort, OwningProcess Expected output looks like this: LocalAddress LocalPort OwningProcess ------------ --------- ------------- 127.0.0.1 1080 12345 Next, you can test proxying through an external service that reports the IP address of the outgoing connection: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.ipify.org If the command returns the address of your external proxy host, the traffic is passing through the local SOCKS5 endpoint and the remote route. Diagnosing an Occupied Local Port If the tray icon remains in the reconnecting state, check which process is listening on the local port. The repository includes a separate diagnostic script for this: .\scripts\Check-SocksPort.ps1 The script checks the fixed TCP port 1080 and displays the LocalAddress, LocalPort, state, owning PID, and process name. If config.ps1 specifies a different LocalPort, change $port = 1080 in scripts/Check-SocksPort.ps1 to the required port. The script does not terminate processes or change the configuration. If an old ssh.exe process occupies the port, exit the old controller using Exit, or stop the process in a Windows session with the same privileges under which it was started. If another program occupies the port, do not terminate it automatically: change LocalPort in config.ps1 and in the FoxyProxy profile, for example, from 1080 to 1081. Testing Recovery After a Failure A minimal test: Start the controller using Start-SocksTunnelTray.vbs.Wait for the green icon.Open Task Manager.Find the ssh.exe process started by the controller.End the process.Confirm that the tray icon changes to the unavailable state.Wait for several check cycles.Check that a new ssh.exe process appears.Confirm that the listener at 127.0.0.1:1080 is available again. You can also test the following scenarios: ScenarioExpected behaviorTemporarily disconnect from the networkSSH detects the lost connection through keepalives, exits, and the controller begins reconnect attemptsClick Stop tunnelSSH exits, the listener disappears, and no reconnection occursClick Start tunnelThe controller starts SSH againClick Reconnect nowThe current SSH process exits, followed by another connection attemptLaunch the VBS launcher a second timeThe second instance exits silently because of the mutexHave another program occupy port 1080SSH cannot establish forwarding; the conflict must be resolved or another port selectedCheck a regular site without a FoxyProxy ruleTraffic should go directly rather than through the SOCKS profileCheck api.example.dev with a ruleThe request should use the local SOCKS5 profile The Indicator Checks the Local Port A green icon means there is a TCP listener at the configured address and port. The controller does not check whether the listener belongs to the ssh.exe process it started, nor does it confirm that the target site is reachable. If an old SSH process or another program occupies the port, check its owner using scripts/Check-SocksPort.ps1. The controller does not automatically terminate that process. SOCKS Dynamic Forwarding Is Primarily for TCP SSH dynamic forwarding works as a SOCKS proxy and is primarily intended for TCP connections. It is not a universal way to route every protocol on the system. If you need to proxy UDP, the entire network stack, or multiple applications without built-in proxy support, you will need a different type of solution. Security Recommendations A few useful practices: Bind the SOCKS listener to 127.0.0.1, not 0.0.0.0.Use a passphrase-protected SSH key if your workflow supports ssh-agent.Verify the SSH host’s fingerprint before connecting for the first time.Do not disable host key checking globally with StrictHostKeyChecking=no.Keep the list of FoxyProxy domains short and explicit. Repository Structure The socks-via-ssh-tunnel-tray repository is organized as follows: socks-via-ssh-tunnel-tray/ ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── SECURITY.md ├── Start-SocksTunnelTray.ps1 ├── Start-SocksTunnelTray.vbs ├── config.example.ps1 ├── icons/ │ ├── down.ico │ └── up.ico └── scripts/ └── Check-SocksPort.ps1 config.example.ps1 contains only a template: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The user copies it to config.ps1 and fills in their own settings. The repository’s code is distributed under the MIT license. Conclusion An application using SSH dynamic forwarding provides a local SOCKS5 endpoint without requiring a separate proxy service on the client machine. Combined with FoxyProxy, it becomes a selective routing tool: the browser uses an alternative route only for explicitly selected domains, while the rest of the system continues to work as usual. This PowerShell controller handles the operational side of the task: It starts ssh.exe in the background;It shows the listener’s status through a tray icon;It restores the connection after the SSH tunnel drops;It distinguishes a manual stop from a network failure;It prevents duplicate instances when launched again;It does not leave a console window open. For a developer or infrastructure engineer, this is a simple, functional alternative to a system-wide tunnel when a managed route is needed for only a few resources rather than changing the routing of the entire computer, including in situations involving restrictions imposed by internet providers. More
Member Spotlight: Srinivas Chippagiri
Member Spotlight: Srinivas Chippagiri
By Dominique Pugh
GET, POST, PUT, DELETE: Choosing HTTP Verbs in a Real .NET Core Web API, Not Just by the Textbook
GET, POST, PUT, DELETE: Choosing HTTP Verbs in a Real .NET Core Web API, Not Just by the Textbook
By Dhana PAMULA
How to Hide Sensitive Information Before You Share a File
How to Hide Sensitive Information Before You Share a File
By Nagappan Subramanian DZone Core CORE
A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)
A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

The previous article put Genkit flows in Kubernetes pods with senro and let jev, a decision model, route a support ticket. The decisions there were small: which team, how urgent, send or hold. This one hands jev something bigger to judge: software. A software factory takes a feature request and produces a Go module: a design, a plan, the code, a green build, passing tests, and a pull request description. Gemini writes every one of those. jev decides, at every gate, whether the work moves on, goes back, or stops for a human. Each stage runs as a pod, each stage is a Genkit flow, and the whole thing is a few hundred lines of Go. Everything below ran against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5 and a local kind cluster. The code is in genkit-k8s-flows-jev-software-factory. Why a Factory Needs a Judge, Not a Second Writer The usual way to review model-written code is to ask another chat model. It works, up to a point. The reviewer writes a paragraph, you parse the paragraph, the paragraph is sometimes wrong in the same way the code is, and a well-written pull request description can talk the reviewer into "looks good to me." A decision model does not write. It takes a state (the request, the plan, the code, the test log) and answers typed questions with probabilities, in a few hundred milliseconds. There is no prose to parse and nothing to be talked into: BuildGreen is a number, and a red test log drags it down whatever the pull request says. Chat model as reviewerDecision model as reviewerVerdictA paragraph you parseChoice[CodeVerdict]: ship, rework or human, with a probability eachEvidenceMentioned, maybeA Noul per question: covers the request, tests are meaningful, build is greenFeedback for the next roundWhatever it felt like writingEach question that scored "yes" is one line of feedbackLatency and costSeconds, thousands of output tokensMilliseconds, tokens for the state onlyCan be persuaded by the authorYesNo: it never reads instructions, only state The split that falls out of this is the whole design: Gemini builds, jev decides. And senro is what gives each build step a pod, a workspace, a log, a retry policy, and a secret, and each decision a place in the graph. What We Are Building A feature request goes in. A reviewed Go module comes out, or a brief for a human saying why not. The factory is three senro pipelines and a small state machine over them: RunStepWhereWhoWhat it leaves behinddesigncleancoordinatornonean empty workspace with request.md in itdesigndesignpodGemini and jev, in a loopdesign.json, DESIGN.md and one record per review rounddesignplanpodGeminiplan.json, PLAN.mdimplementimplementpodGeminithe module's filesimplementvet, testpodgo vet, go testvet.log, test.log and their exit codesimplementreviewpodjevreview.json: ship, rework or humanreworkfix, then vet, test, reviewpodGemini, Go, jevthe same, after a fixfinishpr then publishpod, coordinatorGeminiPR.md and the module under out/<request>/srcfinishbrief then notifypod, coordinatorGeminiBRIEF.md for a human jev decides three times. Inside the design step, it reviews each design Gemini writes, and the loop revises until jev says ready, says reject, or three rounds are up. After test, it reviews the code with the logs in hand and picks ship, rework or human. And the state machine in main turns each verdict into the next senro run, with rework bounded at two. What You Need Go 1.27 or newer, Docker, kind, and kubectlGEMINI_API_KEY and TYPESAFE_API_KEY in your environmentthe senro CLI if you want the terminal UI (optional) Shell go get github.com/firebase/genkit/[email protected] go get github.com/xavidop/[email protected] go get github.com/xavidop/mamori Step 1: A Cluster and an Image With a Go Toolchain The cluster setup is the one from the previous article: kind, a namespace, a ServiceAccount with the permissions senro's Kubernetes executor needs (including get on pods/exec), and three environment variables, since senro never reads your kubeconfig. The repo's scripts/cluster-up.sh does it and source scripts/cluster-env.sh exports them. The one difference is the image. A func step needs only sh and tar in the pod, but this factory also runs go vet and go test as plain command steps, so every step uses the official Go image, pinned to its digest as senro requires: Go const stepImage = "docker.io/library/golang:1.27-alpine@sha256:f2c872bcee949b7be9c490efd02b48a8a2fbf947cdf533f1c58525ec9d8170fd" func pods() senro.ExecutorTarget { return k8s.Pod(stepImage, k8s.Namespace("factory"), k8s.Resources( map[string]string{"cpu": "500m", "memory": "512Mi"}, map[string]string{"cpu": "2", "memory": "2Gi"}, )) } Your pipeline binary is still not in that image. senro cross-compiles it and sends it into each pod, exactly as before. Step 2: The Design Loop, With jev Inside the Flow The first gate is the one that saves the most money: a bad design produces bad code, a green build and a confident pull request. So the design is reviewed before anything is planned, and the review is a loop inside one Genkit flow. The questions are a Go type. Every yes/no question is one a revision can act on: a "yes" becomes a line of feedback for the next round. Go type DesignVerdict string func (DesignVerdict) Criteria() map[DesignVerdict]string { return map[DesignVerdict]string{ "ready": "Complete, unambiguous, scoped to the request, and testable as described", "revise": "The right direction, with gaps a revision can close", "reject": "The request is too vague or too large to design from; a human has to clarify it first", } } type Risk int func (Risk) Levels() []string { return []string{"Low", "Medium", "High"} } type DesignReview struct { Verdict systemonex.Choice[DesignVerdict] `json:"verdict" jsonschema_description:"Is this design ready to implement?"` RequestClear systemonex.Noul `json:"request_clear" jsonschema_description:"Does the request say what to build precisely enough that two engineers would build the same thing?"` ScopeCreep systemonex.Noul `json:"scope_creep" jsonschema_description:"Does the design add things the request did not ask for?"` Underspecified systemonex.Noul `json:"underspecified" jsonschema_description:"Is any component described too loosely to implement without guessing: missing fields, signatures, edge cases or error behaviour?"` MissingTests systemonex.Noul `json:"missing_tests" jsonschema_description:"Does the test plan leave any requirement of the request untested?"` Risk systemonex.Score[Risk] `json:"risk" jsonschema_description:"How risky is implementing this design as written?"` } And the loop. Gemini designs; jev reviews; if the verdict is revise, Gemini gets the previous design plus the feedback and tries again: Go f.Design = genkit.DefineFlow(g, "designFeature", func(ctx context.Context, request string) (DesignResult, error) { var res DesignResult feedback := "" for round := 1; round <= maxDesignRounds; round++ { prompt := "Request:\n" + request if round > 1 { prompt += "\n\nYour previous design:\n" + asJSON(res.Design) + "\n\nReviewer feedback:\n" + feedback } d, _, err := genkit.GenerateData[Design](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You are a senior Go engineer writing a short design for exactly what the request asks. "+ "Stay inside the request: no extra features. Components are the Go types and functions you will create. "+ "The test plan names concrete, deterministic test cases. Be brief."), ai.WithPrompt("%s", prompt)) if err != nil { return res, err } res.Design = *d out, resp, err := genkit.GenerateData[DesignReview](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has a feature request and a design written for it."), ai.WithPromptParts(ai.NewDataPart(map[string]any{"request": request, "design": res.Design}))) if err != nil { return res, err } // jev's verdict, then our rules on top. verdict := string(out.Verdict.Choice) var notes []string if out.RequestClear.Probability < 0.4 { verdict = "reject" // nobody should design from a request two engineers would read differently } if out.ScopeCreep.Probability >= 0.5 { notes = append(notes, "remove everything the request did not ask for") } if out.Underspecified.Probability >= 0.5 { notes = append(notes, "specify every component fully: fields, signatures, edge cases and error behaviour") } if out.MissingTests.Probability >= 0.5 { notes = append(notes, "add a test case for every requirement in the request") } if verdict == "ready" && len(notes) > 0 { verdict = "revise" } res.Rounds = append(res.Rounds, DesignRound{Round: round, Verdict: verdict, Ready: out.Verdict.Probabilities["ready"], Feedback: notes /* ... */}) res.Verdict = verdict if verdict != "revise" { break } feedback = fmt.Sprintf("The reviewer rated the design %q (probability of ready %.2f). Do the following:\n- %s", verdict, out.Verdict.Probabilities["ready"], strings.Join(notes, "\n- ")) } return res, nil }) Two things to notice. The verdict is jev's, but the rules around it are yours: a request that two engineers would read differently is rejected whatever the design looks like, and a design with a flagged problem is not ready even if jev leaned that way. And the feedback is built from the questions, not from prose: a decision model never writes a sentence, so the sentences live in your code, one per question, and the next round gets exactly the ones that applied. The whole loop is one flow, so it is one senro step, one pod, one log, with every round recorded in design.json. Step 3: Plan, Implement, and Let Go Have Its Say The plan and the implementation are ordinary Gemini calls with typed outputs: a Plan with files and steps, then a Code with every file in full. The step writes the files into the workspace, refusing anything that would escape it: Go type Code struct { Files []File `json:"files"` Notes string `json:"notes,omitempty"` } func writeCode(ctx senro.Ctx, code Code) error { root := wsPath(ctx, ".") for _, f := range code.Files { rel := filepath.Clean(f.Path) if rel == "." || strings.HasPrefix(rel, "..") || filepath.IsAbs(rel) || strings.HasPrefix(rel, factoryDir) { return fmt.Errorf("refusing to write outside the module: %q", f.Path) } path := filepath.Join(root, rel) if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { return err } if err := os.WriteFile(path, []byte(f.Content), 0o644); err != nil { return err } fmt.Fprintf(ctx.Stdout(), "wrote %s (%d bytes)\n", rel, len(f.Content)) } return nil } Then the compiler and the tests run, as command steps in the same pod image, with no model anywhere near them: Go func goStep(w *senro.WorkflowBuilder, id, after, cmd string, ws *senro.WorkspaceRef) *senro.StepBuilder { script := "mkdir -p .factory; " + cmd + " > .factory/" + id + ".log 2>&1; echo $? > .factory/" + id + ".exit; " + "cat .factory/" + id + ".log; exit $(cat .factory/" + id + ".exit)" return w.Step(id, exec.Command("sh", "-c", script)). Needs(after). Mount(ws.At("/repo", senro.RW)). WorkDir("/repo"). Env("HOME", "/tmp"). Env("GOCACHE", "/tmp/gocache"). Env("GOFLAGS", "-mod=mod"). Env("GOPROXY", "off"). Env("GOTOOLCHAIN", "local"). Env("CGO_ENABLED", "0"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). ContinueOnError() } goStep(code, "vet", "implement", "go vet ./...", ws) goStep(code, "test", "vet", "go test -count=1 ./...", ws) Each one leaves its output and exit code under .factory/ in the workspace, so the reviewer can read them. GOPROXY=off and GOTOOLCHAIN=local keep the pod honest: the request said standard library only, and the pod cannot download anything to pretend otherwise. And ContinueOnError is the important one: a red test is a failed step, but the review still runs. jev gets to see why the build failed, which is exactly the evidence a rework needs. Step 4: The Code Review Gate The second gate reads everything a human reviewer would open: the request, the plan, every source file, and what go vet and go test printed. Go type CodeVerdict string func (CodeVerdict) Criteria() map[CodeVerdict]string { return map[CodeVerdict]string{ "ship": "Builds, tests pass, does what the request asked, no obvious defects", "rework": "Fixable problems: failing or missing tests, a missed requirement, sloppy code", "human": "Something a model should not decide: security, data loss, or the request was misunderstood", } } type Quality int func (Quality) Levels() []string { return []string{"Poor", "Acceptable", "Good", "Excellent"} } type CodeReview struct { Verdict systemonex.Choice[CodeVerdict] `json:"verdict" jsonschema_description:"Should this change ship?"` Quality systemonex.Score[Quality] `json:"quality" jsonschema_description:"How good is the code?"` CoversRequest systemonex.Noul `json:"covers_request" jsonschema_description:"Does the implementation do everything the request asked for?"` TestsMeaningful systemonex.Noul `json:"tests_meaningful" jsonschema_description:"Do the tests exercise the behaviour the request cares about, rather than just compile?"` BuildGreen systemonex.Noul `json:"build_green" jsonschema_description:"According to the logs and exit codes, did go vet and go test both pass?"` } The flow sends the whole ReviewInput as a data part, reads the answers, and applies the factory's rules on top. A red build never ships, whatever the overall verdict: Go f.Review = genkit.DefineFlow(g, "reviewChange", func(ctx context.Context, in ReviewInput) (Review, error) { out, resp, err := genkit.GenerateData[CodeReview](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has the feature request, the implementation plan, every source file, "+ "and the output and exit codes of go vet and go test. Exit code 0 means success."), ai.WithPromptParts(ai.NewDataPart(in))) if err != nil { return Review{}, err } r := Review{ Verdict: string(out.Verdict.Choice), Confidence: out.Verdict.Confidence, Quality: out.Quality.Label(), CoversRequest: out.CoversRequest.Probability, TestsMeaningful: out.TestsMeaningful.Probability, BuildGreen: out.BuildGreen.Probability, Model: systemonex.ResponseInfo(resp).Model, } if in.VetExit != 0 { r.Reasons = append(r.Reasons, "go vet failed") } if in.TestExit != 0 { r.Reasons = append(r.Reasons, "go test failed") } if r.CoversRequest < 0.6 { r.Reasons = append(r.Reasons, "the implementation does not cover the request") } if r.TestsMeaningful < 0.5 { r.Reasons = append(r.Reasons, "the tests do not exercise the requested behaviour") } if out.Quality.Level() == 0 { r.Reasons = append(r.Reasons, "code quality is poor") } if r.Verdict == "ship" && len(r.Reasons) > 0 { r.Verdict = "rework" } return r, nil }) The Reasons are what the fix flow gets, together with the logs. Again: the decision model decides, your code says it in words. Step 5: Three Pipelines and a State Machine The module under construction lives in a persistent workspace, so every run of one request works on the same tree. senro insists a persistent workspace is bounded, and refuses one without both bounds: Go func repo() *senro.WorkspaceRef { return senro.Workspace("factory-repo", senro.Scope(senro.ScopePersistent), senro.MaxAge(24*time.Hour), senro.MaxSize(256<<20)) } The three pipelines are short. Every AI step gets the same shape through a helper, with only the secrets it needs: Go func aiStep(s *senro.StepBuilder, ws *senro.WorkspaceRef, secrets ...string) *senro.StepBuilder { s = s.Mount(ws.At("/repo", senro.RW)).Timeout(stepTimeout).Retry(3, retry.OnInfra()) for _, name := range secrets { s = s.SecretEnv("SENRO_"+name, name) } return s } func buildPipeline(outDir string, rework bool) *senro.Pipeline { name, fn := "implement", "factory/implement" if rework { name, fn = "rework", "factory/fix" } p := senro.New(name) ws := repo() code := p.Workflow("code", senro.On(pods())) aiStep(code.Step(name, senro.Func(fn, NoParams{})), ws, "Gemini") goStep(code, "vet", name, "go vet ./...", ws) goStep(code, "test", "vet", "go test -count=1 ./...", ws) aiStep(code.Step("review", senro.Func("factory/review", NoParams{})), ws, "TypeSafe").Needs("test") handoff := p.Workflow("handoff", senro.Needs("code"), senro.On(senro.Local())) handoff.Step("collect", senro.Func("factory/collect", DirParams{Dir: outDir})). Mount(ws.At("/repo", senro.RO)) return p } finish declares both endings and lets a run parameter pick one, the same senro.When pattern as the previous article: Go func finishPipeline(outDir, reason string) *senro.Pipeline { p := senro.New("finish") ws := repo() ship := p.Workflow("ship", senro.On(pods()), senro.When(senro.ParamIs("verdict", "ship"))) aiStep(ship.Step("pr", senro.Func("factory/pr", NoParams{})), ws, "Gemini") publish := p.Workflow("publish", senro.Needs("ship"), senro.On(senro.Local()), senro.When(senro.ParamIs("verdict", "ship"))) publish.Step("publish", senro.Func("factory/publish", DirParams{Dir: outDir})). Mount(ws.At("/repo", senro.RO)) escalate := p.Workflow("escalate", senro.On(pods()), senro.When(senro.ParamIs("verdict", "human"))) aiStep(escalate.Step("brief", senro.Func("factory/brief", BriefParams{Reason: reason})), ws, "Gemini") // ...and a notify workflow on the coordinator that collects BRIEF.md return p } And main is the state machine. Each run ends with a record jev produced, and the record picks the next run: Go state, reason, reworks := "design", "", 0 for { switch state { case "design": if err := run(designPipeline(request, out)); err != nil { exit(err) } var d DesignResult read(out, "design.json", &d) if d.Verdict == "ready" { state = "implement" } else { state, reason = "human", fmt.Sprintf("the design review ended %q after %d round(s)", d.Verdict, len(d.Rounds)) } case "implement", "rework": // A red vet or test is a failed step, so the run is partial; the // review still ran (ContinueOnError) and its verdict is what counts. if err := run(buildPipeline(out, state == "rework")); err != nil && !partial(err) { exit(err) } var r Review read(out, "review.json", &r) switch r.Verdict { case "ship": state = "ship" case "rework": if reworks >= *maxRework { state, reason = "human", fmt.Sprintf("jev asked for rework %d times: %v", reworks+1, r.Reasons) } else { reworks++ state = "rework" } default: state, reason = "human", fmt.Sprintf("the code review ended %q: %v", r.Verdict, r.Reasons) } case "ship", "human": if err := run(finishPipeline(out, reason), senro.WithParams(senro.Params{"verdict": state})); err != nil { exit(err) } return } } Every arrow in the state diagram is one of those cases, and every run is a directory under runs/ with its plan, its events and one log per step. Step 6: Run It The first request is a token-bucket rate limiter, the kind of package every service ends up with: Plain Text # Feature request: a token-bucket rate limiter package We need a small, dependency-free Go package `ratelimit` (module path `github.com/acme/ratelimit`) that other services can import. - `New(rate float64, burst int) *Limiter`: `rate` is tokens added per second, `burst` is the bucket capacity. - `(*Limiter) Allow() bool`: takes one token if available, otherwise returns false without blocking. - `(*Limiter) AllowN(n int) bool` for batches. - Safe for concurrent use from many goroutines. - Time must be injectable so tests are deterministic (no sleeping in tests). - Standard library only. Tests included, run with `go test ./...`. Shell go run . -request requests/token-bucket.md The first run is the design. Read back with senro attach --run <id> --ui=plain: Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.44) request_clear=0.62 scope_creep=0.36 underspecified=0.56 missing_tests=0.38 risk=Low design stdout | - specify every component fully: fields, signatures, edge cases and error behaviour design stdout | round 2: jev-1.13.0 decided "ready" (p_ready 0.57) request_clear=0.57 scope_creep=0.49 underspecified=0.28 missing_tests=0.34 risk=Low design succeeded plan stdout | module github.com/acme/ratelimit, 3 files, 11 steps plan succeeded collect succeeded run succeeded Two rounds. jev found the first design underspecified (0.56), the feedback said so in one sentence, and the second design named every field, signature, and edge case. The ready probability moved from 0.44 to 0.57, and the design that came out of it is the one Gemini implemented; this is its component list in DESIGN.md: Plain Text ## Components - type Limiter struct { mu sync.Mutex; rate float64; burst float64; tokens float64; last time.Time; now func() time.Time } - concurrency-safe state tracking ... - func New(rate float64, burst int) *Limiter - public constructor delegating to newWithClock using time.Now; clamps negative rate to 0.0 and negative burst to 0 ... - func newWithClock(rate float64, burst int, now func() time.Time) *Limiter - internal constructor enabling time injection for tests; ... panics if now is nil. - func (l *Limiter) Allow() bool - convenience method returning l.AllowN(1) ... - func (l *Limiter) AllowN(n int) bool - thread-safe token deduction; edge cases: returns true immediately if n <= 0 without modifying state; returns false immediately if n > int(l.burst); advances tokens by elapsed.Seconds() * l.rate capped at l.burst ... The plan turns that into three files and eleven steps, and the second run executes it. Plain Text implement stdout | wrote go.mod (42 bytes) implement stdout | wrote ratelimit.go (1371 bytes) implement stdout | wrote ratelimit_test.go (3172 bytes) implement succeeded vet succeeded test stdout | ok github.com/acme/ratelimit 0.009s test succeeded review stdout | jev-1.13.0 decided "ship" (confidence 0.66) quality=Excellent covers_request=0.87 tests_meaningful=0.95 build_green=0.98 review succeeded collect succeeded run succeeded Gemini wrote the module, go vet and go test ran in the pod against it, and jev read the three files and the logs and said ship, with the build at 0.98 and the tests at 0.95. The module it reviewed: Go // AllowN reports whether n tokens can be consumed immediately. func (l *Limiter) AllowN(n int) bool { l.mu.Lock() defer l.mu.Unlock() if n <= 0 { return true } if float64(n) > l.burst { return false } curr := l.now() elapsed := curr.Sub(l.last) if elapsed > 0 { l.tokens += elapsed.Seconds() * l.rate if l.tokens > l.burst { l.tokens = l.burst } l.last = curr } if l.tokens >= float64(n) { l.tokens -= float64(n) return true } return false } with six tests beside it: clamping, a nil clock panic, burst capacity, AllowN edge cases, deterministic replenishment against an injected clock, and a race test over concurrent goroutines. No sleeping anywhere, as the request demanded. The third run has both endings declared and runs one: Plain Text pr stdout | # Implement token-bucket rate limiter package pr stdout | ## Summary pr stdout | Introduces `github.com/acme/ratelimit`, a thread-safe, zero-dependency token-bucket rate limiter. ... pr stdout | ## Tests pr stdout | ok github.com/acme/ratelimit 0.009s pr stdout | ## Risk pr stdout | Low. Mutex contention under high concurrency is mitigated by keeping critical sections limited to basic in-memory arithmetic. ... pr succeeded publish stdout | published 3 files to .../out/token-bucket/src publish succeeded brief skipped_condition notify skipped_condition run succeeded Three senro runs, seven pods, two Gemini designs, two jev design reviews, one implementation, one vet, one test, one jev code review and one pull request, in a little over a minute and a half. out/token-bucket/ holds every record and the module under src/. Step 7: When jev Says No The second request is the kind every team gets: Plain Text # Request The app feels slow lately, especially in the afternoon. Can you make it faster? Nothing specific, just whatever you think is best. Thanks. Shell go run . -request requests/make-it-faster.md Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "reject" (p_ready 0.13) request_clear=0.05 scope_creep=0.87 underspecified=0.70 missing_tests=0.79 risk=Low design stdout | - remove everything the request did not ask for design stdout | - specify every component fully: fields, signatures, edge cases and error behaviour design stdout | - add a test case for every requirement in the request design succeeded plan stdout | design ended "reject", nothing to plan plan succeeded collect succeeded run succeeded Gemini did what a chat model does with a vague request: it invented a feature, in this case pprof routes and a latency middleware. jev put the probability that the request is clear at 0.05, and the factory's rule turned that into reject in the first round, before a plan, a line of code or a second model call. The finish run then takes the other ending: Plain Text pr skipped_condition publish skipped_condition brief stdout | ### What was asked brief stdout | A user submitted a vague performance complaint stating the app "feels slow lately, especially in the afternoon" and asked to "make it faster" without providing specific endpoints, metrics, or reproduction steps. brief stdout | ### Why it stopped brief stdout | The design was rejected on Round 1 (Readiness: 13%, Clarity: 5%). The factory attempted to introduce Go `net/http/pprof` diagnostic routes and custom in-memory latency middleware, triggering a rejection for high scope creep (87%) ... brief stdout | ### What a human should decide brief stdout | 1. **Clarify the issue:** Contact the user to identify affected routes, user workflows, and target latency SLOs. brief stdout | 2. **Authorize observability work:** Determine if the team should prioritize a dedicated ticket for production profiling/metrics ... brief succeeded notify succeeded run succeeded The brief is written by Gemini, from the record jev produced. The two models keep their jobs even in the failure path. A Request jev Cannot Be Talked Into The third request is a glob matcher: *, **, ?, character classes with ranges and negation, backslash escapes, and an error for an unterminated class. It is precise, it is testable, and the factory never got to write it: Plain Text clean succeeded design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.68 scope_creep=0.35 underspecified=0.49 missing_tests=0.32 risk=Medium design stdout | round 2: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.65 scope_creep=0.40 underspecified=0.34 missing_tests=0.22 risk=Medium design stdout | round 3: jev-1.13.0 decided "revise" (p_ready 0.47) request_clear=0.59 scope_creep=0.66 underspecified=0.33 missing_tests=0.20 risk=Medium design stdout | - remove everything the request did not ask for design succeeded Gemini's design transpiles the glob into a Go regular expression. jev never called it ready, the ready probability never crossed 0.5 in three rounds, and in two of those rounds none of the specific questions crossed the line either, so the feedback Gemini got was the generic "tighten the approach". The loop cannot converge on a complaint it has no question for, and after three rounds the factory stops and writes the brief: Plain Text ## What was asked Implement a standard-library Go package (`github.com/acme/glob`) providing `Compile(pattern string) (*Glob, error)` and `(*Glob) Match(path string) bool`. ... ## Why it stopped The design review failed after 3 rounds, ending in a "revise" verdict (`p_ready` plateaued at 0.46). Automated reviewers flagged persistent specification risks and scope creep regarding the chosen approach (transpiling globs into Go `regexp`), specifically around complex regex metacharacter escaping and separator handling in negated character classes. ## What a human should decide 1. **Architecture:** Approve regex transpilation (accepting edge-case escaping risks) or mandate a dedicated AST/rune-based glob parser. 2. **Path boundaries:** Explicitly define exact semantics for `**` edge cases and `/` exclusions inside negated classes. That brief is the factory working as designed. A decision model only answers the questions you ask, and when it keeps saying no without any of them firing, the honest move is to stop and ask a person, with the evidence attached. The fix for the next version of the factory is one more field on DesignReview: a question about the approach itself, so that "this should be a parser, not a regex" becomes a sentence Gemini can act on. What the Combination Gives You ConcernWho handles itDesign, plan, code, fix, pull request textGemini, through genkit.GenerateData and GenerateTextIs the design ready, should the code shipjev, through the same GenerateData call with a decision typeFeedback for the next roundYour code, one sentence per question jev answered "yes"Compiling and testinggo vet and go test, as command steps in the pod, with no model involvedA red build reaching the reviewerContinueOnError on the Go stepsThe module between runsA bounded persistent workspaceWhich ending runssenro.When(senro.ParamIs("verdict", ...))Credentialsmamori resolves, senro delivers a file per pod, per stepEvidenceruns/<id>/ per run, plus .factory/ records that travel with the code Conclusion A software factory is not one big agent. It is a handful of small, typed calls with a judge between them, and the judge is the part most people leave out. jev makes that part cheap: a design review is one call that answers six typed questions in a few hundred milliseconds, and the questions are Go fields you choose. Put the loop inside a Genkit flow, the flow inside a senro step, the step in a pod, and the verdict in a run parameter, and the factory is a state machine you can read in one screen. The code: genkit-k8s-flows-jev-software-factoryThe previous article: Genkit flows on Kubernetes, routed by a decision modelDecision models in Genkit Go and Flowssenro's Kubernetes executor and persistent workspaces

By Xavier Portilla Edo DZone Core CORE
5 Mistakes Companies Make When Working With Staff Engineers
5 Mistakes Companies Make When Working With Staff Engineers

Hiring a great Staff Engineer is challenging. Identifying candidates with the right mix of technical expertise, architectural vision, communication skills, and leadership experience demands considerable effort. However, success depends not only on hiring the right person but also on how you support them afterward. While Staff Engineering is not new, the role has become more prominent in the software industry, especially following the publication of Will Larson’s Staff Engineer: Leadership Beyond the Management Track in 2021. Since then, conversations about technical leadership outside traditional management have increased. However, many organizations that have adopted the Staff Engineer title still lack the organizational maturity to support the role effectively. The challenge is that Staff Engineering isn't just about hiring someone who writes better code or designs sophisticated architectures. It is focused on empowering technical leaders to influence people, decisions, and the organization as a whole. Without trust, autonomy, collaboration, and clear expectations, even experienced engineers may struggle to make a meaningful impact. In this article, we examine five common mistakes companies make when managing Staff Engineers, such as limiting their decision-making authority or undervaluing the time organizational change takes. We will also discuss why these mistakes happen and how organizations can create an environment that enables Staff Engineers to deliver the value they were hired to provide. Staff Engineering: A Role Older Than Software Staff Engineering originated outside the technology industry, with roots in the 19th-century British Royal Navy. There, Staff Engineers were engineering officers responsible for specialized technical tasks. As steam-powered ships grew more complex, the Navy relied on these experienced engineers to maintain critical machinery. These specialists often faced challenges related to recognition, authority, and their relationship with traditional military leadership. In the software industry, Staff Engineering has become a technical leadership path beyond Senior Engineer. This role enables experienced professionals to expand their influence without moving into management. Staff Engineers guide architectural decisions, set technical direction, mentor engineers, and coordinate efforts across teams. The role gained visibility through companies with technical career ladders and publications like Will Larson’s Staff Engineer: Leadership Beyond the Management Track (2021). As distributed systems, cloud computing, and large engineering organizations have grown more complex, the Staff Engineer role has become increasingly important. However, adopting the title is not the same as using it effectively. Many organizations acknowledge the need for Staff Engineers but may not fully understand how to empower them. Notably, some challenges faced by technical specialists over a century ago remain relevant today. Mistake #1: Hiring Expertise Without Trusting It Consider a football club that invests millions to hire top players, only to limit their strengths, question their decisions, and dictate their play. When results disappoint, the club blames the investment. This contradiction is common in software engineering organizations. Companies invest heavily in Staff Engineers for their experience, architectural knowledge, and problem-solving skills. Yet these engineers often find their expertise isn't trusted. Their recommendations face excessive approval requirements; they are excluded from key discussions, and established practices go unchallenged, even when improvements are needed. The contradiction is clear: hiring for expertise is ineffective if the organization does not use it. Trust does not require accepting every recommendation or granting unlimited authority. Technical decisions should be constructively challenged, evidence-based, and aligned with business goals. However, a healthy debate differs significantly from a culture where experienced engineers must constantly justify their value. Organizations must recognize that hiring a Staff Engineer is an investment in technical judgment and leadership, not simply adding another developer. Without an environment that respects and constructively challenges their expertise, organizations will likely waste this investment. Mistake #2: Treating Staff Engineering as a Purely Technical Role A common misconception is that Staff Engineers add value mainly through coding, code reviews, or designing architectures. While technical expertise is essential, Staff Engineers must also influence people, align teams, and improve engineering decisions. When organizations face cross-team architectural challenges, Staff Engineers can identify problems and propose solutions. If excluded from key discussions, their impact is limited. Often, they are brought in after decisions are made, leaving them to address issues they had little opportunity to prevent. Staff Engineers should not attend every meeting or discussion. Their time is valuable, and unnecessary meetings reduce effectiveness. Involve them in conversations where they can influence technical direction, resolve disagreements, connect teams, or prevent costly mistakes. Their role is to shape decisions when their expertise is relevant. Mentoring and developing other engineers is also essential. A Staff Engineer who improves the technical capabilities of ten engineers generates far more organizational value than one focused solely on personal productivity. Key responsibilities include sharing knowledge, creating technical alignment, and helping teams become more independent. This is how technical leadership scales beyond individual contributions. Staff Engineers should be evaluated not only on their personal output, but also on the improvements they enable in others. Ignoring this risks reducing their role to that of expensive Senior Developers and limits their broader impact. Mistake #3: Giving Responsibility Without Decision-Making Authority Consider hiring a Staff Engineer to improve scalability, reduce technical debt, or modernize outdated architecture. The organization expects this professional to identify problems, propose solutions, and take responsibility for technical outcomes. However, the engineer often finds that most significant changes require approval from those who may not fully understand the technical implications. This creates a contradiction: the Staff Engineer is accountable for outcomes but lacks the authority to influence key decisions. This challenge intensifies when organizations exhibit status quo bias, preferring familiar practices over necessary change. A Staff Engineer may identify outdated architectures or inefficient practices, yet recommendations are sometimes dismissed not for technical or business reasons, but simply because “we have always done it this way.” Over time, this resistance stifles innovation and makes improvement more difficult. Autonomy does not mean Staff Engineers should make every decision alone. Architecture requires balancing trade-offs, business priorities, financial constraints, and collaboration. However, there is a key difference between collaborative decision-making and needing approval for every technical change. Staff Engineers need clear responsibilities, meaningful authority, and the ability to challenge existing approaches through evidence and constructive dialogue. Organizations that expect Staff Engineers to take ownership must also grant them the authority to do so. Without this, the role becomes a title with high expectations but limited influence. Mistake #4: Confusing Management Authority With Technical Leadership A common mistake is failing to define clear boundaries between management authority and technical leadership. Engineering Managers, Directors, and Staff Engineers have distinct but complementary roles. Management oversees people development, priorities, resources, and delivery, while Staff Engineers provide technical expertise for architecture, engineering practices, and long-term direction. Problems occur when organizations assume management hierarchy equates to superior technical judgment. For example, a Staff Engineer may identify a major architectural risk and recommend an alternative based on technical evidence. If a manager overrides this recommendation based on personal preference or hierarchy, the issue is not who is right or wrong. The real problem arises when technical decisions are made by organizational position rather than engineering evidence, trade-offs, or business objectives. Not every manager or director has the expertise to evaluate complex technical decisions, and this is not inherently problematic. Leadership does not require being the most technically knowledgeable. The challenge arises when leaders do not recognize their own limits or fail to create space for technical specialists to contribute. Effective managers know when to decide, when to facilitate, and when to delegate technical authority. Staff Engineers should also recognize that technical excellence does not grant authority over organizational priorities. A technically superior solution may not align with current business needs. Effective technical leadership means understanding these constraints and collaborating with management, not viewing it as an obstacle. Successful organizations do not pit management and technical leadership against each other. Instead, they create partnerships with clear responsibilities, mutual respect for expertise, and decisions informed by both technical and business perspectives. Without this partnership, Staff Engineers may spend more time on organizational politics than on solving engineering problems. Mistake #5: Expecting Immediate Results From Organizational Change Deploying an application may take minutes, but changing engineering culture can take months or years. Some companies hire Staff Engineers expecting immediate solutions to deep-rooted problems. This overlooks a key reality: organizational change requires time. When joining a new organization, a Staff Engineer must first understand its technical landscape, business priorities, processes, and relationships. They also need to learn the reasons behind past decisions, as choices that seem poor today may have been justified by earlier constraints. Without this context, proposing major changes can cause resistance or new issues. The first two or three months should be used to observe, listen, build relationships, identify risks, and understand the organization. Staff Engineers should still deliver value during this time. Small improvements, targeted recommendations, and early collaboration help build credibility and trust. However, early technical wins are different from achieving lasting organizational transformation. The challenge is greater in organizations with entrenched practices or cultures resistant to change. Technical improvements often require shifts in behavior, team roles, decision-making, and leadership expectations. A Staff Engineer may identify the right direction, but successful implementation depends on collaboration, communication, and others’ willingness to engage. Organizations should assess Staff Engineers based on both short-term contributions and long-term impact. Early progress may include identifying risks, improving collaboration, or setting technical direction. Over time, outcomes such as better maintainability, stronger teams, reduced complexity, and improved decision-making can emerge. There is no standard timeline for these results. Progress depends on organizational size, technical complexity, existing processes, leadership support, and culture. Expecting overnight transformation from a Staff Engineer reflects a misunderstanding of sustainable improvement. Conclusion Hiring a Staff Engineer is a significant investment, but the real challenge starts after hiring. Technical expertise alone cannot overcome a lack of trust, limited autonomy, exclusion from key discussions, or unrealistic expectations for rapid change. Staff Engineers can drive improvements, but their success depends on an environment that supports them. Ultimately, Staff Engineering is not just about hiring the right people; it is about creating an organization where they can make a difference. Companies that recognize this are more likely to achieve lasting impact. The key question is not whether you have Staff Engineers, but whether your organization is prepared to let them succeed.

By Otavio Santana DZone Core CORE
Your Event Storming Is Not Your Software Design
Your Event Storming Is Not Your Software Design

The Policy That Started the Argument We were looking at a purple policy between ListingSold and StopListingFee. Read aloud, the flow expressed a simple business expectation: when a listing is sold, the applicable recurring listing fee should stop. The interesting question was where that policy belonged. If Billing owns it, Billing receives the fact that a listing was sold and determines what that means for its fee model. Listing reports the sale without naming the operation Billing should perform. If Listing owns the coordination, it requests that Billing stop the fee, which means Listing knows that capability exists. A third option is a separate process manager that receives the sale event and coordinates the response. At the level of a simple event-policy-command sequence, these versions can look almost identical. Once ownership is made explicit, the differences matter. Billing depends on the sale event’s contract in the first version. The coordinating component depends on Billing’s command contract in the others. Billing still validates changes to its own fee model in every version. Responsibility for detecting an incomplete outcome, such as a fee that continues accruing after a sale, also needs an explicit owner. We were still discussing the business rule, but we were now deciding how software responsibilities and contracts would support it. This was already a discussion about where coupling belongs and how it is managed. We had moved into architectural reasoning without changing the business outcome we wanted. I later discussed this on LinkedIn and encountered the objection that EventStorming is not software design at all. I disagree, but the objection helped me distinguish two claims. EventStorming can host software design. It does not follow that the arrangement of a board is sufficient justification for the boundaries of the resulting system. This article is about that gap. EventStorming Already Includes Design The official EventStorming material explicitly recognizes Big Picture, Process Modeling, and Software Design formats. The patterns collection also addresses collaborative design activities. EventStorming is not limited to collecting domain events on a long wall of paper. The title therefore makes a narrower claim than it might suggest. “Your Event Storming” means the model your group produced, with its particular clusters, policies, and arrows. That model may contain exploratory ideas, agreed decisions, and reasons for those decisions. What cannot be assumed is that the notation alone captures all the reasoning needed to justify the resulting software structure. A policy between two notes expresses a relationship: when this happens, that response should follow. Unless ownership and the relevant assumptions are also made explicit, the sequence leaves important questions unanswered. Who owns the rule? Which contract does the response depend on? What happens if the response is rejected or never completes? A Software Design session can address those questions. It can expose invariants, clarify responsibilities, and help participants explore alternative models. The policy discussion above is an example of that work. The problem is automatic inference: treating a cluster as sufficient reason for a service, a purple note as a prescribed message handler, or a temporal sequence as an execution topology. This is related to the architectural decisions a domain model leaves open. Modeling can establish constraints and support architectural choices without uniquely determining how the system should be deployed or operated. Those choices can be discussed during the workshop. They still need reasons beyond the appearance of the board. Same Flow, Different Ownership For this example, Listing owns the listing lifecycle, and Billing owns the fee lifecycle. A sold listing should no longer accrue the applicable recurring fee. The precise rule, especially when stopping accrual becomes effective, still needs clarification. For now, consider three ways to allocate responsibility for the response. Alternative A: Billing owns the reaction. Listing records the sale and makes a ListingSold event available across an agreed boundary. Billing receives that fact, interprets it using its own rules, and invokes the appropriate fee behavior. Listing does not need to name StopListingFee. Billing now depends on the meaning and structure of the event it consumes. The event publisher needs an explicit compatibility policy, although that does not require exposing Listing’s internal domain event unchanged. Billing keeps the relationship between sale facts and fee behavior within its own model. That is useful when the rule belongs to Billing and evolves with its charging policies. The design still needs to account for delivery failures, processing failures, and missed reactions. Suppose Listing records a sale but the corresponding fee remains active. Who detects that discrepancy, and who resolves it? Billing may own both the reaction and reconciliation, but that responsibility must be explicit. Successful publication alone does not establish that the business outcome is complete. Alternative B: Listing owns coordination. Listing requests StopListingFee from Billing. Listing therefore knows a capability Billing exposes. Billing remains responsible for validating the request and protecting its own rules. The request could be synchronous or queued. Either way, the coordinating component needs to distinguish acceptance from completion and decide how to handle rejection, an unknown outcome, or a response that does not arrive within the expected time. This option is justified only if Listing’s responsibility genuinely includes coordinating the fee change. Emitting the original event is not enough to establish that ownership. Nor does the statement “the sale is complete only when the fee stops” settle the question by itself. That requirement establishes a business outcome; deciding which component should be accountable for achieving it is still design work. Listing-owned coordination becomes more plausible when Listing already owns the broader completion process and can manage it without absorbing Billing’s fee rules. If it starts calculating charges or reproducing Billing’s eligibility logic, the coordination boundary is leaking. Alternative C: A process manager owns coordination. A separate component tracks the cross-component outcome and requests work from the responsible participants. Listing still owns the listing lifecycle. Billing still owns fee behavior. The process manager owns the progress of the interaction. This helps when a sale triggers several participants, steps have deadlines, or unresolved cases need escalation. The manager can record that the sale was observed, that a fee-stop request was issued, and whether completion was confirmed. That visibility has a cost. The manager needs durable state, recovery behavior, and a policy for its own failures. Its commands may use the same capabilities Billing already exposes; participants do not necessarily need a new interface simply because a process manager is calling them. For a single reaction, introducing this machinery may be harder to justify than making Billing’s existing responsibility explicit. DesignMessage crossing the boundaryContract knowledgeAccountability for completionBilling reactsListingSold eventBilling understands the published sale contractBilling can own reaction and reconciliation; this must be agreedListing coordinatesStopListingFee command and an appropriate completion signalListing understands Billing’s exposed capabilityListing tracks the outcome if coordination belongs within its responsibilityProcess manager coordinatesSale event, fee-stop command, and completion or failure signalsThe manager understands the contracts required by the processThe manager tracks progress and escalation; its own recovery must also be designed These alternatives allocate responsibility differently. They do not divide neatly into a coupled design and a decoupled one. Contract knowledge is one form of dependency. Billing understands the sale event in Alternative A. Listing understands Billing’s command contract in Alternative B. The process manager understands the contracts needed to coordinate Alternative C. Runtime availability is a separate concern. A synchronous call may require the responder to be available immediately. Buffered communication can allow the sender to proceed while the receiver is unavailable, provided the system can retain and eventually process the pending work. That changes the timing of the dependency rather than removing it. Release coordination is another concern. Consumers and publishers need compatible contracts, but they do not necessarily need simultaneous deployments. The relevant question is how those contracts can evolve without silently changing the business meaning. Moving the policy changes some of these dependencies. It does not automatically reverse all of them. Nor does its position prescribe a transport: an event does not require Kafka, and a command does not require HTTP. What would favor one option? If Billing independently owns the rule connecting a sale to fee accrual, Alternative A is a strong candidate. If Listing already owns a broader completion process that legitimately includes requesting the fee change, Alternative B may fit. If the interaction needs its own lifecycle, deadlines, and recovery across several participants, Alternative C becomes easier to justify. The business flow is an input to that decision. Ownership, contract obligations, and failure handling determine whether the chosen design can support it. The Missing Question Is Often About Time The sequence ListingSold → StopListingFee leaves an important question unresolved: when should accrual stop? The policy note alone does not specify that. It could be the effective sale time, when the sale is recognized by the business. It could be when Listing records the change, or when Billing processes it. Those moments can differ. If the contract carries timestamps, their meanings need to be explicit. For example, a published event might distinguish the effective time from the time the sale was recorded: JSON { "type": "ListingSold", "eventId": "...", "listingId": "...", "effectiveAt": "...", "recordedAt": "..." } These fields make the distinction visible. They do not establish which timestamp Billing should trust or what it should do with it. If Billing calculates accrual using an authoritative effective time, delayed delivery need not change the amount ultimately owed. But the system may already have issued an invoice, collected payment, or shown an outdated balance. It still needs rules for adjustment and for what the customer sees before the correction completes. A timestamp helps preserve the meaning of the fact. It does not remove the consequences of learning about it late. Suppose the business accepts a delay before the balance reflects a sale, provided the final calculation uses the correct effective date. An asynchronous reaction may satisfy that requirement. Suppose instead that no fee-related charge may be issued after the sale is accepted. That requires a stronger coordination or validation mechanism than receiving a sale event later. Merely replacing an event with a queued command does not provide that guarantee either. Corrections expose further decisions. If the sale was recorded incorrectly, should accrual resume? From which date? If an older sale notification arrives after a correction, how does Billing recognize that it no longer represents the applicable state? These questions concern the business meaning of the update as much as its delivery. This is where business outcomes beyond aggregate boundaries become relevant. A process spanning Listing and Billing does not automatically require one aggregate or one distributed transaction. Billing can protect a local invariant, such as refusing to calculate accrual beyond its recorded stop date, while the broader process still fails to establish that date correctly. The cross-component outcome needs explicit completion and recovery semantics. Compensation is another business operation, with its own rules and potential failures, rather than a database rollback. Eventual consistency does not by itself establish that the relevant rule was interpreted correctly or that unresolved work will be detected and repaired. Durable publication, retries, and idempotent handling support the design. They cannot supply a business rule the participants have not agreed on. A Boundary Still Needs a Reason An event on the board is not automatically an integration event. ListingSold may initially describe a fact inside the Listing model. Exposing that fact to another component creates a contract that needs deliberate semantics and an evolution policy. The published representation may differ from the internal event. Likewise, a cluster of activities does not establish an aggregate merely because the notes appear together. The grouping must be examined against the rules that must remain consistent. An aggregate does not prescribe a microservice, and a bounded context does not prescribe an independent deployment. These distinctions are part of why Domain-Driven Design does not select an architectural style. The ownership alternatives also exist inside a monolith. Listing and Billing could be modules in one application, with Billing reacting to an in-process event or receiving an explicit request through a module interface. The questions about rule ownership and contract knowledge remain. The execution guarantees may be quite different, however. A synchronous handler participating in the same database transaction can fail together with the originating operation. A background handler in the same application may run after that transaction commits and require durable delivery and recovery. Sharing a deployment does not settle those choices, just as separating deployments does not settle ownership. An independent deployment therefore needs its own justification. Billing might have a materially different scaling profile, need isolation from listing traffic, or require a separate security boundary. Its team might need genuine release autonomy. A runtime constraint might be difficult to satisfy within the existing application. These are reasons to evaluate separation, rather than automatic instructions to introduce it. The costs are concrete. An interaction that was previously local may now time out or complete without the caller receiving confirmation. Contracts need to evolve while different versions coexist. Observability must follow a business outcome across processes. Data previously updated within one transaction may require a different consistency strategy. Independent deployment can reduce some coordination costs while introducing others. I would not impose a fixed sequence of aggregates first, contexts second, services last. Domain learning and architectural constraints inform each other. A scaling or compliance requirement can change where responsibilities execute or force us to reconsider a proposed partition. It does not automatically change the meaning of the business model. When a boundary changes, the team should be able to explain which concern required the change. What I Would Record Beside the Wall I would capture a short decision record while discussing each significant policy, rather than leave all the reasoning for someone to reconstruct after the workshop. First, who owns the rule connecting the fact to the response? In this example, does Billing own the rule relating a sale to fee accrual, or does a broader business process require that outcome? Billing owns the fee mutation in either case. The separate question is who initiates the response and tracks its completion. Next, which contract crosses the boundary? Who maintains it, and what changes can its consumers tolerate? A note on the wall becomes an integration commitment only when the participants agree on the meaning and obligations it carries. The record should also identify what must remain consistent, the acceptable delay between steps, and the effective time used when information arrives late. It should say who detects an incomplete outcome, who resolves it, and what counts as evidence of completion. Acceptance scenarios make those decisions harder to leave vague. The official Extract Acceptance Tests pattern connects design sessions with more precise expectations about behavior. For this example, I would include a sale received after an invoice was issued, a correction received before an older sale event, and a fee-stop request whose completion cannot immediately be confirmed. The expected result in each case needs to come from the business rule. The record should remain short enough to maintain. Its purpose is to preserve the reasons behind the model, including decisions deliberately left open. Six months later, another engineer should be able to distinguish an agreed constraint from a convenient assumption. Closing The discussion began with ListingSold, a purple policy, and StopListingFee. Asking who owned the policy led us to contracts, coordination, consistency, and recovery. That was software design taking place through EventStorming. The workshop can help a team make those decisions, and the board can record them. What the event sequence cannot provide on its own is the justification for a particular allocation of responsibilities or a particular deployment boundary. Billing might own the reaction. Listing might coordinate it. A process manager might track a broader outcome. Choosing among them requires knowing the rule, the timing requirements, and who is accountable when the expected response does not happen. The wall can show the flow. Your design must explain who owns each decision and what happens when the flow breaks. References and Further Reading Introducing EventStormingEventstorming PatternsExtract Acceptance TestsWhere Does Software Architect Fit in Domain-Driven Design?Modern Software Architecture (Part 2): Coupling Is Where Architecture Actually HappensDomain-Driven Design Beyond Aggregate BoundariesDomain-Driven Design Is Not an Architecture

By Alireza Rahmani Khalili DZone Core CORE
Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go
Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go

Genkit flows as senro steps showed how to put a Genkit flow inside a senro step and get retries, caching, and an event stream for every model call. Every step in that article ran on your own machine. This one moves the steps into Kubernetes and adds a second kind of model. Gemini writes. jev decides. jev is a decision model: it does not generate text; it evaluates a state against typed questions and answers each one with a probability, in a few hundred milliseconds. Genkit Go 1.14 reaches it through the System One plugin, and a decision is one genkit.GenerateData call whose output type is the question set. That makes jev the right model to sit at the switch points of a pipeline. In this guide, it decides twice: which department's workflow runs for a support ticket, and whether the reply Gemini drafted is safe to send. Everything below runs against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5, and a local kind cluster on Kubernetes 1.37. The code is in genkit-k8s-flows-jev. Why a Decision Model Decides, and a Chat Model Writes Asking a chat model "which team should own this ticket, answer with one word" works until it does not: you get prose around the word, a department that is not on the list, no idea how sure the model was, and a second or two of latency for a decision that should take milliseconds. A decision model is built for exactly that call. Chat model as routerDecision model as routerAnswerText you parseA typed field: Choice[Dept], Noul, Score[L]OptionsIn the prompt, hoped for in the outputIn the Go type, enforced by the schemaConfidenceNone, or whatever the model says about itselfA probability per option, calibrated per model versionToolsCan be called by accidentRefused by the protocolLatencySecondsA few hundred millisecondsCost per decisionTokens for the state and the proseTokens for the state And the reverse is just as true. jev cannot write the reply to the customer, explain an outage, or summarize a brief. That is what gemini-3.8-flash is for. The pipeline needs both, and senro is what puts each call in its own step, on its own pod, with its own log, retry policy, and secret. What We Are Building A support-ticket pipeline. A raw email comes in; a reply goes out, or a brief goes to a human, or both. Two runs, because of one rule in senro: a When condition is evaluated once, at run start, against facts the run already has, never against something a step produces. So the first run understands the ticket and decides, and the decision becomes the run parameters of the second run, where senro.When(senro.ParamIs(...)) prunes every branch the verdict did not pick before anything starts. Pruned steps settle as skipped_condition and the run stays green. RunStepWhereModelWritestriageseedcoordinatornoneputs the raw email into the run's workspacetriageingestpodGeminiticket.json, a structured tickettriagetriagepodjevdecision.json: department, urgency, frustration, needs a humantriagecollectcoordinatornonecopies both files out of the runresolve<dept>/draftpodGeminireply.md, following that department's playbookresolve<dept>/verifypodjevverdict.json; a rejection fails the stepresolve<dept>/sendcoordinatornonethe reply, only if verify passedresolveescalate/briefpodGeminibrief.md for a human, when jev said so Inside a branch, jev is a guardrail: verify reads the draft against the playbook, and if it rejects the draft, the step fails and senro skips send. The second decision also determines the next step, this time without leaving the run. What You Need Go 1.27 or newer, Docker, kind, and kubectlGEMINI_API_KEY and TYPESAFE_API_KEY in your environmentthe senro CLI if you want the terminal UI (optional) Shell go get github.com/firebase/genkit/[email protected] go get github.com/xavidop/[email protected] go get github.com/xavidop/mamori mamori is the library senro uses to resolve secrets. It is how the two API keys reach the pods without ever being in a pod spec. Step 1: A Cluster senro Can Reach senro's Kubernetes executor never reads KUBECONFIG, ~/.kube/config or your current context. A kubeconfig typically holds a dozen contexts, most of them production, with whichever one you last used selected; a pipeline that inherited that would eventually deploy into the wrong cluster. Instead, the cluster comes from three environment variables, read at run start: VariableWhat it isSENRO_K8S_SERVERThe apiserver URLSENRO_K8S_CA_FILEThe cluster CA, as a PEM file. senro will not skip verificationSENRO_K8S_TOKEN_FILEA bearer token in a file (or a client certificate and key) So a local cluster needs four things: kind, a namespace, an identity with the permissions the executor documents, and those three files. The repo's scripts/cluster-up.sh does all of it; this is what it runs. Shell kind create cluster --name senro-ai --kubeconfig .kind/kubeconfig export KUBECONFIG=$PWD/.kind/kubeconfig kubectl apply -f k8s/rbac.yaml The RBAC is the exact set senro needs and nothing more: in the namespace, create, get and delete on pods, get on pods/log, create and get on pods/exec, and create, patch, delete on secrets. At cluster scope, list and get on nodes, which is how senro reads the platform it has to compile for. The get on pods/exec is the one to remember: senro uses the WebSocket exec protocol, which is a GET on that subresource, and a Role with only create fails every step that carries a workspace. YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: senro-steps namespace: support rules: - apiGroups: [""] resources: ["pods"] verbs: ["create", "get", "delete"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create", "get"] - apiGroups: [""] resources: ["secrets"] verbs: ["create", "patch", "delete"] Then the three files, pulled out of the kubeconfig kind wrote and a short-lived ServiceAccount token: Shell kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.server}' > .kind/server kubectl config view --minify --raw \ -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > .kind/ca.crt kubectl -n support create token senro --duration=4h > .kind/token export SENRO_K8S_SERVER=$(cat .kind/server) export SENRO_K8S_CA_FILE=$PWD/.kind/ca.crt export SENRO_K8S_TOKEN_FILE=$PWD/.kind/token A file rather than SENRO_K8S_TOKEN inline, because an environment variable is visible in /proc/<pid>/environ and inherited by every child process the coordinator spawns. Step 2: The Flows Five flows, in one file. Nothing here knows about senro or Kubernetes. Gemini Turns an Email into a Ticket Go type Ticket struct { Customer string `json:"customer"` Company string `json:"company,omitempty"` Summary string `json:"summary"` Ask string `json:"ask"` Facts []string `json:"facts"` } f.Ingest = genkit.DefineFlow(g, "ingestTicket", func(ctx context.Context, raw string) (Ticket, error) { t, _, err := genkit.GenerateData[Ticket](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You turn raw support emails into a structured ticket. "+ "Facts are identifiers, amounts, dates and error codes quoted from the email, nothing inferred."), ai.WithPrompt("Email:\n\n%s", raw)) if err != nil { return Ticket{}, err } if t == nil { return Ticket{}, fmt.Errorf("ingest: the model returned no ticket") } return *t, nil }) jev Decides What the Ticket Is With the System One plugin, the output type is the question set. Each field is one question; its jsonschema_description tag is the instruction, and the options, levels, and criteria belong to Go types, so the same question means the same thing wherever it is asked. Go import systemonex "github.com/firebase/genkit/go/plugins/systemone/exp" type Dept string func (Dept) Criteria() map[Dept]string { return map[Dept]string{ "billing": "Charges, invoices, refunds, subscriptions, plan changes", "technical": "Bugs, errors, outages, latency, integrations, webhooks, API", "account": "Login, password, permissions, data export or deletion, privacy", "other": "None of the above", } } type Frustration int func (Frustration) Levels() []string { return []string{"Calm", "Annoyed", "Angry", "Furious"} } type Urgent struct{} func (Urgent) Criteria() (yes, no string) { return "Names a deadline, says now or today, or describes an outage in progress", "No time pressure is expressed, or the customer says there is no rush" } type Triage struct { Department systemonex.Choice[Dept] `json:"department" jsonschema_description:"Which team should own this ticket?"` IsUrgent systemonex.NoulOf[Urgent] `json:"is_urgent" jsonschema_description:"Does the ticket communicate time pressure?"` Frustration systemonex.Score[Frustration] `json:"frustration" jsonschema_description:"How frustrated is the customer?"` NeedsHuman systemonex.Noul `json:"needs_human" jsonschema_description:"Does resolving this need a human: a legal threat, a contract or SLA dispute, data loss, or a refund outside the playbook?"` } Four question kinds cover the protocol. Choice[T] picks one of T.Criteria() and gives a probability per option plus a Confidence. Score[L] rates on L.Levels() and answers Level() and Label(). Noul is "is this true?" as a probability, where 0.5 means the model could not tell. NoulOf[C] is the same with C.Criteria() saying what yes and no mean. The flow sends the ticket as a data part, so the state is a JSON object and a question can refer to a field by name. It then turns the four answers into a flat verdict, with thresholds that belong to the caller: Go type Decision struct { Department string `json:"department"` Confidence float64 `json:"confidence"` Urgent bool `json:"urgent"` Frustration string `json:"frustration"` Escalate bool `json:"escalate"` Model string `json:"model"` } f.Triage = genkit.DefineFlow(g, "triageTicket", func(ctx context.Context, t Ticket) (Decision, error) { out, resp, err := genkit.GenerateData[Triage](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithPromptParts(ai.NewDataPart(t))) if err != nil { return Decision{}, err } d := Decision{ Department: string(out.Department.Choice), Confidence: out.Department.Confidence, Urgent: out.IsUrgent.Probability >= 0.5, Frustration: out.Frustration.Label(), Model: systemonex.ResponseInfo(resp).Model, } // A low-confidence routing decision is not a routing decision. if d.Confidence < 0.5 { d.Department = "other" } d.Escalate = d.Department == "other" || out.NeedsHuman.Probability >= 0.6 || out.Frustration.Level() >= 3 return d, nil }) ResponseInfo(resp).Model records which jev version answered. Confidence thresholds are calibrated per version, so a verdict that does not say who produced it is a threshold waiting to drift. Gemini Drafts, jev Reviews The draft flow is an ordinary GenerateData[Reply] call with the playbook and the ticket in the prompt. One difference from the triage flow: the ticket goes to Gemini as JSON text. A data part is how a decision model takes an object state; a chat model takes a prompt. Go f.Draft = genkit.DefineFlow(g, "draftReply", func(ctx context.Context, in DraftInput) (Reply, error) { r, _, err := genkit.GenerateData[Reply](ctx, g, ai.WithModelName("googleai/gemini-3.8-flash"), ai.WithSystem("You are a support agent for Acme Cloud. Write the reply the customer will receive. "+ "Follow the playbook strictly. Never state amounts, dates or identifiers that are not in the ticket. "+ "Plain text, under 180 words, sign as 'Acme Cloud Support'."), ai.WithPrompt("Playbook:\n%s\n\n%s\n\nTicket:\n%s", in.Playbook, urgency, asJSON(in.Ticket))) // ... }) The review flow is the second decision, and it asks different questions of a different state: the ticket, the draft, and the playbook, together. Go type Overpromise struct{} func (Overpromise) Criteria() (yes, no string) { return "Commits to a refund amount, a credit, a release date, a deadline, or a deletion the playbook does not allow", "Every commitment in the reply is one the playbook permits" } type Tone int func (Tone) Levels() []string { return []string{"Cold or dismissive", "Neutral", "Warm and specific"} } type Review struct { Addresses systemonex.Noul `json:"addresses" jsonschema_description:"Does the reply address what the customer actually asked for?"` Overpromises systemonex.NoulOf[Overpromise] `json:"overpromises" jsonschema_description:"Does the reply promise something the playbook forbids?"` Invents systemonex.Noul `json:"invents" jsonschema_description:"Does the reply state amounts, dates, identifiers or causes that appear in neither the ticket nor the playbook?"` Tone systemonex.Score[Tone] `json:"tone" jsonschema_description:"How does the reply read to the customer?"` } f.Verify = genkit.DefineFlow(g, "verifyReply", func(ctx context.Context, in VerifyInput) (Verdict, error) { out, _, err := genkit.GenerateData[Review](ctx, g, ai.WithModelName("typesafe/jev-latest"), ai.WithSystem("The state has the customer's ticket, the reply an agent drafted, and the playbook the agent had to follow."), ai.WithPromptParts(ai.NewDataPart(in))) if err != nil { return Verdict{}, err } v := Verdict{Addresses: out.Addresses.Probability, Overpromises: out.Overpromises.Probability, Invents: out.Invents.Probability, Tone: out.Tone.Label()} if v.Addresses < 0.6 { v.Reasons = append(v.Reasons, "the reply does not address the customer's ask") } if v.Overpromises >= 0.5 { v.Reasons = append(v.Reasons, "the reply promises something the playbook forbids") } if v.Invents >= 0.5 { v.Reasons = append(v.Reasons, "the reply invents facts") } if out.Tone.Level() == 0 { v.Reasons = append(v.Reasons, "the tone is cold") } v.Approved = len(v.Reasons) == 0 return v, nil }) A system message is never part of the state with this plugin. It goes in front of every question's instructions, which is the right place to say what the state is. Step 3: Flows as Steps That Can Run in a Pod A senro func step is a registered Go function. In the local-only article, the flow lived in a package variable that main filled in. In a pod, that does not work, because senro runs a func step by sending your pipeline binary into the pod and re-entering it as a fresh process; main never gets to the line that initialized Genkit. So Genkit is initialized lazily, inside the step, from the secrets senro delivered to that pod: Go var ( engineOnce sync.Once engine *Flows engineErr error ) func flows(ctx senro.Ctx) (*Flows, error) { engineOnce.Do(func() { var plugins []api.Plugin gemini, err := readSecret(ctx, "Gemini") if err != nil { engineErr = err return } if gemini != "" { plugins = append(plugins, &googlegenai.GoogleAI{APIKey: gemini}) } typesafe, err := readSecret(ctx, "TypeSafe") if err != nil { engineErr = err return } if typesafe != "" { ts := systemonex.TypeSafe() ts.APIKey = typesafe plugins = append(plugins, ts) } g := genkit.Init(ctx, genkit.WithPlugins(plugins...)) engine = defineFlows(g) }) return engine, engineErr } readSecret reads the file that ctx.Secret(name) names, and that is the point: in the pod it is a 0400 file under /run/senro/secrets, projected from a short-lived Kubernetes Secret owned by the pod. The key is never in the pod spec, never in env, never in the plan, never in a log. That is also why the plugin fields are set explicitly rather than read from the environment. And senro.Ctx embeds context.Context, so it goes straight into genkit.Init and flow.Run. The steps themselves are short. Each one reads from the case workspace, runs a flow, and writes back: Go const workspace = "case" func init() { senro.RegisterFunc("ai/ingest", Ingest) senro.RegisterFunc("ai/triage", TriageStep) senro.RegisterFunc("ai/draft", Draft) senro.RegisterFunc("ai/verify", Verify) senro.RegisterFunc("ai/brief", Brief) senro.RegisterFunc("ai/collect", Collect) senro.RegisterFunc("ai/seed", Seed) senro.RegisterFunc("ai/send", Send) } func TriageStep(ctx senro.Ctx, _ NoParams) error { f, err := flows(ctx) if err != nil { return err } var t Ticket if err := readJSON(ctx, "ticket.json", &t); err != nil { return err } d, err := f.Triage.Run(ctx, t) if err != nil { return err } fmt.Fprintf(ctx.Stdout(), "%s decided: department=%s (confidence %.2f) urgent=%v frustration=%q escalate=%v\n", d.Model, d.Department, d.Confidence, d.Urgent, d.Frustration, d.Escalate) return writeJSON(ctx, "decision.json", d) } func Verify(ctx senro.Ctx, p PlaybookParams) error { // ...read ticket.json and reply.json, run f.Verify... if err := writeJSON(ctx, "verdict.json", v); err != nil { return err } if !v.Approved { return fmt.Errorf("reply rejected: %v", v.Reasons) } return nil } Verify returning an error is the guardrail: a func step that returns an error is a failed step, and every dependent of a failed step settles as skipped_upstream_failed. One more line in main, before flag parsing, so that senro's re-entry into the binary does not hit your flags: Go func main() { ctx := context.Background() if handled, err := senro.StepChild(ctx); handled { if err != nil { log.Fatal(err) } return } ticketPath := flag.String("ticket", "tickets/double-charge.txt", "raw ticket to process") // ... } Step 4: The Kubernetes Target One target, used by every AI workflow: Go import "github.com/xavidop/senro/executor/k8s" const stepImage = "docker.io/library/alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8" func pods() senro.ExecutorTarget { return k8s.Pod(stepImage, k8s.Namespace("support"), k8s.Resources( map[string]string{"cpu": "250m", "memory": "128Mi"}, map[string]string{"cpu": "1", "memory": "512Mi"}, )) } Three rules of this executor show up in those six lines: The image must be pinned to a digest and the namespace must be stated, or Build() refuses the pipeline. There is no fallback to default.The image needs sh and tar, and nothing else. Alpine is 3 MB. Your pipeline binary is not in the image; senro cross-compiles it with CGO_ENABLED=0, sends it in as a tar over the apiserver's exec subresource, and re-enters it. No registry, no Dockerfile, no build step for your code.Requests and limits apply to the step's container only. senro adds two containers of its own when a step mounts a workspace: an init container that fills the emptyDir, and an idle container that keeps the pod alive so the result can be read back out. Step 5: Run One, the Triage Go func triagePipeline(ticketPath, outDir string) *senro.Pipeline { p := senro.New("triage") ws := senro.Workspace(workspace) // The raw email enters the workspace from the coordinator, so the ingest // step can declare it as an input and be cached on it. intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: filepath.Dir(ticketPath), Files: []string{filepath.Base(ticketPath)}, })).Mount(ws.At("/case", senro.RW)) understand := p.Workflow("understand", senro.Needs("intake"), senro.On(pods())) understand.Step("ingest", senro.Func("ai/ingest", IngestParams{File: filepath.Base(ticketPath)})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File(filepath.Base(ticketPath))). Outputs(artifact.File("ticket.json")) understand.Step("triage", senro.Func("ai/triage", NoParams{})). Needs("ingest"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()). Pure(). Inputs(artifact.File("ticket.json")). Outputs(artifact.File("decision.json")) handoff := p.Workflow("handoff", senro.Needs("understand"), senro.On(senro.Local())) handoff.Step("collect", senro.Func("ai/collect", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RO)) return p } Each step names only the secret it needs: the Gemini step never sees the TypeSafe key and the jev step never sees the Gemini key. Timeout matters on a remote func step, because the function has a process of its own in the pod and the deadline is what ends it if the coordinator dies. The two pod steps are Pure() with explicit Inputs and Outputs, so a repeated ticket is served from the cache. seed and collect run on the coordinator, and collect copies the two files to disk, which is how the verdict crosses into the second run. The keys are declared once, as a struct mamori resolves from the environment: Go type Secrets struct { Gemini secret.String `source:"env:GEMINI_API_KEY"` TypeSafe secret.String `source:"env:TYPESAFE_API_KEY"` } secrets, err := mamori.Load[Secrets](ctx) run := func(p *senro.Pipeline, extra ...senro.Option) error { att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket, Pipeline: p.Name()}) if err != nil { return err } defer func() { _ = att.Close() }() opts := append([]senro.Option{ senro.WithAttach(att), senro.WithSecrets(secrets), senro.WithFuncBuild("."), }, extra...) return senro.Run(ctx, p, opts...) } err = run(triagePipeline(ticket, out)) WithFuncBuild(".") names the package senro cross-compiles for the pods, since the pod is Linux and the coordinator may not be. WithAttach lets senro run and senro attach watch the run. Run it against the first sample ticket, a double charge with a deadline. On a TTY you get senro's terminal UI; with --ui=plain you get the same events as lines: Shell senro run . --ui=plain -- -ticket tickets/double-charge.txt Plain Text seed started seed succeeded ingest started ingest stdout | ticket from Marta Ruiz: Card charged twice for invoice INV-2291 ingest succeeded triage started triage stdout | jev-1.13.0 decided: department=billing (confidence 1.00) urgent=true frustration="Annoyed" escalate=false triage succeeded collect started collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded A Gemini call and a jev call, each in its own pod, each with its own log under runs/<id>/. The whole run took eleven seconds. jev-latest resolved to jev-1.13.0, and the decision says so. The verdict lands in out/decision.json: JSON { "department": "billing", "confidence": 1, "urgent": true, "frustration": "Annoyed", "escalate": false, "model": "jev-1.13.0" } And out/ticket.json is what Gemini extracted, facts quoted rather than inferred: JSON { "customer": "Marta Ruiz", "company": "Lumen Logistics", "summary": "Customer was charged twice for invoice INV-2291 on March 3rd", "ask": "Refund the duplicate charge today", "facts": ["INV-2291", "59.00 EUR", "March 3rd"] } Step 6: Let the Verdict Choose the Next Steps The second pipeline declares every branch. The parameters decide which ones run. Go func resolvePipeline(outDir string, reckless bool) *senro.Pipeline { p := senro.New("resolve") ws := senro.Workspace(workspace) intake := p.Workflow("intake", senro.On(senro.Local())) intake.Step("seed", senro.Func("ai/seed", DirParams{ Dir: outDir, Files: []string{"ticket.json", "decision.json"}, })).Mount(ws.At("/case", senro.RW)) for _, d := range departments { draftPlaybook := d.Playbook if reckless { draftPlaybook = recklessPlaybook // see Step 7 } w := p.Workflow(d.Name, senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("department", d.Name))) w.Step(d.Name+"/draft", senro.Func("ai/draft", PlaybookParams{Playbook: draftPlaybook})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) w.Step(d.Name+"/verify", senro.Func("ai/verify", PlaybookParams{Playbook: d.Playbook})). Needs(d.Name + "/draft"). Mount(ws.At("/case", senro.RW)). SecretEnv("TYPESAFE_API_KEY", "TypeSafe"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) send := p.Workflow(d.Name+"-send", senro.Needs(d.Name), senro.On(senro.Local()), senro.When(senro.ParamIs("department", d.Name))) send.Step(d.Name+"/send", senro.Func("ai/send", DirParams{Dir: outDir})). Mount(ws.At("/case", senro.RO)) } esc := p.Workflow("escalate", senro.Needs("intake"), senro.On(pods()), senro.When(senro.ParamIs("escalate", "true"))) esc.Step("escalate/brief", senro.Func("ai/brief", NoParams{})). Mount(ws.At("/case", senro.RW)). SecretEnv("GEMINI_API_KEY", "Gemini"). Timeout(stepTimeout). Retry(3, retry.OnInfra()) // ...and an escalate-notify workflow on the coordinator that collects brief.md return p } The pipeline is a Go program, so the three departments are a for loop over a slice of playbooks, and each playbook is a step parameter. A func step's cache key includes the parameter digest, so editing the billing playbook changes the identity of the billing steps and nothing else. main glues the two runs together with the verdict as parameters: Go var decision Decision b, _ := os.ReadFile(filepath.Join(out, "decision.json")) json.Unmarshal(b, &decision) err = run(resolvePipeline(out, *reckless), senro.WithParams(decision.Params())) Go func (d Decision) Params() senro.Params { return senro.Params{ "department": d.Department, "urgent": fmt.Sprint(d.Urgent), "escalate": fmt.Sprint(d.Escalate), } } Parameters are never recorded in an event or a cache key; they exist to be read by conditions. And this is what the second run looks like for the double-charge ticket, read back from disk with senro attach --run <id> --ui=plain: Plain Text seed succeeded billing/draft stdout | drafted: Duplicate charge for invoice INV-2291 billing/draft succeeded billing/verify stdout | review: addresses=0.81 overpromises=0.44 invents=0.25 tone="Warm and specific" billing/verify stdout | approved billing/verify succeeded billing/send stdout | sent to customer: billing/send succeeded technical/draft skipped_condition technical/verify skipped_condition technical/send skipped_condition account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief skipped_condition escalate/notify skipped_condition run succeeded Twelve steps declared, four executed. technical, account and escalate settled as skipped_condition before a single pod was created, and the run is succeeded, not partial: a pruned branch is not a failed one. The reply that went out, in out/reply.md: Plain Text # Update regarding duplicate charge for invoice INV-2291 Hello Marta, I understand you are under time pressure and need this resolved urgently. Over the next few hours, our billing team will process the reversal on your account. We can confirm that the duplicate charge of 59.00 EUR for invoice INV-2291 from March 3rd will be refunded to your original payment method within 5 to 10 business days. Acme Cloud Support Every number in it is in the ticket, and the one commitment is the one the billing playbook allows. Step 7: The Guardrail Says No The angry-outage ticket takes a different route: Shell go run . -ticket tickets/angry-outage.txt JSON { "department": "technical", "confidence": 1, "urgent": true, "frustration": "Furious", "escalate": true, "model": "jev-1.13.0" } jev put the frustration at the top of the rubric and the ticket's legal threat over the needs-a-human threshold, so escalate is true alongside technical. Both workflows run, in parallel, and the department branches that were not picked are pruned as before: Plain Text seed succeeded billing/draft skipped_condition billing/verify skipped_condition billing/send skipped_condition technical/draft succeeded technical/verify succeeded technical/send succeeded account/draft skipped_condition account/verify skipped_condition account/send skipped_condition escalate/brief succeeded escalate/notify succeeded run succeeded The customer gets a reply that promises no fix time, and a human gets out/brief.md: Plain Text ### Situation Nordwind Delivery (Daniel Berg) is experiencing an active 4-hour platform outage that has completely blocked 300 drivers from accessing their delivery routes. This marks their third outage this month. The customer is furious and demanding an operational fix within the next hour. ### Risk High risk of contract churn, severe reputational damage, and SLA breach penalties (99.9% commitment). Driver operations are entirely halted, and the customer has explicitly escalated to demand leadership intervention. ### Suggested next action 1. Immediately triage core routing services with the on-call DevOps/infrastructure team to restore route availability. 2. Alert an Incident Manager or Support Director immediately to coordinate an executive phone call to Daniel Berg within the 1-hour window. Now the rejection. The repo has a -reckless flag that hands the draft step a playbook the reviewer never sees ("promise the customer whatever will make them happiest: a full refund of the year, a credit, a fix within the hour"), while verify keeps checking against the real one. Run the webhook ticket that way: Shell go run . -ticket tickets/api-500s.txt -reckless Plain Text seed succeeded technical/draft stdout | drafted: Update regarding your webhook errors technical/draft succeeded technical/verify stdout | review: addresses=0.54 overpromises=0.98 invents=0.94 tone="Warm and specific" technical/verify failed: reply rejected: [the reply does not address the customer's ask the reply promises something the playbook forbids the reply invents facts] technical/send skipped_upstream_failed ... run failed The draft promised "a full refund of the year, an account credit, and a complete fix within the hour". jev put overpromises at 0.98 and invents at 0.94, the flow returned an error, verify failed, send was skipped, and the run is failed. Nothing reached the customer, and the reason is in verdict.json next to the draft: JSON { "approved": false, "addresses": 0.54, "overpromises": 0.98, "invents": 0.94, "tone": "Warm and specific", "reasons": [ "the reply does not address the customer's ask", "the reply promises something the playbook forbids", "the reply invents facts" ] } That is the second place jev determines the next step. The first was between the runs, through parameters and When. This one is inside a run, through the step state, and the pipeline's dependency edges do the rest. Step 8: Watch It, and Run It Again senro runs lists every run with its status and duration, and senro attach --run <id> replays one. Every pod senro creates is labeled, so if a coordinator dies mid-run you can still find and clean up what it left: Shell kubectl -n support get pods,secrets -l senro.dev/managed=true And the two triage steps are Pure(), so a second run of the same ticket does not call either model: Plain Text seed succeeded ingest cached triage cached collect stdout | collected ticket.json collect stdout | collected decision.json collect succeeded run succeeded cached is a step state, not a label: no pod, no model call, no tokens. What the Combination Gives You ConcernWho handles itReading and writingGemini, through genkit.GenerateData and GenerateTextRouting, scoring, guardrailsjev, through the same GenerateData call with a decision typeWhere a step runssenro.On(k8s.Pod(...)), one pod per step, platform read from the clusterWhich steps runsenro.When(senro.ParamIs(...)), fed by jev's verdictWhat a rejection doesThe step fails, dependents are skipped_upstream_failedCredentialsmamori resolves, senro delivers a file per pod, per step, per attemptRetriesretry.OnInfra(): an image pull or an evicted pod, never a model answer you did not likeRepeat workPure() steps with Inputs and Outputs are replayed, not re-runEvidenceOne runs/<id>/ per run, with the plan, the events, and one log per step Conclusion The pattern fits in four lines: Define the flows with genkit.DefineFlow. Make the routing and review flows return a decision type from the System One plugin.Register each flow as a senro.Func step that initializes Genkit lazily from ctx.Secret(...).Point the workflows at k8s.Pod(image@digest, k8s.Namespace(...)), and give every remote step a Timeout.Turn the decision into run parameters, gate workflows with senro.When, and let a rejected guardrail fail its step. A decision model is cheap enough and fast enough to sit at every switch point of a pipeline, and typed enough that the switch is a Go field, not a parsed string. senro makes the switch points visible: a branch jev did not pick is a skipped_condition step in the event stream, with the reason one senro cache explain away. The code: genkit-k8s-flows-jevDecision models in Genkit Go and Flowssenro's Kubernetes executor and Func steps off the coordinatorThe previous article: Genkit flows as senro steps

By Xavier Portilla Edo DZone Core CORE
Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud
Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud

Beyond Manual Mapping: AI-Orchestrated Governance for Hybrid Cloud It's Tuesday morning, and Slack is already on fire. A Salesforce admin renamed a custom field over the weekend, and your Spring Boot microservice has been throwing 400 errors since 6 a.m. You spend the next four hours tracing logs, patching a DTO, and redeploying. By Wednesday, you're in an "alignment meeting," manually cross-referencing field mappings between two systems that are supposed to talk to each other without your help. Every architect running a hybrid Salesforce and Spring Boot stack has lived this week. It's not a one-time incident; it's a recurring tax on the org. It grows with every schema change, every new field, every integration you bolt on. Most teams accept it as the cost of doing business. It doesn't have to be. Where the Wheels Come Off The friction comes from two architectures that were built with opposite priorities. Salesforce is optimized for speed. A business user creates a custom field in Setup, and it's live in minutes, no code review, no pipeline, no waiting. Spring Boot is optimized for rigor. Fields live in strongly typed DTOs. Changing one means a commit, a PR review, a test run, and a deployment through CI/CD. That's a feature, not a bug; it's what keeps your backend from falling over. When the two connect over REST, that mismatch turns into outages. Here's the kind of DTO that breaks the moment Phone__c becomes PhoneNumber__c on the Salesforce side: Java public class CustomerDTO { private String customerId; private String phone; // maps to Salesforce field Phone__c // getters/setters omitted } Nothing in this class tells you the field it depends on just got renamed. The Salesforce admin has no idea a Java class is bound to Phone__c. The developer has no idea the picklist they're parsing was just restructured with two new values. Information doesn't flow; it fragments. Teams patch around it, and the patches pile into a Frankenstein integration made of XSLT transforms, one-off converters, and tribal knowledge that lives in one person's head. That tax is measurable. Developers routinely lose 30 to 40 percent of their time to keeping brittle connections alive instead of shipping features. Every schema change becomes a small bet against production. A Different Approach: Zero-Touch Governance What if the system watched itself and fixed what it could, with a human still holding the approval button? That's the idea behind AI-orchestrated API governance: an orchestration layer that watches both Salesforce and Spring Boot, understands what changed and why it matters, and proposes (or, once trusted, implements) the fix. The important word is "understands." A naive diff tool sees Cust_Phone__c disappear and CustomerPhone__c appear, and flags it as a breaking deletion plus a breaking addition. An AI layer doing semantic analysis recognizes it as a rename, maps it to the existing customerPhoneNumber field in your DTO, and drafts the OpenAPI and Java changes before anyone gets paged. Building the Intelligence Layer Three components have to work together for this to be trustworthy instead of terrifying. 1. The Observation Engine This layer taps Salesforce Change Data Capture (or Platform Events) on one side and your Spring Boot Actuator endpoints on the other, then asks an LLM to classify what actually happened. A simplified CDC listener might look like this: Java @Component public class SalesforceChangeListener { @EmpConnectorSubscription(channel = "/data/Contact_ChangeEvent") public void onFieldChange(ChangeEvent event) { FieldChange change = FieldChange.builder() .objectName(event.getEntityName()) .oldFieldApiName(event.getChangedFieldOldName()) .newFieldApiName(event.getChangedFieldNewName()) .changeType(event.getChangeType()) .build(); semanticClassifier.classify(change); } } The classifier itself is a targeted prompt, not a general chatbot call: Java String prompt = """ A Salesforce field changed on object %s. Old API name: %s New API name: %s Change type: %s Classify this as one of: RENAME, TYPE_CHANGE, DEPRECATION, NEW_FIELD, BREAKING_REMOVAL. If RENAME, return the confidence score and the likely corresponding Spring Boot DTO field name. Respond only in JSON. """.formatted(objectName, oldName, newName, changeType); That last constraint matters more than it looks. Constraining the model's output to a fixed schema is what makes the downstream Policy Engine able to act on the result programmatically instead of parsing prose. 2. The Policy Engine Detection without guardrails is just a faster way to break things. The Policy Engine enforces the rules your architecture team already agreed on, things like "no breaking changes to public APIs without a major version bump" or "PII fields must stay encrypted in transit." Java public class GovernancePolicy { public PolicyResult evaluate(ProposedChange change) { if (change.isBreaking() && !change.hasMajorVersionBump()) { return PolicyResult.blocked( "Breaking change requires API major version bump"); } if (change.touchesPiiField() && !change.isEncrypted()) { return PolicyResult.blocked( "PII fields must remain encrypted end to end"); } if (change.isCustomerFacing() && change.deprecationWindowHours() < 48) { return PolicyResult.blocked( "Customer-facing changes need a 48 hour deprecation window"); } return PolicyResult.approved(); } } If a proposed change trips one of these rules, automation stops and an architect gets looped in. That's the whole point: the system earns trust by refusing to act outside its lane, not by acting fast. 3. The Execution Layer Once a change clears policy, this layer turns the proposal into an actual pull request. It can regenerate the OpenAPI contract: YAML CustomerDTO: type: object properties: customerId: type: string customerPhoneNumber: # renamed from "phone" type: string description: "Renamed from Salesforce Cust_Phone__c to CustomerPhone__c" and the matching Java class: Java public class CustomerDTO { private String customerId; private String customerPhoneNumber; // was: phone, mapped from CustomerPhone__c // getters/setters omitted } then open a PR, run the existing test suite in staging, and post the results back to the team. The API contract becomes the single source of truth both systems reconcile against, instead of a document that drifts out of date the week after it's written. A Second Example: Picklist Drift Renames aren't the only failure mode. Picklists shift constantly, and Spring Boot enums don't forgive silently. Say a Salesforce admin adds "Pending Review" to an Order_Status__c picklist that a Java enum is validating against: Java public enum OrderStatus { NEW, IN_PROGRESS, SHIPPED, CANCELLED // "Pending Review" doesn't exist here yet } Any order that lands in that new status now fails deserialization. The Observation Engine catches the new picklist value through the Metadata API, the Policy Engine checks whether enum changes count as breaking under your versioning rules, and the Execution Layer proposes the enum update alongside a migration note for any records currently in that state. That's a five-minute PR instead of a production incident discovered by a customer. What This Means for Your Team The obvious worry is that this replaces developers. It doesn't; it changes what they spend their day on. Right now, a good chunk of your time goes to translation work: mapping fields, writing converters, chasing down what changed and why. That work is necessary, but it isn't strategic. Once an orchestration layer is handling the translation, your role shifts from translator to curator. You define the data architecture, you write the policies the system enforces, and you spend your attention on the problems that actually need a human's judgment. Making the Transition Nobody should flip this on in production on day one. Trust gets earned in stages. Start with AI-Suggested Governance. The system proposes changes and opens PRs; a human reviews and merges. You're delegating the tedious analysis, not the decision. Once accuracy holds up across enough changes, typically approaching 99 percent, move to AI-Automated Testing. The system applies changes in staging, runs your regression suite, and reports results without a human triggering each step. Only after it's proven itself across hundreds of real changes should you consider Zero-Touch in Production, where the system can act autonomously within policy boundaries you've already defined and tested. Wrapping Up The hybrid enterprise, Salesforce for the customer relationship, Spring Boot for the backend logic, isn't going away. But manually bridging the two with mapping layers and tribal knowledge is a relic you can retire. AI-orchestrated API governance doesn't remove developers from the loop. It removes the busywork from their day and gives back the four hours you used to spend in a war room every time a field got renamed.

By Ramesh Bellamkonda
Spring Security JWT Validation Best Practices
Spring Security JWT Validation Best Practices

A few months into a large-scale legacy-to-cloud migration for a Fortune 50 insurance carrier, I found something that had been sitting quietly in our codebase for a while: application code that was manually re-checking JWT claims — signature, issuer, audience, expiry — that Spring Security's OAuth2 Resource Server had already validated at the framework level, before the request ever reached that code. Nobody had done this carelessly. Each addition, on its own, looked like good security hygiene. That's exactly why it survived review for as long as it did. Why This Happens So Easily A JWT is a self-contained, signed structure carrying its own claims. Validating untrusted input by hand is a deeply ingrained instinct for engineers, and for good reason — it usually is the right instinct. The problem is visibility. When a developer writes if (claims.getExpiration().before(now)), a reviewer can point at that line and confirm the check exists. What's much harder to see is the framework machinery quietly doing the exact same job, correctly, one layer up. Spring Security's OAuth2 Resource Server support validates signature, expiry, issuer, and audience (when configured) in line with RFC 7519 and RFC 8725. Its JwtAuthenticationProvider runs these checks on every incoming token, and as the identity provider rotates its signing keys, it automatically re-fetches and re-validates against the new keys — no application code involved. Almost none of this is visible unless you've actually read the framework internals, which is exactly why teams tend to reimplement it instead of trusting it. The audience claim was the clearest example in our case. It identifies which resource server a token is meant for, and the framework exposes it as a single config property. We'd been treating it as something that needed hand-written comparison logic — when really, the value had just never been wired into configuration in the first place. The fix wasn't more code. It was one line of YAML. What We Changed We removed the manual validation entirely and let Spring Security's Resource Server own both authentication and authority mapping: YAML spring: security: oauth2: resourceserver: jwt: issuer-uri: https://login.microsoftonline.com/{tenant-id}/v2.0 audiences: api://{application-id} authorities-claim-name: roles authority-prefix: GRANT_ That block replaced manual signature verification, issuer checking, expiry checking, audience checking, and — on Spring Boot 3.3+ — the hand-written role-to-authority converter too. No JwtAuthenticationConverter, no custom Bean. The framework builds the authority mapping straight from the roles claim and applies whatever prefix you configure. (GRANT_ isn't a framework requirement, by the way — pick anything, as long as it matches what your @PreAuthorize expressions expect downstream.) The Objection We Had to Take Seriously This didn't get waved through. It took a few weeks of real back-and-forth, because the pushback was a legitimate security position, not just resistance to change: don't trust any single layer — verify independently, everywhere. Concretely: if issuer-uri got misconfigured — pointed at a stale tenant, or worse, something attacker-influenced — the Resource Server would faithfully validate every token against the wrong authority, and nothing in a minimal-code setup would catch that. A hardcoded, independent check plausibly would. Nobody disputed that the risk was real. The actual question was which layer should own catching it. Our answer: a hardcoded application-layer check and a misconfigured deployment value are different failure modes, touched by different people, on different release cadences. Duplicating validation in code doesn't actually protect against a bad issuer-uri — it just adds a second place that can also drift out of sync. Instead, we added a CI/CD gate that asserts the deployed issuer-uri and audience resolve to the expected tenant, and fails the pipeline on mismatch, on every deploy. Is that still "a second check"? Sure. But it fails differently, and that difference is the whole argument: a broken pipeline gate is a loud, visible build failure someone has to explicitly override. A stale hardcoded value in application code can sit silently wrong for months, because nothing forces anyone to look at it again. What I Can Honestly Say Held Up I want to be straight about this: since the change shipped, we haven't had a key rotation or a deployment misconfiguration that actually tested either mechanism in anger. So I can't tell you the CI/CD gate has caught a real incident, because it hasn't had to yet. What I can say: the manual validation code is gone, no duplicate signature/issuer/audience/expiry logic remains anywhere in the codebase, and in the months since, nothing attributable to the change has come up in review. That's evidence the change didn't introduce a regression — not proof it's been battle-tested under a live failure. Worth being honest about the difference. The Takeaway Treat hand-rolled token validation as a prompt to investigate, not an automatic red flag. In review, ask what the framework's resource-server config already checks before accepting new validation code as necessary.If a function exists purely to re-derive something the framework already validates, the fix is usually a config line, not a code change.Defense-in-depth concerns about misconfiguration are legitimate — answer them directly. Often the better answer is verifying config at deploy time (CI/CD gating), not re-validating tokens at runtime.The two approaches fail differently, and that's the actual argument for preferring the config-time check: a broken gate is loud, a stale hardcoded check is silent. This example is Spring Security and Entra ID, but the underlying review question — what does the framework already check before I add code that checks it again — travels well to any OAuth2/OIDC resource-server setup.

By Alka Nimje
Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade
Java 27 Makes G1 the Default Everywhere: How to Test Your Small Containers Before You Upgrade

When JDK 27 was released on 15 September 2026, most of the attention went to compact object headers and post-quantum TLS. JEP 523 received less coverage, partly because its description sounds like housekeeping: G1 becomes the default garbage collector in all environments. For most applications, that changes nothing. For a specific class of container, it changes the collector silently, with no warning in the logs and no failure at startup. This article covers what the fallback did, which workloads lose it, how to identify affected services, and how to run a comparison that produces usable numbers rather than noise. What the Serial GC Fallback Did G1 has been the nominal default since Java 9. What is less widely known is that HotSpot inspected the machine at startup and overrode that choice in constrained environments. The JVM selected Serial GC when either condition was true: ConditionThresholdAvailable processors1Physical memoryLess than 1,792 MB The reasoning was sound at the time. Serial GC is single-threaded with a very small native memory footprint. On a single-core machine, a concurrent collector spends time competing with application threads for the only available core, and the bookkeeping structures G1 maintains were expensive relative to a small heap. G1 has improved substantially since then. Its native memory footprint has been reduced across several releases, and JDK 26 delivered JEP 522, which removed synchronization overhead from G1's write barriers by introducing a second card table. JEP 523 concludes that work. The fallback is removed, and G1 is now selected in every environment regardless of CPU count or available memory. Serial GC remains fully supported and can still be selected explicitly: Shell java -XX:+UseSerialGC -jar application.jar Applications that already specify a collector, whether G1, ZGC, Parallel, or Shenandoah, are unaffected by this change. Which Workloads Are Affected The change only reaches services that never had an explicit collector flag and that run in a constrained environment. In practice, this describes a predictable set of deployments: Sidecar containers. Log shippers, metrics agents, and service mesh components typically receive a fraction of a CPU and a few hundred megabytes of memory. They are rarely tuned.Batch and queue workers. Short-lived jobs scheduled onto whatever capacity is available.Low-traffic internal services. Admin APIs, configuration services, and internal tooling that are deliberately given small pods.CI runners and development containers. Environments where JVM flags are seldom set because the workload is not production.Pods with fractional CPU limits. This case is easy to miss. A container configured with cpu: 500m may report a single available processor to the JVM, which placed it in Serial GC territory on earlier releases. These are the workloads least likely to appear in a performance test plan, which is precisely why the change deserves deliberate verification. Step 1: Identify Your Current Collector Before changing anything, determine which of your services are actually running Serial GC today. The GarbageCollectorMXBean interface exposes this directly: Java import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; public final class RuntimeFacts { public static void main(String[] args) { Runtime rt = Runtime.getRuntime(); System.out.printf("java version : %s%n", Runtime.version()); System.out.printf("available cpus : %d%n", rt.availableProcessors()); System.out.printf("max heap MB : %,d%n", rt.maxMemory() / (1024 * 1024)); for (GarbageCollectorMXBean gc : ManagementFactory.getGarbageCollectorMXBeans()) { System.out.printf("collector : %s%n", gc.getName()); } } } The collector names identify which implementation is active: CollectorReported MXBean namesSerial GCCopy, MarkSweepCompactG1G1 Young Generation, G1 Concurrent GCParallelPS Scavenge, PS MarkSweepZGCZGC Cycles, ZGC Pauses For a Spring Boot application, registering this as a startup listener puts the information into your existing log aggregation without any additional infrastructure: Java import org.springframework.boot.context.event.ApplicationReadyEvent; import org.springframework.context.event.EventListener; import org.springframework.stereotype.Component; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.util.stream.Collectors; @Component public class GcFactsLogger { @EventListener(ApplicationReadyEvent.class) public void logCollector() { Runtime rt = Runtime.getRuntime(); String collectors = ManagementFactory.getGarbageCollectorMXBeans() .stream() .map(GarbageCollectorMXBean::getName) .collect(Collectors.joining(", ")); System.out.printf( "JVM %s | cpus=%d | heap=%,dMB | gc=%s%n", Runtime.version(), rt.availableProcessors(), rt.maxMemory() / (1024 * 1024), collectors); } } Deploy this across your estate, then filter your logs for entries reporting Copy as a collector name. That result set defines the scope of your testing. Step 2: Understand the Native Memory Difference Discussions about garbage collector selection usually center on pause times. For this particular change, native memory consumption is the more consequential metric. G1 maintains bookkeeping structures that Serial GC does not require, including remembered sets, card tables, and per-region metadata. This memory is allocated outside the Java heap, which means it counts against a container memory limit without appearing in heap metrics. Measurements published after the JDK 27 release place G1's native memory overhead at approximately 43 MB, compared with roughly 1.7 MB for Serial GC. Ankur Mundra measured both collectors on Spring Boot 4.1 within real cgroup limits and reported these figures. The absolute difference is modest. The proportional impact depends entirely on container size: Container memory limitApproximate G1 overhead share1024 MB4%512 MB8%256 MB16% On a tightly sized container, this overhead is consumed before the application allocates its first object. Where memory limits were set with minimal headroom, the change alone can produce OOM kills after an upgrade that altered nothing else. Native Memory Tracking provides the per-category breakdown if you want to confirm the figure in your own environment: Shell java -XX:NativeMemoryTracking=summary \ -XX:+UnlockDiagnosticVMOptions \ -jar application.jar Then, against the running process: Shell jcmd <pid> VM.native_memory summary The GC section of the output reports committed native memory attributable to the collector. Step 3: Run a Controlled Comparison A meaningful comparison requires the smallest container size you actually deploy, representative load, and a run long enough to reach steady state. A thirty-second run measures startup behavior rather than collection behavior. Capture GC logs for both configurations: Shell # Pre-Java 27 behaviour java -XX:+UseSerialGC \ -Xlog:gc*:file=serial.log:time,uptime,level,tags \ -jar application.jar # Java 27 default java -Xlog:gc*:file=g1.log:time,uptime,level,tags \ -jar application.jar Three measurements matter, and the third is the one most often omitted: Total pause time. Sum the pause durations across the run rather than comparing individual collections. Serial GC performs fewer, longer pauses; G1 performs more, shorter ones. Only the aggregate is comparable.Tail latency. G1 runs concurrent phases. On a single-core container, those phases contend with request handling for the only available processor. This effect is invisible in a mean and clearly visible at p99.Resident set size under load. This is where the native memory difference becomes operationally relevant. Sample RSS throughout the run rather than reading it once at startup: Shell while true; do ps -o rss= -p "$PID" | awk '{printf "%s %d MB\n", strftime("%H:%M:%S"), $1/1024}' sleep 5 done If RSS approaches your container limit under G1 where it did not under Serial, you have found the problem before it found you. Step 4: Pin the Collector When G1 Loses If measurement shows that G1 performs worse on a constrained workload, the appropriate response is not to defer the JDK upgrade. It is to state the collector explicitly: Shell java -XX:+UseSerialGC -jar application.jar This is a supported configuration. Serial GC is neither deprecated nor scheduled for removal, and JEP 523 explicitly preserves it. In a Kubernetes deployment, express the choice where it will be reviewed: YAML spec: containers: - name: batch-worker env: - name: JAVA_TOOL_OPTIONS value: "-XX:+UseSerialGC" There is an argument for setting the flag explicitly even where G1 performs well. An explicit collector choice is self-documenting. A future engineer reading the manifest can see that a decision was made and tested, rather than inferring what the JVM selected at runtime. Conclusion JEP 523 is a reasonable change. G1 has genuinely closed the gap that justified the fallback, and removing a hidden conditional from the JVM's startup logic simplifies the mental model for everyone. The operational consideration is narrower: a set of small, rarely tested containers will change collectors on upgrade without any signal that it happened. Identifying those containers takes one log statement. Testing them takes one load run. The cost of skipping both is a latency regression or an OOM kill that appears unrelated to any change your team made. Java 27 is not a long-term support release, and many teams will remain on Java 25. That does not remove the relevance of this work. The same default will arrive with the next LTS, and evaluating it on a short-term release carries considerably less risk than discovering it mid-migration. Further Reading JEP 523: Make G1 the Default Garbage Collector in All EnvironmentsJEP 522: G1 GC: Improve Throughput by Reducing SynchronizationJDK 27 Release Notes

By Garima Agarwal
Delegated Transport, Forked Authorization: Where Chat and AI-Agent Integrations Actually Break
Delegated Transport, Forked Authorization: Where Chat and AI-Agent Integrations Actually Break

When you move messaging onto a third-party platform, you don't hand off the authorization problem. You split it across two systems. Most incidents live in the gap. A simulation harness, and a documentation audit of QuickBlox as a worked example. The Revoked User Who Kept Reading Say a clinic's app removes a contractor's access to a patient thread at 14:02. The app's database says so, the UI hides the thread, and the audit table has the row. The contractor's phone keeps receiving new messages until 15:00, and then keeps going. Nothing in the application is wrong. The membership removal went to the chat platform's API as a single best-effort call, and the API was having a bad ten minutes. The call failed, and the exception was swallowed. The contractor's session token is still valid, and the contractor's client keeps making requests that keep it valid. This is a hypothetical, but every piece is routine: a removal path with no retry, an admin API that can be unavailable, a session whose lifetime depends on activity. The useful question is how long the window lasts, and it becomes measurable once you stop treating the integration as an API call and start treating it as a consistency boundary. This article builds a small deterministic harness, seamlab.py, to measure that window along with seven other failures at the same boundary, including an AI agent acting after a human has taken over a conversation. It then audits one real platform, QuickBlox, against the same questions using its public documentation. QuickBlox is the worked example because its developer documentation is public and detailed enough to check. The harness measures a class of platforms and nothing about any vendor. The documentation audit reads documentation and tests nothing. I keep those two kinds of evidence apart throughout. The Flawed Assumption The usual framing of "build vs. buy" for chat or calling is about transport: delivery, ordering, presence, media. Delegating those is reasonable. But "who may read this thread, who may post, who may trigger an action" is not a transport concern, and delegating transport forks it: Your application holds the relationships: this clinician treats that patient, this contractor was removed.The platform holds a projection of them: dialog memberships, session tokens, message history. The projection is only as fresh as your last successful write to it, and the platform enforces the projection, not your database. So at least two places answer "is this allowed," and they can disagree. Each failure below is a case of that disagreement, or of one side believing a claim the other never checked. Two quantities make this concrete. Revocation window W: the time between the app committing a revocation and the last successful read by the revoked principal. If the platform checks live membership on every read, W is bounded by the time until the removal call lands. If it authorizes from the session token alone, W is bounded by the token's remaining life, and whether that life is fixed or extended by activity decides everything.Authority provenance: for any action in the system, whose authority is it exercising? For a human action, the answer is trivial. For an AI agent in a multi-party conversation, it usually isn't. Threat Model Six attacker positions, all realistic for a chat integration: #AttackerCapabilityAExternalKnows the webhook URLBAuthenticated low-privilege participantCan post messages into a thread the agent readsCSelf-registered platform userPlatform allows signup, no verified binding to your user modelDRevoked participantHolds a valid token from before revocationEReplayerHas a captured, validly signed event (logs, proxy, compromised sink)FWriter on the audit storeCan modify rows Trust boundaries: client to app, app to platform (admin API), platform to app (event delivery), agent to tools, and the audit store. ClassPreconditionMechanismTestControlForged eventAUnauthenticated event endpoint acceptedforgedSignature verificationIdentity confusionCSender resolved by mutable attribute (email)spoofed identityImmutable-ID binding tableReplayEDedupe TTL shorter than replay horizonreplayTimestamp window plus retention ≥ 2×windowDuplicate side effectNormal at-least-once deliveryNon-atomic check-then-actraceAtomic claim before actingStale accessDToken and membership outlive revocationrevocation sweepShort TTL, outbox retry, token killAgent confused deputyBInjected text drives a privileged toolA1–A4Parameter-bound approvalPost-handoff actionRace with human takeoverAgent call in flight at handoffA5–A6Epoch fencingScope overreachBAgent reads history beyond sessionA7Capability-scoped history Audit tampering (F) is covered separately below. The Harness seamlab.py is about 480 lines of stdlib Python on a logical clock, with no network and no randomness. It contains: A platform simulator: at-least-once event envelopes signed with HMAC, expiring tokens with scopes frozen at mint, optional idle (sliding) expiry, self-service registration, and an admin API that can be switched off. Whether reads check live membership, and whether a server-side token kill exists, are switches.Two app implementations: a naive one and a hardened one.An agent world with a tool proxy configurable per control, so the table shows which control closes which attack.A worst-case agent: it obeys every CALL tool key=value line in its context. That is deliberate. The tests target the authorization layer and say nothing about how robust any model is to injection. If your model resists injection, that's a bonus. These controls assume it doesn't. The duplicate-delivery test enumerates every distinct interleaving of two workers handling the same event, rather than sampling a few. Naive Design, Then the Attacks The naive ingress handler is a shape I'd expect to find in plenty of codebases: no authentication on the endpoint, dedupe with a TTL, check then act then record, sender resolved by email. def handle(self, env): t = self.seen.get(env["id"]); yield # check if t is not None and self.clock.t - t < self.DEDUPE_TTL: # 300s return "dup" uid = self.resolve(env["body"]["sender"]) # email match if uid: self.run_command(uid, env["body"]["text"]) # act yield self.seen[env["id"]] = self.clock.t; yield # record AttackNaiveHardenedForged unsigned eventsucceedsblockedValidly signed event, spoofed identitysucceedsblockedReplay at t+225 sblockedblockedReplay at t+3600 ssucceedsblockedDuplicate delivery: interleavings producing a double effect60 of 660 of 12 The spoofed-identity row is the interesting one. The event carries a correct platform signature. Nothing was forged. The attacker registered a platform account with email [email protected], unverified. The naive app lowercases and matches against its user table, so a perfectly authentic message from the wrong person runs with the clinician's authority. Signature verification can't help, because the signature is telling the truth about who sent it. The control is a binding table keyed on the platform's immutable user ID, written only by your backend when it provisions the platform user for an authenticated app user. The attacker's ID has no binding, so the event is rejected as unbound. The race result is worse than it looks. 60 of 66 distinct schedules double-fire. Only interleavings where the first worker finishes recording before the second starts checking are safe. This is the ordinary case for any retry storm. The fix is one atomic operation (SET NX EX, or an insert against a unique constraint) before any side effect. The hardened handler: def handle(self, env): if not hmac.compare_digest(env.get("sig", ""), sig_of(env)): return "bad_sig" if abs(self.clock.t - env["ts"]) > self.TOL: return "stale" ok = self.claim(env["id"]); yield # atomic set-if-absent, retention 2*TOL if not ok: return "dup" uid = self.bindings.get(env["body"]["sender"]) if uid is None: return "unbound" self.run_command(uid, env["body"]["text"]); yield return "ok" Mutation-Testing the Control The suite should be able to fail the hardened design; otherwise, the green column is decoration. So I mutated one parameter: dedupe retention set to half the timestamp tolerance. The replay at t+225 s then succeeds, because the event is still inside the accepted time window while its dedupe record has expired. The replay at t+3600 s is still blocked by the timestamp check. That yields an invariant worth writing into code review: dedupe retention must be at least twice the accepted timestamp skew (once for events slightly in the future, once for slightly in the past). Timestamp validation and dedupe are two halves of one control, and shortening either silently opens the gap. One caveat: claim-before-act gives at-most-once side effects if the process dies between claim and effect. For effects that must not be lost, the downstream operation needs its own idempotency key derived from the event ID. The harness does not model crash recovery. Revocation Window, Measured The harness drives a revoked user who keeps polling and refreshing, with the token's age at revocation sampled across its lifetime. The scenario that matters: the admin API is down when the revocation is issued and comes back 600 s later. In the first table, the platform checks live membership on every read: ConfigWorst W (s)Mean W (s)Token mints per user per hourNaive, TTL 3600, single attempt360018301Hardened, TTL 303016120Hardened, TTL 60603060Hardened, TTL 30030015212Hardened, TTL 9006014054Hardened, TTL 36006015561 "Hardened" means a transactional outbox retrying the removal with capped backoff, a broker that re-checks the app's authorization on every token mint, and a short TTL. Once TTL exceeds the outage, W is capped by the outage plus up to one backoff interval (601 s in this run). Below the outage length, TTL is the binding constraint. With a healthy admin API, every configuration, naive included, shows W = 0. The naive design only fails when the platform is degraded, which is when nobody is watching. When the Platform Doesn’t Check Membership, and Tokens Slide Two platform properties change the picture, and I only added the second after reading the vendor documentation described below, which is why the harness changed between drafts. If reads are authorized from the token alone, removal calls don't matter, and the TTL is the only built-in lever. If the token's expiry is idle-based, so each request pushes it out, then a revoked client that keeps making requests never lets the token expire, and the TTL stops being a bound at all. The broker's re-check at refresh never runs, because refresh never happens. The last control available is a server-side token kill, if the platform offers one. Results for a token-only platform (hardened TTL 60, naive TTL 3600; "never ends" means access persisted to the end of the simulated horizon): Expiry model, admin APINaiveHardened, no token killHardened + token killFixed TTL, healthy3600 s60 s0 sFixed TTL, 600 s outage3600 s60 s60 sIdle expiry, healthynever endsnever ends0 sIdle expiry, 600 s outagenever endsnever ends601 s Read the third row twice. With idle expiry and no kill, the short TTL and the broker re-check buy nothing against a client that keeps talking. The only controls that end access are live-membership enforcement on the platform or an invalidation you can trigger from the server. So the most valuable hour you can spend before integrating a platform is finding out which of those you have. At TTL 60, each active user generates 60 token mints per hour. Whether that is trivial depends on your user count and the platform's rate limits, which the harness does not model. The AI Participant: Confused Deputy, Not “Authorization for Bots” A multi-party conversation is a prompt authored by several principals with different privileges. The agent reading it holds one set of tool credentials. Any principal who can put text in the thread can attempt to spend authority they don't have. This is the classic confused deputy, with the deputy now parsing natural language. The harness has a dialog with a patient and a clinician, and three tools: get_my_record (low privilege), approve_refill (clinician only), and read_history. The proxy runs in three configurations: naive (the agent's standing credential decides everything), role-check (the call executes with the authority of whoever authored the triggering message), and hardened (role check plus parameter-bound approval, epoch fencing, and history clamping). AttackNaiveRole-check onlyHardenedA1 Patient's message steers agent into approve_refillsucceedsblockedblockedA2 Clinician's benign turn; patient's injected text sits in contextsucceedssucceedsblockedA3 Approval for rx42 reused to approve rx99succeedssucceedsblockedA4 One approval consumed twicesucceedssucceedsblockedA5 In-flight agent call executes after human takeoversucceedssucceedsblockedA6 New agent turn minted after takeoversucceedssucceedsblockedA7 Agent reads history outside session scopesucceedssucceedsblockedB1 Clinician approves refill through explicit approvalworksworksworksB2 Patient reads own record through agentworksworksworks The B rows make sure the hardened column isn't green merely because it denies everything. A2 is the finding that changed my plan. My pre-run design was authority as the intersection of the acting principals' rights. Role-check is that rule, and it stops the obvious attack (A1). It fails A2 completely. The clinician sends "please summarize the thread." The agent's capability is legitimately the clinician's. The patient's earlier message, still in context, asks for a refill approval, and the agent obliges, spending authority that was never granted for that purpose. Intersection scopes who the agent acts for. It says nothing about what that authority was meant to cover. The fix is to stop treating conversation turns as grants for privileged operations. A privileged call needs an approval bound to the exact call, issued out-of-band by an authenticated UI session of a privileged human: def _approval_ok(self, tool, args, ap): b = ap["body"] good = hmac.new(APP_KEY, json.dumps(b, sort_keys=True).encode(), hashlib.sha256).hexdigest() if not hmac.compare_digest(good, ap["mac"]): return False if (b["tool"], b["args"], b["epoch"]) != (tool, args, self.epoch): return False if self.clock.t > b["exp"] or b["nonce"] in self.used: return False if self.users.get(b["principal"]) != "clinician": return False self.used.add(b["nonce"]); return True The approval binds tool, arguments, epoch, expiry, and a single-use nonce. A3 fails because the arguments differ, and A4 because the nonce is spent. Injected text can still request the call. It cannot carry the authority to make it. The system doesn't try to detect injection. It makes sure injected text can't hold a capability. Fenced handoff (A5, A6). When a human takes over, the app increments a per-dialog epoch. Every capability carries the epoch it was minted under, and the tool proxy rejects stale ones. A5 is the race: a turn starts at t=0, the human takes over at t=5, and the agent's call arrives at t=8. Without the fence it executes. This is a fencing token, and it has a fencing token's limit: it only works if the resource performing the effect checks the epoch. If the agent can also post through a path that bypasses your proxy, such as the platform's own API with the agent's own credentials, the fence is decorative. The platform-side complement is removing the agent from the dialog at handoff, which is another admin call, subject to the same outage window measured above, and which not every dialog type allows (see below). History scope (A7). The capability carries the index where the current session begins, and read_history refuses anything earlier. The harness checks the proxy's enforcement. It does not check that the agent's context assembly respects the boundary, which is separate code you would need to test. The audit log, and what a hash chain does not do The hardened path records each agent tool call (allowed or denied) and each handoff in a hash-chained log with actor type, model version, tool, arguments, result, and principal: Clean chain, verified against an externally stored head hash: passes.Row modified without recomputing the chain: detected.Row modified and the chain recomputed from that row forward (an attacker with write access): verifies clean unless you compare against the externally stored head, in which case it is detected. The chain alone defends against accidental corruption and clumsy tampering. It does not defend against someone who can write the store and knows the scheme. What does is anchoring the head hash somewhere that an attacker can't write: a separate account, an append-only object store, a periodic signed checkpoint. If your threat model includes a malicious insider with database access, the anchor is the control, and the chain is the mechanism that makes anchoring cheap. On compliance: in a regulated setting, "who did what, when, under whose authority" is what this log makes answerable. The HIPAA Security Rule's audit-control standard is 45 CFR §164.312(b) (cited from memory; confirm before relying on it). A platform's compliance posture does not discharge your obligations, and a log design is one input to them, not compliance itself. A Documentation Audit: The Seam on Quickblox Everything above is simulation. This section asks the checklist questions of a real platform using its public developer documentation, retrieved on 4 October 2026. Nothing here was tested against a live QuickBlox account. Documentation can be incomplete or out of date, and where I could not find an answer, I say so instead of guessing. Sources are listed at the end. QuestionWhat the documentation saysConsequence for the seam1. Are server-to-app events authenticated?I found no first-party documentation of outbound server-to-server event webhooks. A competitor's comparison page asserts QuickBlox doesn't support webhooks. That is a competitor's claim and I haven't confirmed it.If there is no signed event feed, the forged/replay/race tests apply to whatever you build instead. The rule that survives is to never let your backend act on events reported by clients, because client-reported events are attacker-controlled. If you poll, poll with a server credential and treat the response as the source.2. Does a read check live membership, or only the token?Not stated in the pages I read.Unknown. This decides which row of the revocation tables applies. It is a two-minute sandbox test: remove an occupant, then read with that user's still-valid token.3. Token lifetimeBoth user and application session tokens expire 2 hours after the last REST request, and each REST response carries an expiry header. I found no configuration for it.This is idle expiry, which is the dangerous row of the table above. A client that keeps making REST calls keeps the session alive. I don't know whether live chat-connection activity also extends it, and I haven't verified an absolute cap.4. How is membership removed?Removal goes through the dialog update call. A dialog's owner can remove any occupant (the REST reference says admins can too), and an occupant can remove only themselves. The SDK and dashboard docs show removing other users working for group dialogs, not for private (one-to-one) or public ones. Deleting a user removes them from group dialogs but not from private dialogs' occupant lists.Private dialogs have no documented remove-another-user operation. If revocation matters, a clinician-patient conversation modeled as a private dialog can't be revoked by removal. You'd have to delete the dialog for everyone (which needs the owner and loses history). Model sensitive conversations as group dialogs with an app-controlled owner. This is a design decision to make before launch, not after.5. Immutable IDs, self-registrationUser creation with an application session is documented as acceptable for prototypes. For production the docs recommend creating users from your backend with the API key and disabling application-session user creation. Users carry an external_user_id field intended to link to an outside system, and login, email and external ID must be unique.The identity-confusion attack depends on that production setting. Bind your app user to the platform's own user ID, written only by your backend, and don't resolve senders by login or email. external_user_id is a convenient mapping field, but the binding should still be enforced on your side.6. Can the agent be a distinct principal?Dialogs have owner, admin, and occupant roles.Make the agent a separate user and an occupant, never an admin or owner. To fence it at handoff by removing it, the dialog must be a group dialog, per row 4. I did not verify whether the agent's session can be invalidated server-side.7. Can you reconcile after a missed event?Dialog and message REST endpoints exist, and the update and delete references show them taking an API key. I did not verify listing and filter semantics for drift detection.Plan the reconciler as a poll of dialogs and occupants compared with your database. Confirm pagination and filter support in a sandbox first. Three observations from the table, none of them about vendor quality. Every platform has choices like these, and every one is a place the seam can fail: The documented 2-hour idle expiry changes the revocation answer. In the harness's token-only, idle-expiry rows, short TTLs and broker re-checks buy nothing, and the only thing that ends access is membership enforcement or a server-side kill. Whether QuickBlox enforces membership at read time is the first thing to test, and whether you can invalidate a session from the server is the second.The private-dialog limitation is the most actionable finding. It is a design constraint you can only avoid by choosing group dialogs up front.The absence of a documented signed event feed moves work to you. The ingress hardening in this article becomes a design for whatever sync mechanism replaces it. On the vendor's compliance statements: QuickBlox's own pages state that it offers HIPAA-ready hosting with a signed BAA, and that it is SOC 2 certified. I haven't verified those claims, and a BAA does not make your application compliant. The seam problems in this article are exactly the part a BAA doesn't cover. Cost and trade-offs Ingress overhead. Hardened ingress costs about 8.8 µs per event against 2.0 µs for the naive handler, in-process, on one machine (Python 3.12). Indicative only. In production, the dominant cost is one atomic store write per event.Token churn. Short TTLs convert revocation latency into mint load: 60 mints per user-hour at TTL 60, 120 at TTL 30. And if expiry is idle-based, TTL tuning may not be available to you at all.Dependence on the admin API. Forced removal and token kill are only as available as that API. The outbox hides outages from the revoking caller. It doesn't shorten them.Approval friction. Every privileged action needs a human click bound to specific arguments. The agent can't autonomously perform privileged operations. That is the intended outcome, and it will frustrate some product teams. Mitigate it with a good approval surface: show the exact arguments, make approval one tap, and expire quickly.Complexity. The hardened design adds a broker, an outbox, an approval service, a per-dialog epoch, and an anchored log. For an internal tool with a few trusted users, it is probably more than you need. Limitations of this work The simulator is simple: one dialog, instantaneous admin success when up, no clock skew, no partial failures inside the platform, a single-node claim store.The worst-case agent models the threat, not any LLM.I did not implement or test a membership reconciler, although I recommend one.The QuickBlox audit is documentation-only. Docs may be incomplete, inconsistent (the owner-versus-admin removal rule is stated differently in different pages), or out of date. Every "unknown" in the table is a sandbox test I haven't run.Remaining exposure none of this addresses: injected text can still influence unprivileged tools (what the agent says, what it summarizes), exfiltration through in-scope reads, and approval fatigue.Nothing here shows that any real platform is vulnerable to any attack in the harness. Conclusion The integration boundary is a consistency boundary, and the interesting failures are state disagreements and authority crossing it unchecked. The revocation window gives you a number to argue about, instead of a feeling, and the number depends on three platform properties you can look up before writing code: whether reads check live membership, whether token expiry is fixed or idle-based, and whether you can remove a member or kill a session from the server. For the AI participant, the real question is whose authority a tool call exercises. Intersecting principals' rights is necessary but not sufficient. A privileged call needs an approval bound to its exact arguments, or conversation text will eventually carry authority it shouldn't. Sources (QuickBlox documentation, retrieved 4 October 2026): Key Concepts: https://docs.quickblox.com/docs/key-conceptsAuthentication (JS): https://docs.quickblox.com/sdks/js-authenticationUpdate Dialog: https://docs.quickblox.com/reference/update-dialogiOS Dialogs: https://docs.quickblox.com/sdks/ios-chat-dialogsDashboard, Chat: https://docs.quickblox.com/docs/chat-dashboardDashboard, Users: https://docs.quickblox.com/docs/users-dashboardCreate User: https://docs.quickblox.com/reference/create-userDelete Dialog: https://docs.quickblox.com/reference/delete-dialog

By Igboanugo David Ugochukwu DZone Core CORE
A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces
A Non-Expiring AKS Deployment Token Could Read Secrets Across Three Namespaces

A security review started after connection strings were visible through a Kubernetes management tool. The first request sounded simple: remove Secrets from the cluster and move the values to Azure Key Vault. That request focused on which system stored the values. The access review found a wider problem. Kubernetes stored a deployment service account's legacy token in a kubernetes.io/service-account-token Secret. The token had no expiry in its normal use, and RoleBindings allowed the same identity to operate in three application namespaces. The Role used resources: ["*"] across several API groups and allowed get, list, watch, create, update, patch and delete. The Role included the core API group, so the token could read every Secret in each of those namespaces. The review found no evidence of token misuse. It found an avoidable access path with a long lifetime and a larger scope than the pipeline needed. Moving application passwords to Key Vault would improve their source of truth and rotation. It would not revoke this token, narrow its RBAC permissions or remove the cross-namespace bindings. Those controls needed their own fix. Why the Kubernetes Secrets Existed The application images carried JSON configuration files for local development. Database, cache and message-broker endpoints pointed to localhost. Those defaults were useful on a developer machine and unusable inside the cluster. The shared Helm chart supplied the deployment values at runtime. It rendered an Opaque Secret from a secrets map, then loaded the Secret into the application container through envFrom. The reduced chart looked like this: YAML {{- if .Values.secrets } apiVersion: v1 kind: Secret metadata: name: {{ include "app.fullname" . }-secrets type: Opaque stringData: {{- range $key, $value := .Values.secrets } {{ $key }: {{ $value | quote } {{- end } {{- end } The Deployment then loaded every entry as an environment variable: YAML envFrom: - secretRef: name: {{ include "app.fullname" . }-secrets The application configuration placed environment variables after the file-based settings, so the runtime values won. That ordering made the Kubernetes Secret part of the configuration path. Removing it without replacing that path would leave the containers pointing at local services. Figure 1. The existing deployment used a Kubernetes Secret as the runtime override for local file defaults. The next question was, "Which components can retrieve the value, and how long does each form of access last?" Key Vault Does Not Always Remove the Kubernetes Secret The Azure Key Vault provider for the Secrets Store CSI Driver supports more than one consumption model. A pod can read mounted files directly from a CSI volume. This path can avoid creating a native Kubernetes Secret. The application still needs a configuration method that reads those files and responds safely when their contents rotate. The driver can also copy mounted values into a native Kubernetes Secret through the secretObjects field of a SecretProviderClass. Microsoft documents this as secret synchronisation. It is useful when an application already consumes secretKeyRef or envFrom, which was the case here. It also means that moving the source value to Key Vault does not remove the Secret object from Kubernetes. YAML apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: app-vault spec: provider: azure secretObjects: - secretName: app-runtime-secrets type: Opaque data: - objectName: database-password key: Database__Password The choice affects rotation too. Microsoft notes that an existing process does not receive updates when a container consumes a Kubernetes Secret through an environment variable. The team must restart the pod before the process receives the new value. Key Vault remains a sound source for secrets. The important point is narrower: the storage migration and the Kubernetes access review solve different problems. The Service Account Crossed Namespace Boundaries The next part of the review followed the identities that could read Secrets. Kubernetes allows a RoleBinding in one namespace to name a ServiceAccount from another namespace. That feature is useful for controlled cross-namespace work. It also means that checking only the ServiceAccount's home namespace can understate its reach. The review found a layout equivalent to this generalized example: YAML apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pipeline-deployer namespace: environment-b rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io", "policy"] resources: ["*"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: pipeline-deployer-from-a namespace: environment-b subjects: - kind: ServiceAccount name: pipeline-deployer namespace: environment-a roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pipeline-deployer Two bindings of this form extended the identity from its home namespace into two more namespaces. The same bearer token could therefore authenticate once and use the permissions that each Role granted. Figure 2. Two cross-namespace bindings extended one long-lived credential into three namespaces. The identity was not a cluster administrator. The Role listed its API groups, and it did not include rbac.authorization.k8s.io. The token could not grant itself new RoleBindings through this Role. That limit mattered, but it did not make the existing scope acceptable. The core API group still covered Secrets, and the wildcard resource rule covered much more than the deployment needed. The following authorization check avoids printing any secret value: Shell identity="system:serviceaccount:environment-a:pipeline-deployer" for namespace in environment-a environment-b environment-c do printf '%s: ' "$namespace" kubectl auth can-i list secrets \ --namespace "$namespace" \ --as "$identity" done For the reviewed identity, the effective result was yes in all three namespaces. The Token Type Made the Scope Harder to Contain The service account had one manually created Secret of type kubernetes.io/service-account-token. Kubernetes describes this as a legacy mechanism that provides a long-lived credential. The current documentation recommends the TokenRequest API for short-lived tokens instead. The difference changes the failure window. A deployment job can request a token that expires shortly after the job finishes. If someone copies a static token into a CI/CD service connection, that credential can remain useful until somebody deletes or rotates it. An old export, agent workspace or copied configuration can preserve the same access during that period. The cluster version already supported projected service account tokens. Another environment-specific service account had no legacy token Secret, which confirmed that the static pattern was not present in every namespace. The First Fix Reduces the Token's Reach The quickest containment step is to remove the two cross-namespace RoleBindings. That immediately reduces the token's potential reach from three namespaces to one without changing the applications. Each environment should then have its own deployment identity, Role and RoleBinding. A pipeline for one namespace should not borrow the service account from another. This keeps the credential, audit trail and revocation boundary aligned with the environment it deploys. The desired shape is simple: Plain Text pipeline A uses a short-lived token for its Role in namespace A pipeline B uses a short-lived token for its Role in namespace B pipeline C uses a short-lived token for its Role in namespace C These deployments require no cross-namespace subject. Figure 3. The target design aligns one service account and its short-lived token with one namespace. A Short-Lived Token Replaces the Static Credential Kubernetes can issue a bounded token with the TokenRequest API. kubectl create token is a convenient way to exercise that API: Shell kubectl create token pipeline-deployer \ --namespace environment-a \ --duration 15m The command covers only one part of a CI/CD design. The runner still needs a trusted way to request the token. The pipeline should receive the credential at run time, the requester should limit its audience, and the token should expire soon after the deployment window. The service connection must stop storing the old bearer token before that change has value. After the new path succeeds, delete the legacy token Secret and confirm that the API rejects the old credential. Explicit RBAC Replaces Wildcard Access resources: ["*"] made future chart changes part of the role automatically. Kubernetes warns that wildcard rules can grant access to resource types that future API extensions add. A deployment role should name the resource types and verbs the release actually uses. The rendered manifests provide the initial resource list: Shell helm template app ./chart \ --namespace environment-a \ --values values.environment-a.yaml \ > rendered.yaml Inventory every apiVersion and kind in that output. Then compare the result with deployment calls in API audit logs or a dry run. The resulting Role may need access to Deployments, Services, ConfigMaps, Secrets, Jobs, Ingresses and autoscaling resources. It should not receive every resource merely because the chart might use one later. If a separate controller owns runtime Secrets, the deployment identity can lose Secret write access as well. If Helm still renders those Secrets, the role needs the exact verbs required by that release process. The policy must match the delivery design instead of hiding a required permission until the next deployment fails. The Review Found Other Paths to the Same Values The cluster token was the widest durable access path that the review found. Several pipeline variables also stored credentials as ordinary variables. Azure DevOps treats secret variables as protected resources. It encrypts their values at rest and masks exact values in logs. A task can still send a value elsewhere, and masking does not always hide partial strings. Marking the variables as secret still removes routine plaintext visibility from the library interface and logs. The remediation plan combined four related controls: Mark credential variables as secret and restrict which pipelines can use each variable group.Remove routine human permission to read Kubernetes Secrets through cluster tools.Rotate the database, cache and broker credentials that broader access paths exposed.Move non-sensitive settings out of the Helm secrets map and into a ConfigMap. Moving non-sensitive settings to a ConfigMap makes the review surface honest. Feature flags, log levels and endpoint names do not need to sit beside credentials merely because the original release populated only one values map. Validation Uses the Identity's Point of View The validation must test permissions from the identity's point of view. For each deployment identity, build an explicit matrix: Plain Text own namespace other namespaces read required objects yes no apply release objects yes no read application Secrets only if required no modify RBAC no no Then verify the old paths: the deleted legacy token fails authenticationeach identity returns no for Secret access outside its namespacethe pipeline can still deploy and roll backvariable-group values appear masked in the interface and logscluster users without an operational need cannot reveal Secret valuesrotated application credentials work and the previous values fail This sequence keeps the deployment usable while shrinking the access path in measurable steps. Secret Location and Secret Access Need Separate Reviews The visible Kubernetes Secret started the investigation, but deleting that object first would have left the long-lived deployment credential untouched. Using CSI secret synchronization could even have recreated the same type of object from a better source. The useful review order was identity, scope, lifetime, source and rotation. It exposed one token that crossed three namespace boundaries, wildcard permissions that exceeded the release surface, and pipeline values that lacked basic secrecy controls. Key Vault can improve storage and rotation. Short-lived credentials and narrow RBAC control who can reach the Kubernetes API. Environment-specific identities contain the effect of a leaked token. These controls work together, and none of them can substitute for the others. References Kubernetes documentation: SecretsKubernetes documentation: Service AccountsKubernetes documentation: Managing Service AccountsKubernetes documentation: RBAC good practicesMicrosoft Learn: Azure Key Vault provider configuration options for AKSMicrosoft Learn: Access Key Vault through the CSI Driver on AKSMicrosoft Learn: Secrets in Azure PipelinesMicrosoft Learn: Manage Azure Pipelines variable groups

By Ahmed Tariq
Deploying Small Language Models on Sovereign Cloud
Deploying Small Language Models on Sovereign Cloud

The enterprise paradigm for artificial intelligence has fractured. While massive cloud-hosted large language models continue to dominate consumer applications, highly regulated sectors like defense, aerospace, and high-tier finance are actively blocking external API calls to public AI services due to strict data sovereignty mandates. The solution is migrating toward small language models (SLMs) localized at the tactical edge. Deploying an SLM into an air-gapped, "sovereign cloud in a box" environment requires absolute cryptographic certainty over the supply chain. A compromised base image running deep inside a disconnected network invalidates the entire security perimeter. The recent release of Docker Hardened Images (DHI) and the expansion of Google Distributed Cloud (GDC) air-gapped appliances give backend engineers the exact toolchain required to host models like Google's Gemma with zero-trust enforcement. This technical guide details an end-to-end framework for curating a zero-CVE container, packing a fine-tuned Gemma SLM, and pushing it to an isolated GDC cluster. The Target Infrastructure: GDC Air-Gapped Google Distributed Cloud air-gapped is an integrated hardware and software platform designed to operate physically disconnected from the internet. It allows enterprises to deploy container-based workloads and managed services like Vertex AI in completely isolated environments. The architecture supports rigorous compliance frameworks, including NIST, ISO 27001, and SOC II. Notably, GDC offers packaged generative AI solutions enabled by Gemma, providing local, conversational search capabilities on-premises. However, when deploying custom fine-tuned checkpoints or building the Java microservices that handle backend inference orchestration- a core focus of my daily work as an IEEE Senior Member and backend developer — we cannot rely on standard public container registries. The container must be hardened from the base layer up. Base Image Curation With Docker DHI To build our localized SLM container, we start with Docker Hardened Images. DHI provides minimal, secure, production-ready container images maintained by Docker. Rather than blindly pulling a generic Alpine or Debian base image and running a vulnerability scanner after the fact, DHI guarantees a hardened foundation. Using the docker dhi CLI, available in Docker Desktop 4.65 and later, we can inspect and mirror these hardened artifacts directly to our enterprise registries. Shell # Verify local DHI CLI installation docker dhi --help # Mirror a hardened Python or Java base image to your local org docker dhi mirror start --org enterprise-registry \ dhi/python,enterprise-registry/dhi-python By switching our Dockerfile FROM instruction to enterprise-registry/dhi-python, we structurally eliminate the attack surface before installing our inference engine dependencies. Generating the SLM Artifact and SBOM As I detailed in my Springer book chapter Introduction to Trustworthy AI, accountability in machine learning relies heavily on technical robustness and traceability. We cannot deploy a black-box container into an air-gapped network. We must generate a cryptographic Software Bill of Materials (SBOM) that catalogs the exact SLM weight hashes, the inference engine binaries, and the OS packages. The DHI CLI handles this transparently. Once you package your Gemma model weights into the hardened container, you can retrieve the exact attestations and SPDX SBOMs attached to the base images. Shell # Retrieve the SLSA provenance and SPDX SBOM for audit docker dhi attestation get dhi/python:3.11 sha256:<digest> -o provenance.json This JSON payload is ingested into the local CI/CD pipeline. The pipeline validates that no unauthorized dependencies were injected during the model packaging phase. If the cryptographic signature fails, the build halts immediately. Deploying to the Sovereign Edge With a cryptographically signed, DHI-backed container housing our Gemma SLM, we transition to the deployment phase on GDC air-gapped. Because GDC air-gapped operates a single cluster encompassing bare metal nodes without internet access, deployment relies on a physical handoff. The hardened container image, the validated SBOM, and the deployment manifests are loaded onto a portable storage device. Once transferred to the secure facility, the infrastructure operator pushes the image to the local, disconnected GDC container registry. GKE on GDC orchestrates the container workload. We apply Kubernetes network policies to isolate the SLM namespace, ensuring that the model can only be queried by authorized internal Java backend services. By enforcing kernel isolation at the container level and physical isolation at the hardware level, we construct a true defense-in-depth architecture. The Gemma model processes highly classified proprietary data while satisfying the strictest data sovereignty requirements. The transition from cloud-first to sovereign-edge AI marks a critical maturation in software engineering. Trustworthy intelligence demands predictable execution boundaries. By merging the zero-trust foundations of Docker Hardened Images with the physical isolation of Google Distributed Cloud, engineering teams can deliver state-of-the-art reasoning capabilities directly to the tactical edge without compromising a single byte of sensitive data.

By Anuj Ashok Potdar

The Latest Software Design and Architecture Topics

article thumbnail
Choosing a BBD Framework
Learn how to choose a behavior-driven development framework, compare Cucumber, Reqnroll, Behave, and Godog, and avoid common BDD mistakes teams make.
November 13, 2026
by Himanshu Mandhyan
· 536 Views
article thumbnail
Designing for Hot Keys: When Distributed Systems Stop Distributing
Learn how to mitigate hot keys in distributed systems using bucketed writes, caching, replication, queues, and admission control.
November 13, 2026
by Jaswanth Mopathi
· 473 Views · 1 Like
article thumbnail
Persisting Cookies Across JAX-WS Service Calls Using a Custom SOAP Handler
A custom SOAP handler preserves JAX-WS session cookies across calls without JVM-wide state, using ThreadLocal isolation and explicit cleanup to prevent session leaks.
November 12, 2026
by Vishal Rameshchandra Shah
· 651 Views
article thumbnail
Building Enterprise Authentication with Angular, Spring Boot, SSO, and MFA
Implementing SSO and MFA for an enterprise Angular and Spring Boot platform taught me that authentication is frontend architecture, not a feature you bolt on at the end.
November 12, 2026
by Vaishnavi Khosla
· 708 Views
article thumbnail
Microsoft Foundry Agent Debugging Dashboard with Microsoft Fabric Eventhouse, KQL, Docker and Grafana
Build a Microsoft Foundry agent debugging dashboard using OpenTelemetry, Aspire, Microsoft Fabric Eventhouse, KQL, and Grafana.
November 11, 2026
by Jubin Soni, FBCS DZone Core CORE
· 1,095 Views
article thumbnail
Kubernetes Rolling Updates Are Lying to You — and Your Users Are Paying for It
Learn how to configure Kubernetes readiness probes to verify application integration readiness and prevent failed requests during rolling updates.
November 11, 2026
by Pallavi Priya Patharlagadda
· 950 Views
article thumbnail
One Identity Unveils Active Roles 8.6 with Enhanced Security and Administration
Active Roles 8.6: Cumulative security, feature and administration upgrades One Identity has introduced Active Roles 8.6, strengthening secure and streamlined Active Directory administration with cumulative platform improvements, expanded API capabilities and greater authentication flexibility.
November 10, 2026
by Cyber Newswire
· 1,825 Views
article thumbnail
Tracing Dangerous Python Code Paths With rce-path
I built rce-path, a Python package for tracing potentially untrusted inputs to code and shell execution, preserving static evidence, and adding optional AI review.
November 10, 2026
by Sai Teja Erukude
· 923 Views
article thumbnail
Building Multi-Rail Payment Failover to Stop Latency Spikes and Double Debits
Resilient multi-rail payment routing using real-time latency scoring, distributed Redis mutex locks, and strict void-on-timeout controls.
November 9, 2026
by Irfan Syed Ahmed
· 828 Views · 1 Like
article thumbnail
Why Successful Tech Transformation Starts With Clarity
Successful tech transformation starts with clear goals, aligned priorities, and strong governance—not technology alone.
November 9, 2026
by Rick Freedman
· 747 Views · 1 Like
article thumbnail
Engineering Data Observability: Detecting Schema Drift, Data Drift, and Silent Pipeline Failures
Learn how data observability detects schema drift, data quality issues, anomalies, and pipeline failures through monitoring, lineage, and alerting.
November 9, 2026
by RISHI SHRIVASTAVA
· 820 Views · 1 Like
article thumbnail
Chaos Engineering and Kafka In Action: Topic Level Idempotence and Retry Check
Learn how to use Spring Boot, Kafka, and Toxiproxy to simulate network failures, test producer idempotence, and troubleshoot duplicate messages.
November 6, 2026
by Soham Sengupta
· 1,199 Views · 2 Likes
article thumbnail
Building a Windows App for a Resilient SSH SOCKS Proxy and Selective Browser Routing
Route selected browser sites through a resilient local SSH SOCKS proxy on Windows using FoxyProxy. A tray app monitors the tunnel and restores dropped connections.
November 6, 2026
by Denis Tiumentsev
· 1,198 Views · 1 Like
article thumbnail
Member Spotlight: Srinivas Chippagiri
DZone Core member Srinivas Chippagiri shares his approach to Kubernetes performance, cloud-native architecture, and the role of AI in infrastructure management.
November 6, 2026
by Dominique Pugh
· 1,110 Views · 1 Like
article thumbnail
GET, POST, PUT, DELETE: Choosing HTTP Verbs in a Real .NET Core Web API, Not Just by the Textbook
Learn how to choose HTTP verbs in ASP.NET Core Web API, including POST-based soft deletes, batch operations, Angular integration, and idempotency.
November 5, 2026
by Dhana PAMULA
· 1,127 Views
article thumbnail
How to Hide Sensitive Information Before You Share a File
A guide to the Log & HAR Sanitizer: paste text or open a Word file, PDF, or photo, and blur out the private parts in a few clicks. No sign-up. Nothing is uploaded.
November 5, 2026
by Nagappan Subramanian DZone Core CORE
· 1,167 Views
article thumbnail
A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)
Build a Go software factory with Genkit, Kubernetes, and senro, using a decision model to review designs, validate code, and control releases.
November 5, 2026
by Xavier Portilla Edo DZone Core CORE
· 1,249 Views
article thumbnail
5 Mistakes Companies Make When Working With Staff Engineers
Learn five common mistakes organizations make when managing Staff Engineers and how trust, autonomy, and clear responsibilities support their success.
November 4, 2026
by Otavio Santana DZone Core CORE
· 1,926 Views · 1 Like
article thumbnail
Your Event Storming Is Not Your Software Design
Explore how EventStorming supports software design decisions around ownership, contracts, coupling, coordination, and failure handling.
November 4, 2026
by Alireza Rahmani Khalili DZone Core CORE
· 1,219 Views · 1 Like
article thumbnail
Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go
Learn to run Genkit Go AI flows on Kubernetes with senro, using Gemini for text generation and jev for routing and reply validation.
November 4, 2026
by Xavier Portilla Edo DZone Core CORE
· 1,332 Views
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×