DZone Spotlight

Everyone is rushing to build AI agents right now, but GenAI integration really just puts a stronger spotlight on solid system design and backend reliability. An AI agent relies entirely on the APIs and data pipelines underneath it. If your backend drops events or double-processes payments during a network blip, even the best agent will just execute a broken workflow faster. To build resilient systems, you need to see how they break. Reading Kafka documentation on fault tolerance helps, but intentionally introducing network chaos shows how the underlying mechanics actually behave. In this post, we will set up a local testing environment using Spring Boot, Kafka, and Toxiproxy. We'll configure producer properties for topic-level idempotence and walk through common network simulation pitfalls that lead to duplicate data. 1. Setting Up the Chaos Test Architecture To test network issues accurately, stopping the Kafka broker isn't enough. We need to simulate realistic network conditions like latency spikes, packet drops, and bandwidth throttling. Toxiproxy acts as a proxy between our producer and broker to simulate these conditions. The Network Architecture Creating a Local/Dev Kafka Cluster We start by spinning up the Kafka cluster and Kafka UI using Docker. YAML # docker-compose-kafka.yml version: "3.8" #avoid this services: kafka: image: apache/kafka:latest container_name: kafka ports: - "9092:9092" environment: CLUSTER_ID: "4L62xdwCQKWvZs96A39vGA" KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: "broker,controller" KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093" KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL" # Configure distinct internal/external routing ports KAFKA_LISTENERS: "INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093" # We will bookmark this line, dear reader! KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:19092" KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER" KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: "/var/lib/kafka/data" volumes: - kafka_data:/var/lib/kafka/data kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "9080:8080" depends_on: - kafka environment: DYNAMIC_CONFIG_ENABLED: "true" KAFKA_CLUSTERS_0_NAME: "local-cluster" KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:29092" volumes: kafka_data: To spin it up, we use Docker Compose: docker compose -f "docker-compose -kafka.yml" up Both your local Kafka cluster and the kafka-ui should spin up. Note that KAFKA_ADVERTISED_LISTENERS includes the external port localhost:19092; we'll see why. A Local Proxy to Sit Between Kafka Broker and Producer We use Docker for this, too! YAML # docker-compose-toxiproxy.yml version: "3.8" # Don’t use this version - it’s deprecated services: toxiproxy: image: shopify/toxiproxy:latest container_name: toxiproxy ports: - "8474:8474" # Toxiproxy Management API - "8000:8000" # Example port exposed for a proxied application service - "19092:19092" networks: - toxiproxy-net toxiproxy-ui: image: buckle/toxiproxy-frontend:latest container_name: toxiproxy-ui ports: - "8080:8080" # We don’t use UI in this article environment: - TOXIPROXY_URL=http://toxiproxy:8474 depends_on: - toxiproxy networks: - toxiproxy-net networks: toxiproxy-net: driver: bridge docker compose -f docker-compose-toxiproxy.yml up Once running, create a proxy tunnel via the Toxiproxy CLI to listen on port 19092 and forward traffic to Kafka on port 9092: ./toxiproxy-cli create -l 0.0.0.0:19092 -u host.docker.internal:9092 kafka-proxy Note: We use the above CLI inside the toxiproxy container. So, you must use docker exec. In some versions, the CLI tool might not be included in the $PATH, and you might need to switch the current directory: Shell /go/bin # ls toxiproxy toxiproxy-cli If you're using this toxiproxy tool for the first time, you can refer to the official Toxiproxy documentation or this starter guide we created. 2. The Kafka Client Producer Code We create a basic application using Spring Boot 4.1 (though we do not have any version restriction until now) and use the following dependency: XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-kafka</artifactId> </dependency> We'll keep the Spring Boot service simple. We define a basic payload configuration and direct messages to topic T3 via KafkaConfig.java. Config: Java // KafkaConfig.java package com.soham.selfteaching.kafka.kafkadeepdive.config; import lombok.Data; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @Configuration @Data public class KafkaConfig { @Value("${app.topic.name:T3}") private String topic; } Our application.yml is minimal, accepting defaults provided by the Kafka library. Properties files # application.yml server: port: 8082 spring: application: name: kafka-deep-dive kafka: bootstrap-servers: "localhost:19092" producer: acks: "1" # Required to bypass idempotence shield key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: request.timeout.ms: 3000 delivery.timeout.ms: 120000 enable.idempotence: false # We disable it first before we explore producer level idempotence The domain event is defined as a Java record. Java // EventDto.java package com.soham.selfteaching.kafka.kafkadeepdive.dto; public record EventDto(String id,String payload) { } Producer Code The core logic sits in ProducerService.java. We use CompletableFuture to handle delivery timeouts asynchronously without blocking thread execution. If the broker is unreachable initially, max.block.ms handles the immediate block on the main thread while background retries handle transient issues. Java // ProducerService.java package com.soham.selfteaching.kafka.kafkadeepdive.service; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.springframework.kafka.core.KafkaTemplate; import org.springframework.kafka.support.SendResult; import org.springframework.stereotype.Service; import java.util.concurrent.CompletableFuture; @Service @Slf4j @RequiredArgsConstructor public class ProducerService { private final KafkaTemplate<String, Object> kafkaTemplate; public void sendMessage(String topic, String key, Object payload) { log.info("Attempting to send message to topic: {} with key: {}", topic, key); CompletableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, key, payload); future.whenComplete((result, exception) -> { if (exception == null) { log.info("Message successfully delivered. Topic: {}, Partition: {}, Offset: {}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } else { log.error("Fatal delivery failure for key {}. Exhausted delivery.timeout.ms.", key, exception); } }); } } 3. Configuring Topic-Level Idempotence During a network timeout, the producer cannot verify whether the message failed to reach the broker or if the acknowledgment (ACK) was lost on the return path. By default, the producer retries the request, which can produce duplicate records without proper safeguards. Exactly-Once Semantics (EOS) and topic-level idempotence address this by assigning a producer ID (PID) and sequence number to each message batch. If the broker receives a retried batch with a sequence number it has already processed, it ignores the duplicate write. Disabling Idempotence to Force Duplicates To demonstrate retry behavior, we can explicitly disable these safety settings in application.yml. Since Kafka 3.0, enable.idempotence defaults to true. To allow duplicates during testing, set acks: "1" and explicitly set enable.idempotence: false. Setting request.timeout.ms to 3000 (3 seconds) creates a tight window to trigger network retries during latency testing. 4. Running the Kafka Producer and Verify We run the Spring Boot application: Java @SpringBootApplication public class KafkaDeepDiveApplication { public static void main(String[] args) { SpringApplication.run(KafkaDeepDiveApplication.class, args); } } And hit a REST API (Hold on! It's a simple HTTP API which may NOT quite follow REST paradigms). curl -X GET http://localhost:8082/api/send Java @RestController @RequestMapping("/api") @Slf4j @RequiredArgsConstructor public class ProducerController { private final ProducerService service; private final KafkaConfig kafkaConfig; @GetMapping("/send") public String send() { long ts = System.currentTimeMillis(); Object payload = new EventDto("p1.t1." + UUID.randomUUID().toString(), ("Hello message " + i)); service.sendMessage(kafkaConfig.getTopic(), "k1", payload); return "Process took :%s ms".formatted(System.currentTimeMillis() - ts); } } In the case when idempotence is disabled, it's fair to expect that duplicate messages will arrive on a topic if a producer does not receive timely ACK from the broker leaders in alignment with the ack flag. But we do not yet see any duplicate messages at all. This is fairly explainable, since you're all set up locally, and there's usually no lost acknowledgment. Introduce Chaos to the Local Network Add 5000ms latency to the proxy. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy Send a message to the endpoint. After an API hit, wait for 2–3 seconds and remove the toxic Remove the Toxic While the Retry Is in Flight Instantly run this command to remove the toxic while the retry is in flight: ./toxiproxy-cli toxic remove -n false_positive kafka-proxy While this should have produced duplicates due to retries by the producer following a timed-out broker ACK, we find NO DUPLICATE MESSAGES even after repeating the process above many times. 5. Let Us Debug Our Disconnect Common Pitfalls During Network Injection Simply starting the proxy and adding latency doesn't always trigger duplicates as expected. Below are two common issues related to how Kafka clients handle metadata and connections. Issue 1: Advertised Listeners Bypass If Toxiproxy is running on port 19092 and a 5000ms latency is emulated, messages sent through the proxy might still complete in milliseconds. ./toxiproxy-cli toxic add -n false_positive -t latency -a latency=5000 -d kafka-proxy This occurs because of how Kafka advertises broker addresses during metadata discovery. If you look closely at the internal Kafka NetworkClient logs during startup, you will find out why: Plain Text brokers=[MetadataResponseBroker(nodeId=1, host='localhost', port=9092)] When the producer connects to port 19092 for initial cluster metadata, the broker returns its configured advertised listener (e.g., localhost:9092). The producer client caches this response and routes subsequent produce requests directly to port 9092, bypassing the proxy entirely. We must ensure that our KAFKA_ADVERTISED_LISTENERS explicitly broadcasts EXTERNAL://localhost:19092 so the producer continues routing traffic through Toxiproxy. If updated, clear existing Docker volumes (docker-compose down -v) to avoid using cached listener configurations. Issue 2: Initial Metadata Fetch Blocking If high latency (e.g., 15,000ms) is applied before sending any messages, the application may time out after 60 seconds with a TimeoutException: Topic T3 not present in metadata, rather than failing on delivery.timeout.ms. The delivery.timeout.ms timer only applies once a record enters the producer's accumulator buffer. If the client hasn't fetched topic metadata yet, the call blocks on max.block.ms (defaulting to 60 seconds) while attempting to retrieve metadata through the degraded network path. We apply the fix: Send an initial message without active toxics to populate the local metadata cache before injecting network latency. Finally With latency removed, the retried request reaches the broker successfully. Because acks=1 and enable.idempotence=false, the broker writes the payload again without verifying duplicate sequence numbers. Checking Kafka UI confirms two identical records are present in the topic partition. Simulating network edge cases directly demonstrates why configurations like advertised listeners, metadata timeouts, and idempotence defaults exist in modern Kafka clients. Now we get duplicate messages across a topic partition. Duplicate Messages We can now enable back idempotence flag to see that no duplicate messages are created repeating the above steps with enable.idempotence:true, ensuring that idempotence is preserved enable.idempotence= true only ensures idempotence across a partition only and not the entire topic. We shall explore it in a next article.

A full VPN is not always necessary. In addition, VPN connections may be unavailable because of restrictions imposed by a local or national internet provider. For the same reasons, a direct connection to an external SOCKS proxy may not work. Sometimes an alternative network route is needed not for the entire system, or even the entire browser, but only for specific addresses: an external API, an admin panel, documentation, a registry, a test environment, or another resource that is unavailable through the usual route. In this article, we’ll look at a narrower approach to solving this problem: To route traffic at the Chrome level, we use an extension suited to the task. In this case, we use FoxyProxy;Chrome sends only traffic for domains specified in FoxyProxy rules through the proxy;A local SOCKS5 endpoint runs on 127.0.0.1:1080;The endpoint is created using SSH dynamic forwarding;A PowerShell controller starts ssh.exe, monitors its state, and reconnects after a disconnection;An icon in the Windows notification area shows whether a local listener exists at the configured address and port;Launching the launcher again does not create a second controller or a second tray icon. I could not find an available tool that met my needs: one that provided this functionality, started quickly, was simple to use, and made its status easy to see. Existing solutions either have unnecessary settings or have complicated, unfriendly interfaces that require lengthy configuration before they can be used for this task. So, to create a local SSH tunnel on Windows, I developed my own PowerShell controller, launched through a VBS script. It displays a system tray icon, lets the user restart the tunnel, and automatically restores the connection when it drops. The complete source code for the PowerShell controller and VBS launcher, along with a sample configuration and installation instructions, is available in the repository: https://github.com/dtiumentsev/socks-via-ssh-tunnel-tray The controller is free to use under the MIT license. When Selective Routing Beats a Full VPN Suppose a developer needs access to a single API: api.example.dev Other websites, corporate services, local resources, and applications should continue to use the usual route. We also assume that outbound SOCKS traffic or VPN protocols may be unavailable. In this scenario, FoxyProxy matches a URL or domain to a proxy profile defined in its settings, while the proxy itself runs only locally: 127.0.0.1:1080 The goal is to manage the local tunnel as a tray application and be able to check its status at any time without extra steps. A local SOCKS5 proxy created through SSH dynamic forwarding is available at this address. Chrome connects to the local SOCKS5 endpoint, SSH encrypts the connection and carries it to the SSH host, and the remote side establishes a connection to the requested web resource. The setup looks like this: Chrome │ ├── Regular domains ──────────────────────────────► Direct connection │ └── api.example.dev │ ▼ FoxyProxy rule │ ▼ SOCKS5: 127.0.0.1:1080 │ ▼ ssh.exe with dynamic forwarding (-D) │ ▼ VPS / SSH host │ ▼ api.example.dev The advantage of this approach is its limited scope. The tunnel does not change Windows’ system-wide routing and does not affect applications that are not configured to use this SOCKS5 proxy. Prerequisites This article assumes that a VPS or other SSH host already exists and that the reader has legitimate key-based access to it. Preparing a VPS, installing sshd, and configuring server-side authentication are outside the scope of this article. The minimum requirements are: Windows 10 or Windows 11;The OpenSSH Client component installed;PowerShell 5.1 or later;A VPS or another SSH host accessible with a key;A private key corresponding to a public key authorized on the server;Google Chrome, Chromium, or another compatible browser;The FoxyProxy extension;Permission to use the SSH host and access the target resources. To check whether the built-in OpenSSH Client is present, run: Test-Path "$env:WINDIR\System32\OpenSSH\ssh.exe" A result of True means the path to ssh.exe exists. To check whether the private key is available, run: Test-Path "$env:USERPROFILE\.ssh\id_rsa" All environment-specific parameters are kept in a separate config.ps1 file. Here is an example configuration: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The main script loads config.ps1 from its own directory using $PSScriptRoot. The repository contains config.example.ps1; the working config.ps1 contains the actual local settings. SSH Dynamic Forwarding as a Local SOCKS5 Proxy The solution is based on SSH dynamic forwarding, an OpenSSH client mode enabled with the -D option. A simplified command looks like this: ssh.exe ` -i "$env:USERPROFILE\.ssh\id_rsa" ` -p 22 ` -D 127.0.0.1:1080 ` -N ` [email protected] After a successful connection, the SSH client begins listening on a local TCP port 1080 at 127.0.0.1. Any application that supports SOCKS5 can use this endpoint. Importantly, the listener is bound to the loopback address: 127.0.0.1:1080 rather than: 0.0.0.0:1080 The first option is accessible only from the local machine. The second would make the SOCKS proxy accessible to other devices on the network, which is unnecessary for this scenario and increases the risk of unauthorized use. SSH Options The controller uses the following arguments: $arguments = "-4 -i `"$keyPath`" -p $remotePort -D ${bindAddress}:$port -N " + "-o ExitOnForwardFailure=yes " + "-o ServerAliveInterval=15 " + "-o ServerAliveCountMax=3 " + "-o ConnectTimeout=10 " + "-o StrictHostKeyChecking=accept-new " + "-o IdentitiesOnly=yes " + $sshHost Here is what the main options do: OptionPurpose-4Forces IPv4 for the connection to the SSH host-i <key>Specifies a particular private key-p <port>Sets the server’s SSH port-D 127.0.0.1:1080Creates a local SOCKS5 listener on the loopback interface-NDoes not open an interactive shell session or execute a remote commandExitOnForwardFailure=yesExits SSH if dynamic forwarding cannot be establishedServerAliveInterval=15Sends SSH-level keepalive packets every 15 seconds of inactivityServerAliveCountMax=3Closes SSH after three unanswered keepalive messagesConnectTimeout=10Limits the time allowed for a TCP connection attempt to the SSH hostStrictHostKeyChecking=accept-newAdds a new server’s key to known_hosts but does not silently accept a change to a previously known keyIdentitiesOnly=yesUses the specified key rather than trying every identity available through ssh-agent ExitOnForwardFailure=yes is especially important for automation. If the local listener cannot be created, SSH should not keep running as though the tunnel were available. This option makes the client exit when it cannot establish dynamic, local, remote, or tunnel forwarding. StrictHostKeyChecking=accept-new is a practical compromise for the first connection to a new SSH host. It does not disable checking a previously known host key: if the server’s key changes unexpectedly, the client should terminate the connection rather than silently trust the new key. Host key verification remains part of SSH’s security model. Why Starting SSH Once Is Not Enough Running ssh.exe -D solves only the initial problem: it creates a SOCKS listener. In practice, long-lived connections regularly encounter unstable conditions: A laptop goes to sleep and wakes up;The device switches between Wi-Fi, Ethernet, and mobile internet;The external IP address changes;A home router restarts;The SSH host becomes temporarily unavailable;The TCP session’s network state becomes stale;The ssh.exe process exits because of an authentication error, timeout, or network interruption. If SSH was started manually in a separate window, the user has to notice the interruption, determine that the listener is gone, and run the command again. That is inconvenient for a background tool. A small controller can handle four tasks: Start ssh.exe in the background.Check whether the local SOCKS port is listening.Wait for a specified interval after SSH exits, then try to reconnect.Show the current state through a system tray icon. The PowerShell Controller The complete script is in the repository, but its logic is fairly straightforward. The controller keeps a reference to the SSH process: $script:sshProcess = $null It also sets a reconnect delay: $reconnectDelaySeconds = 5 After starting ssh.exe, the controller does not immediately declare the tunnel operational. It checks for a local listener using Get-NetTCPConnection: Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue This is better than relying only on the PID returned by Start-Process. A process starting successfully does not mean SSH has authenticated, established a session, and opened the local SOCKS listener. A simplified version of the check looks like this: function Is-Port-Listening { $listener = Get-NetTCPConnection ` -LocalAddress $bindAddress ` -LocalPort $port ` -State Listen ` -ErrorAction SilentlyContinue return ($null -ne $listener) } When the controller detects a TCP listener at the configured address and port, it shows a green icon. If the listener disappears, the icon turns red. Why Use a WinForms Timer? To keep the tray icon working, the PowerShell script runs a WinForms message loop: [System.Windows.Forms.Application]::Run() If the script exits, the icon disappears. The tray interface also stops responding to clicks and context-menu actions. The tunnel is checked periodically using System.Windows.Forms.Timer: $timer = New-Object System.Windows.Forms.Timer $timer.Interval = $checkIntervalMs In the current version, the check runs every five seconds. This reduces how often the port is polled; the exact time of the next connection attempt depends on when the timer fires after the reconnect delay expires. NotifyIcon is the standard Windows Forms component for displaying an icon in the Windows notification area. It works well for a small background controller with a minimal interface. How Reconnection Works The recovery logic is intentionally simple: ssh.exe exits ↓ The controller clears the saved PID ↓ It waits five seconds ↓ It starts another attempt if the port is free Here is the part that detects when SSH has exited: if ($script:sshProcess -and $script:sshProcess.HasExited) { $script:sshProcess = $null $script:nextStartTime = (Get-Date).AddSeconds($reconnectDelaySeconds) } Then, once the wait is over: if ( -not $script:sshProcess -and -not (Is-Port-Occupied) -and (Get-Date) -ge $script:nextStartTime ) { Start-Ssh } Before starting a new process, the controller checks whether the configured local port is free. It does not terminate a process that may have occupied the port: in the event of a conflict, the user must free the port manually or choose another LocalPort in config.ps1. Reconnection cannot begin until the SSH client realizes that the connection is no longer working. The following SSH keepalive settings help with that: ServerAliveInterval=15 ServerAliveCountMax=3 With ServerAliveInterval=15 and ServerAliveCountMax=3, SSH may detect an unresponsive server after approximately 45 seconds (15 × 3). The controller then waits another five seconds before attempting to reconnect. This is not a strict recovery-time guarantee. The actual result depends on the TCP connection’s state, the type of interruption, network latency, and the behavior of the SSH host. Still, this mechanism is much more reliable than passively waiting for the system’s TCP timeout. The System Tray Icon Keeping a PowerShell window open for a background tunnel is inconvenient. An icon in the notification area provides a simple interface: A green icon means a local TCP listener was detected at the configured ${BindAddress}:${LocalPort}; in the example configuration, that is 127.0.0.1:1080; A red icon means the listener is absent or the tunnel has been stopped; A context menu provides manual control over the connection lifecycle. Version 1 provides the following commands: Menu itemActionReconnect nowTerminates the current managed SSH process and initiates another connection attemptStop tunnelStops SSH and disables automatic reconnectionStart tunnelResumes the controller and starts another connection attemptExitStops SSH, removes the tray icon, and exits the PowerShell controller A manual stop is fundamentally different from a network failure. After Stop tunnel, the controller sets an internal stopped state and does not start SSH again. Otherwise, the script would restore the tunnel five seconds after the user stopped it. Why the Icon Is Not Updated on Every Timer Tick I do not consider reassigning NotifyIcon.Icon on every timer tick a good approach. Even when the state has not changed, Windows has to redraw the icon, which may look like flickering. In version 1, the port state is stored: $script:lastPortState = $null The icon changes only when the state changes: if ($alive -eq $script:lastPortState) { return } This is a small detail, but it matters for the normal behavior of a background tool. Preventing Duplicate Launches It is easy to launch a tray application again—for example, by double-clicking the launcher a second time. Without protection, this could produce two tray icons and multiple PowerShell controllers trying to manage the same local port. A named mutex prevents that: $mutexName = "Local\Denis_SocksTunnelTray_v1" $createdNew = $false $mutex = New-Object System.Threading.Mutex( $true, $mutexName, [ref]$createdNew ) if (-not $createdNew) { $mutex.Dispose() exit 0 } The first instance creates the mutex and owns it for as long as it runs. When launched again, the second instance detects the existing mutex and exits immediately. It does not create a second SSH process or add another tray icon. The Local\ prefix limits the mutex’s scope to the current Windows user session. That is sufficient for a desktop tool running in a normal interactive session. Launching Without a Black Console Window Launching a .ps1 file through a .cmd file opens a cmd.exe window. Even if the PowerShell process is minimized or hidden, the console launcher can still be visible briefly. A minimal VBS launcher is more convenient for users. It starts PowerShell in hidden mode and then exits. The Start-SocksTunnelTray.vbs file sits next to the PowerShell script: Option Explicit Dim fso, shell, folder, ps1, command Set fso = CreateObject("Scripting.FileSystemObject") Set shell = CreateObject("WScript.Shell") folder = fso.GetParentFolderName(WScript.ScriptFullName) ps1 = folder & "\Start-SocksTunnelTray.ps1" command = "powershell.exe -NoLogo -NoProfile -STA -WindowStyle Hidden -ExecutionPolicy Bypass -File """ & ps1 & """" shell.Run command, 0, False Here: -NoLogo removes the PowerShell startup banner;-NoProfile prevents the user’s PowerShell profile from affecting the process;-STA starts PowerShell in a single-threaded apartment model suitable for Windows Forms;-WindowStyle Hidden hides the PowerShell window;-ExecutionPolicy Bypass applies only to this process and does not change the system’s execution policy;shell.Run command, 0, False starts the process without displaying a window and does not wait for it to exit. The user double-clicks the VBS file and sees only the tray icon. Configuring FoxyProxy Once the tunnel is running, the browser needs to know which requests to send through the local SOCKS5 proxy. In FoxyProxy, create a proxy profile with these settings: Proxy type: SOCKS5 Host/IP address: 127.0.0.1 Port: 1080 Next, add a rule that applies the profile only to the required domain. For example: https://api.example.dev/* Or, if subdomains are needed: https://*.example.dev/* The key is not to enable proxying for all URLs when the goal is selective routing. The narrower the rule, the easier it is to control its effects and troubleshoot problems. DNS Through SOCKS If the extension offers a “DNS resolution through SOCKS5” setting, it makes sense to enable it for domains that should use the proxy route. The practical reason is simple: a standard SOCKS5 profile may forward the TCP connection, but the browser or operating system may resolve the hostname locally before opening that connection. Depending on the configuration, this means some network information does not follow the selected route. To test from the command line, you can use: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.example.dev/ The h in socks5h indicates that the proxy, rather than the local system, should resolve the hostname. Verifying the Tunnel First, make sure Windows sees the listener: Get-NetTCPConnection -State Listen -LocalPort 1080 | Select-Object LocalAddress, LocalPort, OwningProcess Expected output looks like this: LocalAddress LocalPort OwningProcess ------------ --------- ------------- 127.0.0.1 1080 12345 Next, you can test proxying through an external service that reports the IP address of the outgoing connection: curl.exe --proxy socks5h://127.0.0.1:1080 https://api.ipify.org If the command returns the address of your external proxy host, the traffic is passing through the local SOCKS5 endpoint and the remote route. Diagnosing an Occupied Local Port If the tray icon remains in the reconnecting state, check which process is listening on the local port. The repository includes a separate diagnostic script for this: .\scripts\Check-SocksPort.ps1 The script checks the fixed TCP port 1080 and displays the LocalAddress, LocalPort, state, owning PID, and process name. If config.ps1 specifies a different LocalPort, change $port = 1080 in scripts/Check-SocksPort.ps1 to the required port. The script does not terminate processes or change the configuration. If an old ssh.exe process occupies the port, exit the old controller using Exit, or stop the process in a Windows session with the same privileges under which it was started. If another program occupies the port, do not terminate it automatically: change LocalPort in config.ps1 and in the FoxyProxy profile, for example, from 1080 to 1081. Testing Recovery After a Failure A minimal test: Start the controller using Start-SocksTunnelTray.vbs.Wait for the green icon.Open Task Manager.Find the ssh.exe process started by the controller.End the process.Confirm that the tray icon changes to the unavailable state.Wait for several check cycles.Check that a new ssh.exe process appears.Confirm that the listener at 127.0.0.1:1080 is available again. You can also test the following scenarios: ScenarioExpected behaviorTemporarily disconnect from the networkSSH detects the lost connection through keepalives, exits, and the controller begins reconnect attemptsClick Stop tunnelSSH exits, the listener disappears, and no reconnection occursClick Start tunnelThe controller starts SSH againClick Reconnect nowThe current SSH process exits, followed by another connection attemptLaunch the VBS launcher a second timeThe second instance exits silently because of the mutexHave another program occupy port 1080SSH cannot establish forwarding; the conflict must be resolved or another port selectedCheck a regular site without a FoxyProxy ruleTraffic should go directly rather than through the SOCKS profileCheck api.example.dev with a ruleThe request should use the local SOCKS5 profile The Indicator Checks the Local Port A green icon means there is a TCP listener at the configured address and port. The controller does not check whether the listener belongs to the ssh.exe process it started, nor does it confirm that the target site is reachable. If an old SSH process or another program occupies the port, check its owner using scripts/Check-SocksPort.ps1. The controller does not automatically terminate that process. SOCKS Dynamic Forwarding Is Primarily for TCP SSH dynamic forwarding works as a SOCKS proxy and is primarily intended for TCP connections. It is not a universal way to route every protocol on the system. If you need to proxy UDP, the entire network stack, or multiple applications without built-in proxy support, you will need a different type of solution. Security Recommendations A few useful practices: Bind the SOCKS listener to 127.0.0.1, not 0.0.0.0.Use a passphrase-protected SSH key if your workflow supports ssh-agent.Verify the SSH host’s fingerprint before connecting for the first time.Do not disable host key checking globally with StrictHostKeyChecking=no.Keep the list of FoxyProxy domains short and explicit. Repository Structure The socks-via-ssh-tunnel-tray repository is organized as follows: socks-via-ssh-tunnel-tray/ ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── SECURITY.md ├── Start-SocksTunnelTray.ps1 ├── Start-SocksTunnelTray.vbs ├── config.example.ps1 ├── icons/ │ ├── down.ico │ └── up.ico └── scripts/ └── Check-SocksPort.ps1 config.example.ps1 contains only a template: $SshTunnelConfig = @{ SshPath = "$env:WINDIR\System32\OpenSSH\ssh.exe" KeyPath = "$env:USERPROFILE\.ssh\id_rsa" SshHost = "[email protected]" RemotePort = 22 BindAddress = "127.0.0.1" LocalPort = 1080 } The user copies it to config.ps1 and fills in their own settings. The repository’s code is distributed under the MIT license. Conclusion An application using SSH dynamic forwarding provides a local SOCKS5 endpoint without requiring a separate proxy service on the client machine. Combined with FoxyProxy, it becomes a selective routing tool: the browser uses an alternative route only for explicitly selected domains, while the rest of the system continues to work as usual. This PowerShell controller handles the operational side of the task: It starts ssh.exe in the background;It shows the listener’s status through a tray icon;It restores the connection after the SSH tunnel drops;It distinguishes a manual stop from a network failure;It prevents duplicate instances when launched again;It does not leave a console window open. For a developer or infrastructure engineer, this is a simple, functional alternative to a system-wide tunnel when a managed route is needed for only a few resources rather than changing the routing of the entire computer, including in situations involving restrictions imposed by internet providers.

Have you considered what the defining infrastructure challenges of the next few years will be? If not, my interview with Srinivas Chippagiri may have you thinking more about how AI could change the way you manage infrastructure and what that means for building reliable, secure systems. But before we get into all of that, I always like to start by learning a little more about the person behind the expertise. So, let's kick it off with where Srinivas's journey began! Tell me what initially drew you into the field of technology. My journey began in software engineering, drawn to what I still think is the hardest question in the field: how to build software systems reliably at scale? I didn’t want to just build systems. I wanted to understand and document why they behave the way they do, which is what led me into research. Over the years, I’ve published peer-reviewed papers and book chapters on cloud security, Kubernetes performance, distributed scheduling, and applied AI. For me, engineering and research feed each other. Building at scale gives me the hard problems, and writing about them forces the clarity that makes the next system better. You mentioned that engineering and research feed into each other, which, for some developers, can make it a bit harder to navigate. When you're faced with a difficult technical problem, how do you typically approach finding a solution? I start by quantifying the problem before I touch a design. Who depends on this, what breaks if it fails, and what “good” actually looks like in numbers. Then I design for reuse rather than for the immediate fix. A lot of my research work comes directly out of that discipline. My work on Kubernetes network performance and on distributed task scheduling and load balancing using nature-inspired optimization began as real engineering problems that I then studied rigorously and published. That loop is my process: observe a hard problem in production, model it, measure it, and generalize it into something reusable. I trust measurement over intuition, and I try to leave behind a pattern others can apply, not just a one-off solution. I like that you focus on creating solutions others can reuse rather than just addressing the immediate problem. With that in mind, what's something you wish more people understood about cloud-native development? That “moving to the cloud” and “being cloud-native” are not the same thing. Many teams lift-and-shift a monolith onto managed infrastructure, watch the bill climb, and conclude cloud-native was oversold. The real win isn’t the platform. It’s designing systems as reusable, observable, governed building blocks, with security and compliance treated as engineering problems from day one rather than checklists bolted on after an incident. I’ve argued this in my research too. Several of my papers and book chapters look at cloud security and the organizational side of cloud transformation, because the hardest part of cloud-native isn’t the technology. It’s the discipline and architecture around it. You brought up the importance of building systems with security and compliance in mind from the start. Looking ahead, what emerging technology or trend has your attention right now, and why? Applied AI in infrastructure, meaning using AI not just as the workload but to operate and self-manage the systems underneath it. It sits right at the intersection of my two research threads: distributed systems and applied AI. My published work has gone deep on both sides. In a recent journal article, my co-authors and I designed an IoT-based AI framework for real-time health monitoring that pairs edge and fog processing with federated learning and differential privacy, so intelligence runs close to the data while privacy is preserved. On the systems side, I’ve studied scheduling and optimization. What fascinates me now is the reliability question. As we let intelligent systems make operational decisions, how do we keep them observable, safe, and accountable? That’s the same reliability-first instinct that has driven my entire career, applied to a far more autonomous stack, and I think it’s the defining infrastructure challenge of the next few years. It's interesting that reliability has remained a priority throughout your career, even as the technology evolves. What's one lesson you've learned along the way that's had a lasting impact on how you approach your work? That your highest-impact work is the work other people can build on. That reframed how I work. I now deliberately design and document for reuse, whether that’s a system architecture or a paper. I optimize for the multiplier, not just the deliverable, and the recognition I value most isn’t finishing something. It’s seeing my approach become a pattern others adopt. And now for my final question, which is usually my favorite: When you’re not working or thinking about technology, how do you like to spend your free time? A lot of my off time still points back toward the community. I’m active in the professional cloud computing community, and I mentor younger engineers and researchers, which I genuinely enjoy. Outside of tech, I like unhurried time with family and getting away from screens entirely. Travel and being outdoors help me return to hard problems with a clearer head. I’ve found the best architectural ideas rarely arrive at a keyboard. Check out more of his content here.

Every system that ingests data from someone else's API eventually writes a poller. It starts as fifteen lines in a cron job: hit the endpoint, compare against the last timestamp, insert what's new. It works immediately, which is the problem — it keeps appearing to work long after it has started losing records. I maintain an ingestion layer that polls public discussion APIs continuously, and the gap between "the poller is running" and "the poller is correct" took considerably longer to close than I expected. What follows are the four failure modes that caused real data loss, and the patterns that fixed them. None of this is exotic. It is the set of things I wish had been in one place before I wrote the first version. Failure Mode 1: The Fixed Interval The naive poller runs on a fixed schedule — every 60 seconds, every source, forever. This is wrong in both directions simultaneously. High-volume sources produce more records between polls than a single page returns, so you lose everything past the page boundary. Low-volume sources return nothing on the vast majority of calls, burning quota that the high-volume sources needed. With a shared quota across sources, the fixed interval guarantees you are under-sampling exactly where the data is. The fix is to let observed yield drive the interval. If the last poll came back full, poll sooner. If it came back empty, back off. TypeScript interface PollState { sourceId: string; intervalMs: number; consecutiveEmpty: number; } const MIN_INTERVAL = 30_000; const MAX_INTERVAL = 1_800_000; // 30 min function nextInterval(state: PollState, recordsFound: number, pageSize: number): number { // Page came back full — we are probably truncating. Poll aggressively. if (recordsFound >= pageSize) { return MIN_INTERVAL; } if (recordsFound > 0) { // Productive but not saturated. Ease toward the current rate. return clamp(state.intervalMs * 0.8, MIN_INTERVAL, MAX_INTERVAL); } // Empty. Back off geometrically, but bounded. const factor = 1.5 ** Math.min(state.consecutiveEmpty, 6); return clamp(MIN_INTERVAL * factor, MIN_INTERVAL, MAX_INTERVAL); } function clamp(n: number, lo: number, hi: number): number { return Math.min(hi, Math.max(lo, n)); } The condition worth highlighting is recordsFound >= pageSize. A full page is not a success signal — it is evidence that records may exist beyond what you fetched. Treat it as a truncation warning and either paginate immediately or shorten the interval. Systems that treat a full page as "great, got everything" lose data proportional to source velocity, which means they lose the most from the sources that matter most. After deploying adaptive intervals, our total request volume dropped by roughly 40% while records captured went up, because quota moved from silent sources to active ones. Failure Mode 2: The Watermark Boundary The standard incremental pattern stores the timestamp of the newest record seen and requests everything after it: SQL SELECT * FROM source WHERE created_at > :watermark ORDER BY created_at This loses records, for three independent reasons. Clock skew. The upstream timestamp is assigned by their server, not yours. Distributed writers mean created_at is not guaranteed to be monotonic in the order records become visible to the API.Visibility lag. A record can be written at T but not appear in query results until T+2s due to replication or index lag. Poll at T+1s, advance the watermark past T, and that record is never returned.Ties. Multiple records sharing the same timestamp, split across a page boundary, silently drop the remainder when you use a strict >. The fix is to rewind the watermark by a safety margin and deduplicate downstream: TypeScript const OVERLAP_MS = 120_000; // tune to observed upstream lag function queryWindow(watermark: number): { since: number; until: number } { return { since: watermark - OVERLAP_MS, until: Date.now(), }; } This deliberately re-fetches records you already have. That is the point — you trade a small amount of redundant bandwidth for the guarantee that a record has to be missed by two consecutive polls to be lost permanently. Size OVERLAP_MS From measurement, not intuition. Log the delta between upstream created_at and your own ingest time, take the 99th percentile, and double it. Ours sat around 40 seconds at p99, so 120 seconds gives real headroom without meaningful cost. Failure Mode 3: Deduplication That Isn't Overlap windows only work if dedup is correct, and ID-based dedup handles less than people assume. Upstream records get edited. A post whose body changes after ingestion keeps its ID, so an INSERT ... ON CONFLICT DO NOTHING on the ID silently discards the update. Meanwhile, some sources reuse or re-issue identifiers across deletion, so the ID alone is not always a stable key either. Hash the fields that constitute a meaningful change, and store both: TypeScript import { createHash } from "node:crypto"; function contentHash(record: SourceRecord): string { // Only fields whose change means "this record is materially different." // Volatile counters (score, comment count) are excluded on purpose. const canonical = JSON.stringify({ id: record.id, title: record.title ?? "", body: record.body ?? "", author: record.author ?? "", }); return createHash("sha256").update(canonical).digest("hex"); } SQL INSERT INTO records (source_id, external_id, content_hash, payload, first_seen_at, last_seen_at) VALUES ($1, $2, $3, $4, now(), now()) ON CONFLICT (source_id, external_id) DO UPDATE SET content_hash = EXCLUDED.content_hash, payload = EXCLUDED.payload, last_seen_at = now(), revision = records.revision + 1 WHERE records.content_hash IS DISTINCT FROM EXCLUDED.content_hash; Two details carry most of the weight. The WHERE clause means the overlap window costs nothing when nothing changed — no write amplification, no spurious updated_at churn, and downstream consumers watching for changes don't get woken up by re-fetches. And excluding volatile counters from the hash is what makes this viable at all. Include a score that ticks every few seconds and every single poll registers as a change, which defeats the purpose and turns your overlap window into a write storm. Failure Mode 4: Retry Storms and Undetected Gaps When a rate-limited API returns 429, the instinct is to retry. Retry immediately, and you extend the penalty. Retry on a fixed delay across N workers, and they resynchronize into a thundering herd that hits the limit together, forever. Most rate-limited APIs tell you what you need in response headers. Read them instead of guessing: TypeScript interface RateLimitState { remaining: number; resetAt: number; } function parseRateLimit(res: Response): RateLimitState | null { const remaining = res.headers.get("x-ratelimit-remaining"); const reset = res.headers.get("x-ratelimit-reset"); if (remaining === null || reset === null) return null; return { remaining: Number(remaining), // Some APIs send seconds-until-reset, others an absolute epoch. Normalize. resetAt: Number(reset) > 1e9 ? Number(reset) * 1000 : Date.now() + Number(reset) * 1000, }; } async function backoff(attempt: number, res?: Response): Promise<void> { const retryAfter = res?.headers.get("retry-after"); if (retryAfter) { const ms = Number(retryAfter) * 1000; if (Number.isFinite(ms)) return sleep(ms); } // Full jitter — see AWS Builders' Library on backoff. const ceiling = Math.min(60_000, 1000 * 2 ** attempt); return sleep(Math.random() * ceiling); } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); Full jitter matters more than the backoff curve. Exponential backoff with a deterministic delay keeps clients synchronized; randomizing across the whole interval is what actually spreads load. Retry-After, When present, it is authoritative — honor it over your own calculation. The subtler problem is that these failures are invisible. A source that throws 429 for six hours produces no errors in your application logs if the poller catches and backs off correctly. It just quietly returns nothing, and your data has a hole in it that nobody notices until someone asks why a report looks wrong. Detect gaps explicitly rather than inferring them from error rates: TypeScript async function detectGaps(sourceId: string): Promise<Gap[]> { const { rows } = await db.query(` SELECT window_end AS gap_start, LEAD(window_start) OVER (ORDER BY window_start) AS gap_end FROM poll_log WHERE source_id = $1 AND status = 'ok' AND window_start > now() - interval '48 hours' ORDER BY window_start `, [sourceId]); return rows .filter(r => r.gap_end && r.gap_end - r.gap_start > GAP_THRESHOLD_MS) .map(r => ({ sourceId, from: r.gap_start, to: r.gap_end })); } This requires logging every poll attempt with its requested window and outcome — a table most people skip because it feels like overhead until the first time it tells you about four hours of missing data you would otherwise never have found. Feed detected gaps into a low-priority backfill queue that runs when quota is available. What Actually Matters If you take three things from this: Log the window, not just the result. Without a record of what range each poll requested, gaps are undetectable. This is the single highest-value thing to add to an existing poller.Overlap deliberately, dedup by content. A safety margin plus content-hash dedup converts "we might have missed something" into "a record must be missed twice to be lost."Treat a full page as a warning. It is the most common silent truncation point in production pollers, and it is one comparison to check for. None of these are difficult. They are just easy to omit, and their absence produces a system that looks healthy while quietly dropping records. I work on the ingestion layer at SignalMelo, which polls public discussion sources continuously — the patterns above came out of production incidents there. Disclosure: it's a commercial product I help build.

Intermittent test failures in a distributed CI/CD environment can slow down development and make failures harder to investigate. When an automated test suite fails on a local dev or staging machine, an engineer can view local logs, attach a debugger, or manually follow state changes. But when tests execute in a temporary container, AWS Lambda, or an agentless runner, post-mortem debugging is significantly more involved. In a traditional setup, a failed build only shows a generic assertion error (504, ElementNotFoundException, etc.). An engineer may need to manually sift through CloudWatch log groups for an approximate timestamp, query Splunk, or reason about state changes across downstream microservices to understand why the test failed. Once the log retention period expires or if the execution did not have a shared, identifiable correlation ID, the engineer has no further information about the cause of the failure. An enterprise test framework should save the test results and logs before the test runner shuts down. This article explores how to capture rich failure artifacts, stitch distributed logs (via correlation ID), and preserve retry states per-attempt. 1. Capturing Evidence Before the Ephemeral Runner Exits Capturing telemetry on failure seems trivial, but ephemeral environments introduce special exit modes which may skip cleanup routines: Assertion failures: The framework captures the exception; standard teardown hooks executeExecution timeouts: The process or handler is externally aborted; finalizers may be skippedHard kills (OOM/SIGKILL): The underlying container or VM is killed; no in-process code executed Your test harness needs to capture evidence in gracefully terminated failure modes and before hard timeouts Managing Runner Lifecycles AWS Lambda: A test runner deployed as a Lambda must check context.get_remaining_time_in_millis(). If it falls below a certain threshold (2,000 ms), it must terminate early, flush diagnostic buffers, and write metadata to S3. Your ephemeral /tmp storage defaults to 512 MB; holding large trace files or video recordings in memory would lead to out-of-memory (OOM) errors, so artifacts should be streamed directly to object store.AWS Fargate/ECS: When a task is stopped, it will receive a SIGTERM followed by a SIGKILL after a configured stopTimeout (30 seconds). Signal handlers in your test suite must capture SIGTERM and begin uploading artifacts.CI Build Agents (Bamboo/Jenkins): Agent-side logs are ephemeral and will be deleted when the agent's workspace is cleaned. Remote execution tasks should push telemetry to central storage (like AWS S3) and pass back the stable HTTP reference, not the local agent's file path. Pytest Hook Architecture In a Python setup, pairing a pytest_runtest_makereport hookwrapper with an autouse fixture guarantees that failure status is captured and attached to the test item before final teardown. Python # conftest.py import pytest @pytest.hookimpl(tryfirst=True, hookwrapper=True) def pytest_runtest_makereport(item, call): # Execute all other hooks to obtain the report object outcome = yield rep = outcome.get_result() # Store the report on the item for fixture access setattr(item, f"rep_{rep.when}", rep) @pytest.fixture(autouse=True) def failure_evidence_collector(request): yield # Code below executes during fixture teardown rep_call = getattr(request.node, "rep_call", None) if rep_call and rep_call.failed: execution_id = request.config.getoption("--execution-id", default="local") attempt = getattr(request.node, "execution_count", 1) # Extract environment details captured at run start env_metadata = { "test_name": request.node.name, "stage": getattr(request.node, "current_stage", "unknown"), "commit_hash": os.getenv("GIT_COMMIT", "HEAD"), "aws_request_id": getattr(request.node, "aws_request_id", None) } upload_failure_artifacts(execution_id, attempt, env_metadata) 2. Run IDs and Cross-Service Correlation Although capturing client-side logs is easy, if an integration test makes an HTTP call to a microservice architecture, the underlying defect may be found in downstream service logs. Without a unified identifier, correlating a client-side test failure with backend anomalies requires time-consuming manual timestamp matching. Propagating the Correlation Context To stitch frontend execution to backend telemetry, every outgoing request from the test runner must inject a unique tracking ID into transport headers. HTTP headers: Standardize on custom headers such as X-Correlation-ID or X-Run-ID. Initialize this ID once on the test client's session object so that every HTTP call automatically inherits itOpenTelemetry baggage: The standard W3C Trace Context (traceparent) requires a rigid 32-hex trace ID and 16-hex span ID format. CI build keys (e.g., PROJECT-JOB1-102) will not fit in standard trace fields. Instead, carry build context in the W3C baggage header (baggage: ci_build_key=PROJECT-JOB1-102,test_id=test_user_checkout)Async boundaries (SQS/SNS): HTTP headers do not automatically transition across message queues. When a test triggers an async workflow via AWS SQS, inject correlation metadata in the Message attributes. If there is an SNS in front of SQS, enable raw message Delivery in order to prevent SNS from wrapping the message payload in a JSON envelope (SQS attributes would need to be explicitly parsed by the consumer).Unowned platform services: Managed components like AWS API Gateway or Application Load Balancers (ALBs) will not process your application headers, but they will include their own tracking keys. API Gateway returns x-amzn-requestid in every response header, while ALBs preserve incoming X-Amzn-Trace-Id headers and record them in access logs. When an assertion fails, the test framework outputs a pre-formatted log query targeting the exact correlation key, enabling one-click diagnostic retrieval in CloudWatch Logs Insights or Splunk. MS SQL -- CloudWatch Logs Insights Query fields @timestamp, @message, service_name, log_level, error.type | filter correlation_id = "BUILD-8492-test_checkout_flow-att1" | sort @timestamp asc | limit 500 Fallback strategy: For legacy third-party microservices that do not accept or log correlation headers, configure the evidence collector to generate a time window query that is constrained to the specific UTC start and end timestamps of that specific test case execution. 3. Retries Kept Per Attempt Retries as implemented, say, by pytest-rerunfailures are often used to mitigate the risk of infrastructure-related failures, but such practice has its costs: retrying all tests that failed on the first attempt may mask intermittent bugs in the application under test. If a test fails on the first attempt but passes on the retry, the build will be marked as successful, even though the intermittent bug may still be present. If a test failed on attempt 1 due to a race condition but passed on attempt 2 because a resource freed up, overwriting attempt 1's evidence erases the bug's trace. Standard JUnit XML output often only records the final execution state, hiding the initial failure. Never write diagnostic files to a static path (e.g., s3://bucket/logs/test_checkout.log). Telemetry storage paths must include both the run identifier and the attempt counter. PowerShell s3://ci-test-evidence-bucket/ └── build-key=PROJECT-8492/ └── test_checkout_user/ ├── attempt-1/ │ ├── metadata.json │ ├── client_session.log │ └── failure_screenshot.png └── attempt-2/ ├── metadata.json └── client_session.log Once per-attempt isolation is implemented, diff analysis can be performed: JSON // S3 Path: build-key=PROJECT-8492/test_checkout_user/attempt-1/metadata.json { "test_name": "test_checkout_user", "execution_id": "PROJECT-8492", "attempt": 1, "status": "FAILED", "duration_seconds": 12.4, "http_status_code": 503, "target_host": "ip-10-0-4-12.ec2.internal", "failure_reason": "ServiceUnavailable: Database connection pool exhausted" } If attempt 1 fails with a 503 Service Unavailable from a particular host, and attempt 2 succeeds against a different host, the delta immediately indicates a node-specific configuration drift or pool exhaustion, not a code-level assertion flaw. 4. Keeping Artifacts Safe, Small, and Reachable Collecting unlimited telemetry rapidly leads to two major issues: credential leakage and excessive storage costs. Automated Redaction at Write Time Diagnostic artifacts (typically HTTP request/response dumps) often contain authorization tokens, cookies, or personal data. They must be redacted from memory before writing to disk: Strip headers that match Authorization, Cookie, X-Api-Key, or Set-Cookie.Remove signature tokens from URL parameters. Note that AWS S3 Presigned URLs act as credentials, and logging them directly would accidentally expose an ability to read/write to any S3 bucket the application has access to. Storage Optimization and Retention To keep storage costs at enterprise scale under control, enforce explicit lifecycle and collection rules: Selective retention: only store full-payload logs and screenshots if a test failed. Store only a short tail of a log output for passing tests.S3 Lifecycle Rules: configure S3 bucket lifecycle rules to delete failed test artifacts 30 days after creation, and passing test run artifacts 3 days after creation. It should be configured with either prefix- or tag-based policies.Do not put raw S3 presigned links inside JUnit XML or CI failure logs. Create links using a stable internal proxy/route that is capable of signing requests dynamically upon access. Presigned links created with temporary IAM role credentials will become invalid as soon as the web server process that issued them reloads. It would make sense to have it turned into a dead link if an engineer looks at a CI failure several hours (or days) after its occurrence. Failure Summary Surface Inject the direct artifact link and correlation ID straight into the framework's assertion exception message. Because virtually all CI engines parse and display the top-level exception string in build summaries, engineers can jump directly to the evidence: Python # Custom assertion failure formatter def raise_with_evidence(message: str, correlation_id: str, evidence_url: str): formatted_msg = ( f"{message}\n" f"--------------------------------------------------\n" f"Correlation ID : {correlation_id}\n" f"Evidence Bundle: {evidence_url}\n" f"--------------------------------------------------" ) raise AssertionError(formatted_msg) Architectural Limitations to Consider While evidence collection is made dramatically easier by the structured nature of the solution, there are a few boundary conditions that should be considered: Hard OS/platform kills: An AWS Lambda function reaching its 15-minute hard limit or worker node power loss will prevent in-process code execution. Secondary monitoring (container stdout monitoring via CloudWatch/Datadog) becomes a necessary alternative.Unowned external third parties: SaaS downstream APIs or gateway services will not instrument custom correlation IDs. In these scenarios, evidence would be strictly client-side HTTP request timestamp logging.Redaction leakage: Regex sanitization could miss non-standard secret key formats in unstructured response bodies. References W3C Recommendation: Trace Context - W3C Recommendation (2020).Luo, Q., Hariri, F., Eloussi, L., & Marinov, D.: "An Empirical Analysis of Flaky Tests" (ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014).Google Testing Blog: "Flaky Tests at Google and How We Mitigate Them" (2016).AWS Documentation: AWS Lambda Context Object in Python & ECS Task Lifecycle.Pytest Documentation: Writing and using plugins & pytest-rerunfailures specification.

Most explanations of HTTP verbs in ASP.NET Core Web API stop at the dictionary definition: GET reads, POST creates, PUT updates, DELETE removes. That's true, but it doesn't explain why real enterprise APIs often bend these rules — and understanding why matters more in practice than reciting the definitions. This article walks through HTTP verb selection using a real HR/Payroll workflow scenario: submitting, updating, and removing leave requests. The Textbook Mapping In a typical Web API controller, the four core verbs map to CRUD operations like this: C# [ApiController] [Route("api/[controller]")] public class LeaveRequestController : ControllerBase { [HttpGet("get-emp-details")] public IActionResult GetEmployeeDetails(int finSlno, string empId, int createdBy) { // fetch and return data return Ok(employeeDetails); } [HttpPost("save-leave-request")] public IActionResult SaveLeaveRequest([FromBody] LeaveRequestDto request) { // create a new record return Ok(result); } [HttpPut("update-leave-request")] public IActionResult UpdateLeaveRequest([FromBody] LeaveRequestDto request) { // update an existing record return Ok(result); } [HttpDelete("delete-leave-request/{id}")] public IActionResult DeleteLeaveRequest(int id) { // remove a record return Ok(result); } } This is correct, and it's what most tutorials teach. But real enterprise systems — particularly HRMS and Payroll applications — frequently deviate from this pattern for reasons that only make sense once you understand what's actually happening to the data underneath. Why "Delete" Often Isn't Really DELETE Consider a leave request that an employee wants to withdraw. The textbook answer says: call [HttpDelete], remove the row from the database, done. In practice, most HR and Payroll systems don't want a "delete" action to genuinely erase the record. There are real, practical reasons for this: Audit trails – organizations often need to prove what happened and when, especially for anything touching attendance, payroll, or approvals. Permanently deleting a leave request destroys that history.Compliance – payroll systems in particular are subject to regulations that may require retaining records for a defined period, regardless of whether the user considers them "deleted."Traceability during disputes – if an employee claims a leave request was withdrawn incorrectly, having the original record (even if marked inactive) is the only way to investigate what actually happened. Because of this, many systems implement what's called a soft delete: instead of removing the row, the operation updates a status field: C# public class LeaveRequest { public int LeaveRequestId; public string EmpID; public int Status; // 0 = Pending, 1 = Approved, 2 = Withdrawn } A "delete" action becomes, in reality, an update to Status: C# [HttpPost("save-delete-role-mapping")] public IActionResult SoftDeleteLeaveRequest([FromBody] List<LeaveRequestDto> requests) { foreach (var req in requests) { req.Status = 2; // mark as withdrawn, don't remove the row } // save changes return Ok(result); } Because this is technically an update operation, not a true deletion, using [HttpPost] (or [HttpPut]) instead of [HttpDelete] is a defensible, common real-world choice — not a mistake or a shortcut. The verb should reflect what actually happens to the data, not just what the button on the screen says. Why POST Sometimes Wins Even for Genuine Deletes There's a second, more practical reason POST shows up in place of DELETE in real systems: payload complexity. [HttpDelete] is traditionally designed around simple identifiers, often passed in the URL: C# [HttpDelete("delete-leave-request/{id}")] public IActionResult Delete(int id) { // remove a single record by id return Ok(); } This works cleanly for deleting one record by one ID. But real operations are frequently batch operations — deleting or withdrawing multiple leave requests at once, each potentially needing additional context (who's performing the action, why, which workflow step it affects): C# [HttpPost("save-delete-role-mapping")] public IActionResult BatchDelete([FromBody] List<LeaveRequestDto> requests) { // process a batch of records, each carrying extra context return Ok(); } Passing a list of complex objects in a DELETE request body is technically possible but goes against how DELETE has traditionally been used and supported across tooling. POST, which has always supported rich request bodies, handles this case more naturally — which is exactly why it's common to see "delete" operations implemented as POST endpoints in real enterprise codebases. How This Looks From the Angular Side The verb choice on the backend directly shapes how the frontend calls it. In an Angular service, each HTTP verb maps to a specific HttpClient method: C# // GET - fetching data, no body needed getEmployeeDetails(empId: string): Observable<EmployeeDto> { return this.http.get<EmployeeDto>(`${this.apiUrl}/get-emp-details`, { params: new HttpParams().set('empId', empId) }); } // POST - creating a new record, or a batch/complex operation saveLeaveRequest(request: LeaveRequestDto): Observable<BaseResultModel<boolean>> { return this.http.post<any>(`${this.apiUrl}/save-leave-request`, request); } // POST used for a soft delete - batch of complex objects saveDeleteWorkFlows(models: LeaveRequestDto[]): Observable<BaseResultModel<boolean>> { return this.http.post<any>(`${this.apiUrl}/save-delete-role-mapping`, models); } Notice that both the create and the soft-delete operations use .post() from the Angular side, mirroring the [HttpPost] decorator on the backend. This consistency matters: if the backend soft-deletes via POST but the Angular service tries to call .delete(), the request will fail or hit the wrong endpoint entirely, since Angular's HttpClient.delete() doesn't send a request body the same way .post() does by default. Keeping verb usage consistent across frontend and backend isn't just a style preference — mismatched verbs are a common, easy-to-miss source of failed API calls that can take a while to diagnose, since the error often looks like a routing problem rather than a verb mismatch. A Note on Idempotency A related concept worth understanding alongside verb selection is idempotency — whether calling the same operation multiple times produces the same result as calling it once. GET, PUT, and DELETE are conventionally idempotent: fetching the same data twice returns the same result; updating a record to the same values twice leaves it in the same final state; deleting an already-deleted record (in systems that handle this gracefully) doesn't cause additional harm. POST is conventionally not idempotent — calling it twice can create two records instead of one. This matters directly for the soft-delete pattern discussed above. If SoftDeleteLeaveRequest is implemented as a POST that simply sets Status = 2, calling it twice by accident (say, from a double-clicked button or a retried failed request) produces the same end state — the record is still just marked as withdrawn. This is a case where a POST endpoint behaves idempotently in practice, even though POST isn't required to be idempotent by convention. It's worth being deliberate about this: an endpoint that changes a status flag is safer to retry than one that increments a counter or appends to a list, and understanding which category an operation falls into helps predict how it will behave under real-world conditions like network retries or double submissions. A Practical Rule of Thumb None of this means HTTP verb conventions don't matter — they're still a useful default. The practical guidance is: Use GET for anything that only reads data and has no side effectsUse POST for creating new records, and also for operations — including some deletes — that involve complex payloads or don't map cleanly to a single identifierUse PUT for straightforward updates to an existing, fully-identified recordUse DELETE for simple, single-identifier removals where a true, permanent delete is actually intended When you encounter an API that doesn't follow the textbook mapping exactly — like a "delete" endpoint implemented as POST — it's worth asking why, rather than assuming it's wrong. Often, as in the soft-delete case above, there's a real business reason: the data isn't actually being removed; it's being marked, and the verb reflects that reality rather than the UI label. Takeaway HTTP verb selection in a real Web API isn't just about matching a CRUD acronym to a decorator. It's about accurately representing what actually happens to the underlying data. A "delete" that's really a status update should probably be a POST or PUT, not a DELETE — and recognizing that distinction is a sign of understanding the system, not a shortcut around convention.

AI coding agents can write a working FastAPI endpoint in seconds, refactor a module, or track down a bug across a few files. Anyone who's used Cursor, Claude Code, or a similar tool has felt how genuinely useful that is. What's much less settled is a harder question: how capable are these things, actually, and where do they reliably fail? That's not a rhetorical question. Coding agents pass some benchmarks impressively and stumble on tasks that, at a glance, look easier than ones they've already solved. They can produce code that passes a quick visual check but fails on an edge case a careful engineer would have caught. And critically, an agent under evaluation has an incentive problem a human engineer mostly doesn't: if the way it's graded is visible or guessable, it can satisfy the grading without actually doing the work. For example, it can pass the test without fixing the bug, or produce output that looks right without being right. Knowing whether an agent is actually good at something, not just good at appearing to be, is genuinely hard, and it's why a serious discipline of AI agent evaluation exists. Benchmarks like SWE-bench and Terminal-Bench, and the harnesses that run them, are built to solve exactly this problem. This article is about that problem, worked from the ground up. You don't need to know anything about evaluation frameworks to start. You just need to have written software before and, at some point, asked yourself, "How would I actually know if this worked?" We'll reason through what a rigorous test of an AI agent actually requires, then look at a real open-source framework (Harbor) that implements it, and finally build two real tasks with it: a simple one from nothing, and a harder one against an existing project with a real bug in it. The Problem, Before the Tool How do you test whether an AI coding agent can actually do a task? Think about what that requires as an engineer before any AI-specific tooling enters the picture. First, you need somewhere for it to work that isn't your actual machine. If you're going to let something, human or AI, run arbitrary commands and edit files, you want to contain it. You also want every attempt to start from the exact same clean state, or your comparisons are meaningless. This is just... a container. Nothing novel yet. Second, you need to tell it what to do, in plain language, without accidentally telling it how. This sounds obvious, but it's the part people get wrong first. If your task description says "open the file, find the loop on line 12, change <= to <," you haven't tested whether the agent can debug anything. What you have tested is whether it can follow a numbered list. The instruction has to describe the symptom, the way a real bug report does, not the fix. Third, you need a way to check whether it succeeded, and that check cannot be visible to the thing being tested. Think about why. If the agent can read your test file, it doesn't need to solve the problem since it can just read the assertions and produce output that satisfies them directly, or edit the test itself so it passes trivially. A test the subject can see stops being a test of ability and becomes a test of "can you find the answer key." This one constraint is the single most important idea in this entire article, and everything below follows from it. That's it. Isolated environment, plain-language task, hidden check. If you understand why those three things are necessary, you already understand Harbor conceptually. Everything else is just how one specific framework implements them. Harbor Is One Implementation of That Idea Harbor is an open-source framework, built by the team behind the Terminal-Bench benchmark, for running exactly this pattern at scale: put an AI agent in a container, give it a task, and grade the result with a check it never sees. Once you have the three-part model above in your head, Harbor's file layout stops looking arbitrary: Markdown my-task/ ├── instruction.md # the plain-language ask ├── task.toml # metadata/config ├── environment/ │ └── Dockerfile # the isolated place the agent works ├── solution/ │ └── solve.sh # a reference answer (kept hidden) └── tests/ ├── test.sh # the hidden check, runs after the agent └── test_outputs.py # what the check actually verifies Map this back to the three requirements. environment/ is the isolated container. instruction.md is the plain-language task. tests/ is the hidden check. Notice that the tests/ is a sibling folder to environment/, not something nested inside it. That's not a style choice; it's the whole point: tests/ never gets copied into the container the agent works in. The agent can't read what it's being graded on, for the same reason a student can't see the answer key during an exam. solution/ is new — we haven't justified it yet. It holds a reference fix: what a correct answer actually looks like. Why does that need to exist if the agent is going to produce its own answer anyway? Because before you trust a test, you need to know two things about it: does a correct answer pass it, and does a wrong answer fail it. You can't know either of those without something to run through the test first. That's what solution/ is for. It is not for the agent, but for you, the task author, to validate your own work. The Lifecycle, Still Without Writing Any Code Harbor calls one agent's attempt at one task a trial. When you want to sanity-check a task you've built, rather than pointing a real (slow, costly) AI agent at it, Harbor gives you an Oracle agent. It does no reasoning at all; it just runs your solution/solve.sh directly and submits whatever that produces. If Oracle doesn't pass your own test, your task is broken, full stop, before any real agent has spent a single second on it. There's a mirror image of this you'll want to, once we get further in: a NOP agent, which submits the environment completely untouched. If NOP also passes your test, that tells you something important: your test isn't actually checking anything. A check that accepts a wrong answer isn't a check. Oracle should pass, NOP should fail. We will come back to it as the real discipline behind building a task well, not just building one that runs. The result of a trial is a reward, usually just 1 for pass or 0 for fail, written by your test script to a specific file Harbor reads. Build One From Scratch: A Simple FastAPI Endpoint Let's make all of this concrete with something small enough to hold in your head at once. The task: build a FastAPI endpoint, POST /add, that takes {"a": 2, "b": 3} and returns {"result": 5}. The instruction — plain language, states the contract, not the implementation: GitHub Flavored Markdown # Build a FastAPI "Add" Endpoint Create a FastAPI application at `/app/main.py` that exposes a single endpoint: - Method: `POST`, Path: `/add` - Request body: `{"a": <number>, "b": <number>}` - Response body: `{"result": <number>}`, where `result = a + b` The app must be importable as `main:app`. Accept both integers and floats. If a field is missing or non-numeric, a `422` is fine (FastAPI's default validation behavior handles this). The environment — just enough installed for the agent to do the work, deliberately not including a working solution: Dockerfile FROM python:3.11-slim WORKDIR /app RUN pip install --no-cache-dir fastapi uvicorn pytest httpx The reference solution, which only you and Oracle ever see: Shell #!/bin/bash cat > /app/main.py << 'EOF' from fastapi import FastAPI from pydantic import BaseModel app = FastAPI() class AddRequest(BaseModel): a: float b: float @app.post("/add") def add(payload: AddRequest): return {"result": payload.a + payload.b} EOF The hidden test, using FastAPI's own TestClient to hit the endpoint the way a real caller would: Python from fastapi.testclient import TestClient from main import app client = TestClient(app) def test_add_integers(): resp = client.post("/add", json={"a": 2, "b": 3}) assert resp.status_code == 200 assert resp.json() == {"result": 5} def test_missing_field_returns_422(): resp = client.post("/add", json={"a": 2}) assert resp.status_code == 422 The verifier entrypoint, which runs after the agent is done, and translates pytest's result into the file Harbor actually reads: Shell #!/bin/bash cp "$(dirname "$0")/test_outputs.py" /app/test_outputs.py cd /app pytest test_outputs.py -v if [ $? -eq 0 ]; then echo 1 > /logs/verifier/reward.txt else echo 0 > /logs/verifier/reward.txt fi Two details worth stopping on, because they're easy to get backward the first time. Harbor doesn't look at pytest's exit code directly. Instead, it reads whatever's in /logs/verifier/reward.txt, so your script has to write that explicitly; skip it, and you get an error even if your tests ran fine. And don't put set -e at the top of that script. The reason is that if pytest fails, set -e would kill the script on that line, before it ever reaches the part that writes the reward file, turning a clean "fail" into a confusing error instead. Run it: Shell harbor trials start -p my-task --agent oracle # should be reward: 1.0 harbor trials start -p my-task --agent nop # should be reward: 0.0 If both of those come back the way you expect, you've built a working task, and you've proven it's working, not just assumed so. Leveling Up: Debugging an Existing Project The build-from-scratch pattern above teaches the mechanics well, but it's not the most common real-world use case. More often, you're not asking an agent to write something new. Many times you're asking it to find and fix a problem in something that already exists and mostly works. That's a meaningfully different kind of task, and it's the one I'd actually recommend practicing next, on a real project. I built one of these against a small open-source FastAPI currency converter — full code at https://github.com/iyanuashiri/frankfurter-http-error so you can clone it, run the task yourself, and see every file mentioned below in context. The project itself is a normal small service: users authenticate with an API key, spend "credits" on requests, and the service proxies out to a public exchange-rate API (Frankfurter) for live rates. It already has a real test suite — six files covering auth, conversions, currency listing, historical rates, and history — none of it written with Harbor in mind. That matters: the task doesn't require a specially-crafted project; it works against ordinary application code. The task folder wraps that project without modifying its shape: GitHub Flavored Markdown frankfurter-http-error/ ├── instruction.md ├── task.toml ├── environment/ │ ├── Dockerfile │ └── backend/ ← the actual FastAPI project, untouched │ ├── app/ │ ├── frankfurter/ │ │ └── rest_adapter.py ← where the bug lives │ ├── tests/ ← the project's own existing suite │ └── pyproject.toml ├── solution/ │ └── solve.sh └── tests/ ├── test.sh └── test_outputs.py The bug is one character in the HTTP client the app uses to talk to the external rate API: Python # environment/backend/frankfurter/rest_adapter.py if response.status > 400: # should be: >= 400 error_text = await response.text() raise HTTPException(status_code=response.status, detail=f"API request failed: {error_text}") return await response.json() A 400 Bad Request from the external API — a real, common response when an invalid currency code is requested — no longer counts as an error. > 400 lets it fall straight through to return await response.json(), handing the caller whatever error payload the external API sent back, as if it were a valid exchange rate. Nothing crashes. The bug corrupts data silently, which is a more realistic and more dangerous failure mode than one that throws — and a good one for testing whether an agent actually reasons about the boundary instead of skimming the condition. The instruction stays a symptom report, matching the actual file shipped with the task: GitHub Flavored Markdown When a request is made with an invalid or unsupported currency code, the external Frankfurter API returns an error response. Instead of surfacing that as a clear error to our users, something downstream is returning malformed or unexpected data without raising any exception. Investigate /app/frankfurter/rest_adapter.py, specifically how HTTP error responses from the external API are detected and handled. No mention of 400 specifically, no mention of the operator. The agent has to trace the request path and find the exact boundary itself. Shell harbor trials start -p . --agent oracle # reward: 1.0, expected harbor trials start -p . --agent nop # reward: 1.0, NOT expected NOP scoring 1.0 on a task where the bug was never fixed is the clearest possible signal a test suite is hollow. The fix was adding the exact boundary case: Python @pytest.mark.asyncio async def test_get_raises_exactly_at_400_boundary(): """Status exactly 400 (the boundary the >= comparison must include) must raise.""" adapter = RestAdapter() url = f"https://{adapter.hostname}/{adapter.ver}/latest?base=XXX&symbols=USD" with aioresponses() as mocked: mocked.get(url, status=400, payload={"message": "bad request"}) with pytest.raises(HTTPException) as exc_info: await adapter.get(url) assert exc_info.value.status_code == 400 Along with a companion test one status code below the boundary (399, which must succeed), so the suite proves both sides of the >= — not just that something raises somewhere above 400. After that change, NOP correctly dropped to reward: 0.0. That's really the whole lesson of debugging-task authoring, distilled: a held-out test only counts if it fails on the buggy version and passes on the fixed one. If it would pass either way, it isn't a test, it's decoration. Prove this for every task you build — Oracle passes, NOP fails — the same way you'd want to prove it for the build-from- scratch example earlier. Neither check alone tells you the task is good; you need both. What to Actually Try If you've followed this far, you understand the whole shape of the problem, not just Harbor's specific commands: isolate the environment, describe the symptom without describing the fix, hide the check from the thing being checked, and prove your check can fail before you trust that it can pass. Clone the repo, run the credits task yourself, then try seeding a different bug into the same project and building your own task around it. The framework mechanics are secondary. The discipline of "can I prove this task actually measures what I think it measures" is the actual skill, and it transfers to any evaluation framework you'll encounter after this one.

You open the page, put your text or file in, press Transform, check the result, and download it. Private details such as email addresses, phone numbers, card numbers, passwords, and secret keys are swapped for harmless labels like [EMAIL_1]. Everything happens inside your own browser tab. Why This Exists If you test software or manage a product, you constantly send things to other people: a bug report with an attached log, a screenshot of a customer problem, a contract, an ID card photo. Those files often contain details that should not leave the team: a customer's name, a phone number, a login token, a password that scrolled past in a terminal. Most "redaction" websites ask you to upload the file first. That means you hand over the very secret you wanted to protect. This tool does the opposite: it runs on your computer, so the file never travels anywhere. How Safe Is It? Here is what is happening behind the scenes, in plain words, and what the tool can and cannot promise. 1. Your File Stays on Your Computer The page is a single program that runs in your web browser. When you paste text or choose a file, the program on your device reads it. There is no server receiving it, no account to create, and no place where your file is saved. Check it yourself in 20 seconds Open the tool and wait for it to load.Turn off your Wi-Fi or switch on airplane mode.Paste some text and press Transform. It still works because nothing needs the internet. One honest detail: when you open a PDF or Word file for the first time, the page downloads a few helper programs (the code that knows how to read those formats) from well-known public code libraries. That code comes in. Your file never goes out. Plain text works without any download. 2. "Masked" Means Replaced, Not Just Covered File typeWhat you get backText, logs, JSON, HARThe private value is replaced by a label. The original value is not in the result.PDFEach page is re-drawn as a picture with dark boxes over the private parts. The new PDF has no hidden text layer, so you cannot select or copy the covered words. (Trade-off: the new PDF is not searchable.)Word (.docx)The text is replaced inside the document, so the fonts and layout stay. Author names, "last saved by", company name, and the small preview picture are cleared. Pictures can be blacked out or removed.Images (JPG, PNG, WebP)The covered area is painted over in the picture itself, then the image is saved again. Hidden camera and location details (EXIF) are not carried over. 3. The Same Value Always Gets the Same Label If the same email appears ten times, it becomes [EMAIL_1] all ten times. The person you share it with can still follow what happened ("[EMAIL_1] logged in, then [EMAIL_1] failed to pay") without knowing who it is. 4. What It Is Not Please read this part. It is a safety net, not a guarantee. The tool finds common patterns very reliably (emails, phone numbers, card numbers, keys, passwords). Names, company names, and project code names are harder. It catches names that follow a label such as "Customer:" or "Name:", but a name sitting alone in a sentence can be missed.It cannot read text inside pictures. A screenshot is just a picture to it. You draw boxes over the private parts yourself (use case 4).Always look at the right-hand side before you share. It takes thirty seconds and is the most important step.It is not a legal or compliance certificate. If your company has rules for handling regulated data, follow them. A good habit: try it first with made-up data (like the examples below), then with the real thing. What It Finds Automatically CategoryExamplesContact detailsEmail addresses, phone numbers, street addresses, city and stateMoney and identityCredit card numbers (checked so random numbers are not flagged), IBAN, social security style numbersSecretsPasswords, access tokens, API keys, cloud keys, long random-looking stringsNetworkIP addressesNamesNames after a label (Customer, Name, Contact, Prepared by...) or a title (Mr, Dr...)Anything elseYou select it, pick a type, and it is masked everywhere it appears All the examples below use invented people and numbers. 1. Paste Text From the Clipboard Best for log lines, error messages, chat transcripts, or anything you can copy. Copy the text you want to share.Click inside the Original box on the left and paste (Ctrl+V, or Cmd+V on a Mac).Click the blue Transform button in the middle.Read the Masked preview on the right. Click Copy text and paste it into your email or ticket, or click Download. Before: Fake log lines pasted into the left box. They contain an email, a login token, a phone number, a card number, and a password. After: The right side shows labels instead of the private values. The chips under the boxes list what was found. Missed something? Select the word with your mouse (or double-click it), choose what it is (name, company, secret...) in the small pop-up, and press OK. Every match turns coloured. Press Transform again. 2. Import a Word File and Mask the Text and the Picture Best for reports, contracts, resumes, and case notes. The masked file keeps the same look as the original. Click Choose file (top of the left side) and pick your .docx file. It appears with its original fonts and layout.If the document contains pictures, set Pictures in the Word file to Black out (solid black) or Remove (blank space). The default, Keep, leaves them as they are.Click Transform. The right side shows the masked document.Click Download. You get a normal Word file named masked-…docx. Step 2: The report opens on the left. The Pictures menu is set to Black out. Before: Further down the same page, the report contains a picture of an ID card (fake). After: Names, email, phone, address, and the key are replaced by labels, and the ID card picture below them is solid black. The page layout is unchanged. Good to know: Pictures are never hidden unless you choose Black out or Remove. Older .doc files are not supported: open them in Word and use Save As → .docx first. 3. Import a PDF, Auto-Mask, Then Add Your Own Mask Best for invoices, statements, scanned forms, and anything that must stay a PDF. Click Choose file and pick the PDF. The pages show on the left exactly as they look in the file.Click Transform. The right side shows the pages with dark boxes and labels over the private parts.Add your own mask: the tool does not know that "Project Falcon" is secret. Select those words, choose a type in the pop-up (here, Secret/ID), and press OK.Click Transform again. The right side updates and now covers your custom words too.For a picture, a signature, or a whole area, click Draw box and drag a rectangle over it before you press Transform.Click Download to save the masked PDF. Step 1: The PDF opens with its original layout. Step 2: After Transform, the name, email, phone, address, and card number are covered. The secret project name is still visible because the tool cannot guess it. Step 3: Select the words, choose a type, and press OK. Step 4: The right side now covers the custom words, too. The coloured chip above the page shows your custom mask. Step 5: After downloading, the status line confirms the saved file. Why the new PDF cannot be searched: To make sure nothing hides underneath the boxes, each page is saved as a picture. That is also why the covered words cannot be copied out of the file. 4. Import a JPEG Photo and Hide Parts With a Black or White Box Best for screenshots, photos of documents, and ID cards. This is done by hand, because the tool does not read text inside pictures. Click Choose file and pick a .jpg, .png or .webp image. It appears on the left.Drag with your mouse to draw a box over each private part. Click a box to move it or resize it from the corners. Press Delete to remove it, or Ctrl+Z to undo.Pick the color in the first drop-down: Black, White (blanks the area out), or Pixelate. Use Black or White for anything readable. Pixelated text can sometimes be recovered.Click Transform and check the right side.Click Download. You can keep the same format or save as PNG or JPEG. Draw: Four red boxes over the photo, name, ID number, and contact lines (fake card). Black: The covered areas are solid black. The rest of the card is unchanged. White: Same boxes, white style. Handy when the background is light, and you want a clean look. Bonus privacy: The saved picture is recreated from scratch, so hidden details like the camera model, date, and GPS location don't carry over. A 30-Second Checklist Before You Press Send Look at the right-hand side, not the left.Scan for names, company names, project names, and anything in a picture.Zoom in on PDFs and images. Make sure each box fully covers the text.Use Black or White boxes rather than Pixelate for readable text.Send the downloaded masked file, never the original. Questions People Ask Do I need to create an account or pay?No. It is free and needs no sign-up. Does it keep a copy of my file?No. It runs in your browser tab. Close the tab and it is gone. There is no server that receives your file. Can it undo the masking later?No. The masked file contains only the labels, not the original values. Keep your original file somewhere safe if you need it. Will it find everything?No tool can promise that. It is very good at patterns like emails, phones, cards and keys. For names, company names, secret project names and anything inside a picture, you add masks yourself. That is why the review step matters. Does it work on my phone?The page adapts to a phone screen, but it is designed for a computer. Drawing boxes on images and checking PDFs is much easier with a mouse or trackpad. Which files can I use?Pasted text, .txt, .log, .json, .har, .csv, .pdf, .docx, .xlsx (masked as text) and images (JPG, PNG, WebP). See It In Action This short clip shows the Word, PDF, and image flows from start to finish, using a sample screenshot. Demo: a screenshot inside a Word file and a PDF, then the same screenshot as a standalone image. Try It Now Open the Log & HAR Sanitizer: https://mask-online-tool.vercel.app/. The examples in this post use made-up names, numbers, and keys. This tool reduces the risk of sharing private data but cannot guarantee that every item is found. Always review the result before you share it.

By Nagappan Subramanian CORE

We have documents that need full-text search, but those documents also have meaningful relationships. Do we split search and graph traversal across two systems or keep them together? Most developers reach for Elasticsearch or a dedicated search service when they need full-text search. It's a reasonable reaction, but it introduces an extra service to operate, an extra data pipeline to maintain, and an extra hop in every query. What's easy to miss is that Neo4j ships with Apache Lucene built in — the same underlying search library on which Elasticsearch is built — and when our data are already in a graph, adding full-text search on top is a single Cypher command! More importantly, full-text search in Neo4j can do something a standalone search engine can't: it serves as an entry point into the graph. We can find our initial results with Lucene, then traverse relationships to discover what surrounds them. That two-phase pattern — search-then-traverse — is what this article describes. We'll build a research paper explorer using a generated dataset of 200 papers with titles, abstracts, authors, fields, venues, and citation relationships. The same papers appear in the Weaviate chapter of Seven Vector Databases in Seven Days, where we used hybrid vector search to find them. Here we use Neo4j's full-text search instead, and we add something the Weaviate version didn't have: a citation graph that we can traverse after search. The comparison is instructive — same papers, similar queries, but a very different graph layer. The full source code is available on GitHub. Prerequisites AuraDB is Neo4j's fully managed cloud database. A free tier is available with no credit card required. Sign up at Get Started for Free.Create a new AuraDB Free instance.When the instance is created, download or note the credentials — the connection URI, username, and password.Once the instance is running, open the Query tab, and connect to the instance.Confirm it's empty with MATCH (n) RETURN count(n) which should return 0. The code runs in a Jupyter notebook with Python 3.12 or later. Before starting Jupyter, export the connection details as environment variables in your shell: Shell export NEO4J_URI="neo4j+s://xxxx.databases.neo4j.io" export NEO4J_USERNAME="your-username" export NEO4J_PASSWORD="your-password" One version note. This article queries the full-text index with Cypher's SEARCH clause, which replaces the older db.index.fulltext.queryNodes() procedure. The documentation lists SEARCH on full-text indexes as available from Neo4j 2026.09 and it requires Cypher 25. Our AuraDB Free instance accepted it, so the most reliable check is simply to try the clause, which the notebook does. On instances that don't support it, the procedure does the same job but now produces a deprecation warning. The Graph Model The dataset has three node labels and two relationship types: Plain Text (:Author)-[:AUTHORED]->(:Paper)-[:CITES]->(:Paper) Paper nodes carry the properties we'll search and filter on: title, abstract, field, year, venue and citation_count. Author nodes have a single property: name. The AUTHORED and CITES relationships carry no properties — all the meaning is in the graph structure. A conventional search index doesn't give us Neo4j's native graph model and Cypher traversal over relationships. A paper's citation neighborhood, the other work its authors have published, the papers that are frequently cited alongside it — that structure lives in the relationships, and it's what we'll traverse after search. Generating the Dataset We'll generate 200 papers from pools of fields, venues, author names, methods, datasets, and abstract templates, using a fixed random seed so the results are reproducible. The generation code is adapted directly from the Weaviate chapter, so the papers are the same across both articles. Python random.seed(RANDOM_SEED) papers = [generate_paper(i) for i in range(NUM_PAPERS)] Each paper gets between two and five authors, drawn from pools of first and last names. We add two things the Weaviate version didn't have: author nodes and citation edges. Citation edges are generated with three rules that keep the graph plausible: A paper can only cite papers with a lower paper_id and a publication year no later than its own, so nothing cites the futureCitations are biased toward papers in the same or adjacent fields, using a weighted random selectionEach paper makes between three and eight citation attempts, and duplicates are removed Python weights = [ 3 if c["field"] == paper["field"] else 2 if c["field"] in adjacent else 1 for c in candidates ] This gives us 937 citation edges that cluster by field, which makes the traversal results more interesting. Because the relationships are synthetic, the traversal results demonstrate the mechanics of graph-based discovery rather than real bibliometric relationships. Loading Into Neo4j Before loading, the notebook clears any existing data — dropping constraints and indexes first, then deleting nodes in batches of 1,000 to avoid transaction timeouts on Aura. It then creates uniqueness constraints on Paper.paper_id and Author.name, which keep authors unique and make the lookups in the loading queries fast: Cypher CREATE CONSTRAINT paper_id IF NOT EXISTS FOR (p:Paper) REQUIRE p.paper_id IS UNIQUE; CREATE CONSTRAINT author_name IF NOT EXISTS FOR (a:Author) REQUIRE a.name IS UNIQUE; We'll load the data in three passes using UNWIND to batch the inserts. First, the papers: Cypher UNWIND $records AS r CREATE (p:Paper { paper_id: r.paper_id, title: r.title, year: r.year, field: r.field, venue: r.venue, citation_count: r.citation_count, abstract: r.abstract }) For authors, we use MERGE rather than CREATE, since the same author can appear on multiple papers and we want a single node per author: Cypher UNWIND $records AS r MERGE (a:Author {name: r.author_name}) WITH a, r MATCH (p:Paper {paper_id: r.paper_id}) CREATE (a)-[:AUTHORED]->(p) Finally, the citations: Cypher UNWIND $citations AS c MATCH (from:Paper {paper_id: c.from_id}) MATCH (to:Paper {paper_id: c.to_id}) CREATE (from)-[:CITES]->(to) After loading, the graph has 200 Paper nodes, 215 unique Author nodes, 691 AUTHORED relationships and 937 CITES relationships. One caveat about the authors: names are drawn from 15 first names and 15 last names, so there are only 225 possible combinations. "The same author" here just means "the same generated name". Real bibliographic data needs author disambiguation, which is a hard problem in its own right. Creating the Full-Text Index We'll create a single Lucene index across both the title and abstract properties of Paper nodes: Cypher CREATE FULLTEXT INDEX paper_fulltext IF NOT EXISTS FOR (p:Paper) ON EACH [p.title, p.abstract] OPTIONS { indexConfig: { `fulltext.analyzer`: 'standard' } } A single index across multiple properties means a single query scores both together — no merging of separate result sets in application code. The OPTIONS line chooses the analyzer, the component that splits text into searchable terms. By default Neo4j uses standard-no-stop-words, which indexes every word, including "and", "for," and "the." That makes long natural-language queries noisy, because a paper can pick up a score just by containing "and." The standard analyzer drops common English stop words but doesn't stem, so fuzzy matching, which we'll use shortly, still works on the original terms. The index is created asynchronously, so we'll poll until it's ONLINE before querying: Cypher SHOW INDEXES YIELD name, state WHERE name = 'paper_fulltext' RETURN state Basic Full-Text Search We'll query the index with the SEARCH clause, which returns matching nodes alongside a Lucene BM25 relevance score: Cypher CYPHER 25 MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC Two details are worth knowing. First, LIMIT is part of the clause: it sets how many top-scoring matches the index returns. Second, the CYPHER 25 prefix selects the Cypher version that provides SEARCH. We show it once here and omit it from the snippets that follow, although the notebook includes it in every query. The notebook builds this text with a small helper that fills in the limit, so every example below reuses the same clause. The score is based on BM25, which considers term frequency, inverse document frequency, and document length. A paper whose title and abstract both mention "transformer models" will score higher than one where the term appears only once in the abstract. Python search_papers("transformer models for natural language understanding") Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Symposium on Knowledge Discovery and Data Mining | 2022 | Citations: 18 Score: 9.926 Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 9.741 ... The generated titles aren't unique, so the same title can appear twice. They are different papers, which the year and venue make clear. Special Characters in User Input Lucene treats characters such as + - ! ( ) { } [ ] ^ " ~ * ? : \ / as query syntax. If the query text comes from a user, an unbalanced bracket or a stray colon will raise a parse error or quietly change the meaning of the query. Unless we deliberately want operators like phrases, ~ and boosts to work, we should escape the text first: Python def escape_lucene(text: str) -> str: """Escape Lucene query-syntax characters in raw user input.""" return re.sub(r'([+\-!(){}\[\]^"~*?:\\/&|])', r'\\\1', text) search_papers("stream processing (survey") # raises a parse error search_papers(escape_lucene("stream processing (survey")) # works The unescaped query fails with a message that Neo4j could not parse the full-text search string. The escaped one returns the stream processing papers. Boosting Titles Over Abstracts Because the index covers both properties, we can address them by name in the query and weight them. Here a match in the title counts three times as much as a match in the abstract: Python search_papers("title:transformer^3 OR abstract:transformer", top_k=5) The Index Stays in Sync One of the costs of a separate search service is the pipeline that keeps it in sync with the primary database. Here the full-text index is updated as part of the same transaction that writes the node. To see it, we create a paper containing a unique word, search for it straight away, and then delete it: Python with driver.session() as session: session.run(""" CREATE (:Paper {paper_id: 9999, title: 'Quokka Navigation for Robotics', abstract: 'A throwaway paper used to show the index updates on commit.', year: 2024, field: 'Robotics', venue: 'Demo', citation_count: 0}) """) search_papers("quokka", top_k=1) # found immediately with driver.session() as session: session.run("MATCH (p:Paper {paper_id: 9999}) DETACH DELETE p") search_papers("quokka", top_k=1) # gone again The first search returns the new paper (score 2.815) and the second returns nothing. Phrase Search Wrapping a query in double quotes tells Lucene to treat it as a phrase — the words must appear together in that order: Python search_papers('"graph neural networks"', top_k=3) Without quotes, Lucene scores documents that contain any of the words, awarding higher scores for more matches. With quotes, every result must contain the exact phrase. Score Comparison: Keyword vs. Phrase Search The chart below (Figure 1) runs the same query twice, once as keywords and once as a phrase, and plots the scores of the top eight results of each. A bar of zero means the paper is outside that mode's top eight, not that it fails to match. Figure 1. Keyword vs. Phrase Search. The result is worth reading carefully. In this dataset, every paper in the Graph Neural Networks field contains the exact phrase, because the field name appears in its title and abstract. So both queries match essentially the same papers and their scores are close: 6.1 to 6.8 for keywords and 5.4 to 5.65 for the phrase. What changes is which papers make the top eight. Only three of the eight appear in both lists, and the top result differs: keyword search ranks "Efficient Graph Attention Networks for Graph Neural Networks" first (6.774), while phrase search ranks "Efficient Link Prediction for Graph Neural Networks" first (5.649). A plausible reason is that keyword scoring rewards papers that repeat the individual words, such as "graph" and "networks" in a title that also mentions graph attention networks, while phrase scoring only counts the phrase itself. Raw scores aren't directly comparable across the two query forms, so membership and order are the things to compare. On noisier text, where the words in a query don't usually sit together, the phrase search would also drop papers entirely, which is the more familiar reason to use it. Fuzzy Search Appending ~ to a term enables Lucene's fuzzy matching, which finds terms within a given edit distance. A misspelled query like tranformer~ correctly matches transformer: Python search_papers("tranformer~", top_k=3) Example output: Plain Text Efficient Transformer Models for Natural Language Processing Natural Language Processing | Transactions on Machine Learning and Data Mining | 2020 | Citations: 196 Score: 3.015 ... Without the ~, the misspelled query returns nothing. With it, Lucene finds the closest matching terms and scores them. This is useful for any field where exact spelling can't be guaranteed, such as technical terms or partially remembered titles. Filtered Search We can combine the full-text index with filters on structured node properties. There is one thing to be aware of: for full-text indexes, SEARCH doesn't allow a WHERE inside the clause. Filtering is a second step. The index first returns its top-scoring candidates, then a WHERE after it filters them by field, year or citation count: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 200 ) SCORE AS score WITH p, score WHERE p.field = $field AND p.year >= $min_year AND p.citation_count >= $min_citations RETURN p.title, p.field, p.venue, p.year, p.citation_count, score ORDER BY score DESC LIMIT $top_k The two limits do different jobs. The LIMIT 200 inside SEARCH sets the size of the candidate pool, here every paper in the collection, and the final LIMIT $top_k trims the filtered results. If we asked the index for only the top five and filtered afterward, a selective filter could leave us with nothing. On a large collection, we would choose a pool big enough for our filters to leave enough results, without asking the index to score everything. These are the same three filtered queries used in the Weaviate chapter. Weaviate ran them as hybrid searches, combining vectors and keywords, whereas this is keyword-only BM25, so we should expect overlap in the results rather than identical rankings: Python # NLP papers from 2020 onwards with at least 100 citations search_papers_filtered( "attention mechanisms and transformer architectures", field = "Natural Language Processing", min_year = 2020, min_citations = 100, ) # Recent computer vision papers search_papers_filtered( "object detection in real time", field = "Computer Vision", min_year = 2022, ) # Highly cited papers across all fields search_papers_filtered( "self-supervised learning for representation learning", min_citations = 200, ) The first query returns two papers (scores 3.350 and 3.154) and the second returns one (7.250). The third returns five: the three self-supervised learning papers score between 10.273 and 9.537, then the scores drop to 3.543 for the fourth paper, which matches only more generic terms such as "learning". With the default analyzer, an earlier run of the first query also returned papers scoring below 0.1, apparently matching only on common words such as "and". With stop words removed, that tail no longer appears. Search Then Traverse: The Graph Layer Full-text search answers "Which documents match?" Graph traversal answers "What is connected to those matches?" This section is where the two work together. We'll use the Lucene results as an entry point into the graph, then traverse CITES and AUTHORED relationships to discover related content. The pattern is always the same: SEARCH finds the initial set, then Cypher traverses from there. In a synthetic graph, some papers have few connections, so the notebook runs ten candidate queries and measures each top result's neighborhood: how many papers it cites, how many cite it, and how strongly it is co-cited with others. It picks the richest as the demonstration query. That query is "protein structure prediction", whose top result, "Efficient Protein Structure Prediction for Bioinformatics" (score 12.926), cites 3 papers and is cited by 17. What Does a Top Result Cite? Starting from the highest-scoring paper, we'll follow outgoing CITES edges to see what it builds on. Note that citation_count is a property generated independently for each paper — it isn't derived from the CITES relationships in this graph. We use it here as a proxy for importance when ordering results. Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(cited:Paper) RETURN p.title AS source_title, score AS source_score, cited.title, cited.field, cited.year, cited.citation_count ORDER BY cited.citation_count DESC LIMIT $top_k Example output: Plain Text Drug Discovery for Bioinformatics: A Scalable Approach Bioinformatics | 2016 | Citations: 483 Improving Bioinformatics with Drug Discovery Bioinformatics | 2016 | Citations: 467 Cryptographic Protocols in Cybersecurity: Challenges and Opportunities Cybersecurity | 2016 | Citations: 283 What Cites a Top Result? Traversing in the opposite direction — incoming CITES edges — finds papers that build on a highly relevant result: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) RETURN p.title AS source_title, score AS source_score, citing.title, citing.field, citing.year, citing.citation_count ORDER BY citing.year DESC LIMIT $top_k The most recent citing papers include "A Unified Framework for Reinforcement Learning using Hierarchical Rl" (2024), "Efficient Malware Classification for Cybersecurity" (2024), and "Meta-Learning for Machine Learning: A Scalable Approach" (2024). Citations reach across fields because the generator only biases them toward related fields; it doesn't confine them. Visualizing the Citation Network We can visualize the citation neighborhood of the top result as an interactive graph with neo4j-viz. In Figure 2, the central red node is the top-scoring paper, blue nodes are papers it cites, and green nodes are papers that cite it. Arrows point from the citing paper to the cited paper, and node size reflects citation count. In the notebook, hovering over a node shows the full title, year, and citation count. Python # One Cypher query returns the top result with the papers it cites and the papers that cite it nodes, relationships = {}, [] nodes[top["id"]] = Node(id=top["id"], caption=short(top["title"]), properties={"role": "Top result", "citations": top["citations"]}) for p in top["cited"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cites", "citations": p["citations"]}) relationships.append(Relationship(source=top["id"], target=p["id"])) for p in top["citing"]: nodes[p["id"]] = Node(id=p["id"], caption=short(p["title"]), properties={"role": "Cited by", "citations": p["citations"]}) relationships.append(Relationship(source=p["id"], target=top["id"])) VG = VisualizationGraph(nodes=list(nodes.values()), relationships=relationships) VG.color_nodes(property="role", colors={ "Top result": "#EF553B", "Cites": "#636EFA", "Cited by": "#00CC96"}) VG.resize_nodes(property="citations", node_radius_min_max=(14, 40)) VG.render(height="520px") Figure 2. Citation Network. This kind of neighborhood visualization isn't something we'd get from a search engine, as it requires the graph. Find Other Work by the Same Authors From the top result, we'll traverse AUTHORED edges to the authors, then follow their other AUTHORED edges to find their other papers: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (a:Author)-[:AUTHORED]->(p) MATCH (a)-[:AUTHORED]->(other:Paper) WHERE other <> p RETURN p.title AS source_title, score AS source_score, a.name AS author, other.title, other.field, other.year, other.citation_count ORDER BY other.citation_count DESC LIMIT $top_k The highest-cited results are papers by Chen Singh and Lena Li, two of the authors of the top result, including "Improving Computer Vision with Generative Adversarial Networks" (475 citations) and "Efficient Multi-Agent Systems for Reinforcement Learning" (433). This is the kind of discovery path that's natural in a graph but would typically require additional application-side queries or data modeling outside the search index. Co-Citation Discovery Two papers are co-cited when a third paper cites both of them. Frequently co-cited papers can indicate related research topics, shared foundations, or intellectual proximity — a relationship that doesn't appear in the papers themselves, only in how others use them: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 1 ) SCORE AS score WITH p, score MATCH (citing:Paper)-[:CITES]->(p) MATCH (citing)-[:CITES]->(cocited:Paper) WHERE cocited <> p WITH p, score, cocited, count(citing) AS shared_citations RETURN p.title AS source_title, score AS source_score, cocited.title, cocited.field, cocited.year, shared_citations ORDER BY shared_citations DESC LIMIT $top_k For our top result, the most co-cited paper is "Drug Discovery for Bioinformatics: A Scalable Approach" with 6 shared citing papers, followed by "Rethinking Robotics with Imitation Learning" with 5, and three papers with 4 each. The Robotics paper is a reminder that the generator's field bias is only a bias. Co-citation analysis is a classical technique in bibliometrics, and here it is a single Cypher query. Traversing From More Than One Result So far we've started from the single best hit. The same pattern works from the top N results, and aggregation then surfaces something no single result shows: the papers that several of the best matches have in common. Here we ask which papers are cited by the most of the top five results, a rough proxy for foundational work on the topic: Cypher MATCH (p) SEARCH p IN ( FULLTEXT INDEX paper_fulltext FOR $search_query LIMIT 5 ) SCORE AS score WITH p, score MATCH (p)-[:CITES]->(ref:Paper) RETURN ref.title AS title, ref.field AS field, ref.year AS year, ref.citation_count AS citations, count(DISTINCT p) AS cited_by_hits ORDER BY cited_by_hits DESC, citations DESC LIMIT $top_k Two papers, "Drug Discovery for Bioinformatics: A Scalable Approach" and "Improving Bioinformatics with Drug Discovery," are each cited by two of the five best matches, ahead of papers cited by only one. A standalone search engine would return the five matches and stop there. When to Use This Neo4j's full-text search is worth reaching for when our data are already in a graph, and we want to add keyword search without operating a separate service. The setup is minimal: one Cypher command creates the index, one clause queries it, and we immediately get phrase matching, fuzzy matching, field boosting, and BM25 relevance scoring. Because the index is updated in the same transaction as the data, there is no synchronization pipeline to build or monitor. The search-then-traverse pattern is the real differentiator. If our data have relationships, such as citations, authorship, product categories, or social connections, and we want search to serve as an entry point into those relationships, Neo4j gives us both in one place. It is not the right choice for every workload. A dedicated search engine offers capabilities that weren't part of what we used here, such as faceted aggregations and result highlighting, and it lets us scale search independently of the database. If search traffic is large, if it needs different hardware from the graph workload, or if the data have no meaningful relationships, a separate service may be the better fit. The SEARCH behavior described above also matters at scale: the index returns a bounded candidate set, and filters apply afterward, so queries with very selective filters need a candidate pool sized for them. For purely semantic workloads where we want vector similarity rather than keyword matching, Neo4j also supports vector indexes, which we can combine with full-text search in the same query. But for the common case where keyword search and graph traversal belong together, the built-in Lucene integration is often all we need. Summary We've seen how to create a multi-property full-text index in Neo4j, query it with the SEARCH clause using keyword, phrase, and fuzzy search, escape user input and boost fields, combine the results with structured property filters, and use them as the entry point for graph traversals that follow citations, authorship, co-citation, and multiple matches at once. Further examples of Neo4j's multi-model capabilities can be found in the free online book Neo4j Beyond the Graph. The full source code is available on GitHub.

By Akmal Chaudhri CORE

Have you ever asked an LLM for JSON and got back "Sure! Here is your JSON:" followed by the JSON? Your parser fails, your pipeline stops, and someone adds a retry loop. I have been there. For a long time, the fix was a polite prompt. "Reply with valid JSON only. No extra words." It works most days. On the other days, it breaks. There is a better fix, and it does not rely on the model behaving well. It is called constrained decoding. In this article, you will learn: What encoders and decoders are, and when to use eachHow constrained decoding works, in plain wordsWhich tools do it todayWhat research says about its strengths and limitsHow it compares with a plain LLM, and with new decision models like Jev and LayaHow to use it safely inside an agent A quick note on the name. In this article, CDM means constrained decoding model: a decoder LLM run together with a constraint engine. CDM is the label I use for this setup. It is not a product name. Encoder and Decoder in Plain Words Almost every modern language model is built from blocks called transformers. What changes is how the blocks are arranged. A survey of language models for software work describes three layouts: BERT as encoder-only, LLaMA as decoder-only, and CodeT5+ as encoder plus decoder (arXiv 2408.02479). Encoder and Decoder Here is the easy way to remember it. An encoder is a reader. It looks at the whole message at once, and every word can see the words on both sides. The output is understanding, a set of numbers that capture meaning. That is why encoders are good at jobs like classifying, routing, searching, and filtering. Researchers who built small safety checkers note that encoder models are efficient at narrow jobs such as classification, name tagging, and sentiment (arXiv 2410.08442). A decoder is a writer. It produces one small piece of text, called a token, at a time. Each new piece depends on what it already wrote. Chat models like Llama work this way. Decoders became popular because they are flexible: almost any task can be turned into "write the answer." An encoder plus decoder reads one thing and writes another, like a translation or a summary. T5 and CodeT5+ are examples. If you need toUseWhyLabel, filter or route textEncoder, or a decision model built on oneReads once, small, cheapSearch by meaningEncoderProduces meaning numbersWrite, explain, plan or codeDecoder (LLM)Generates new textMake a decoder answer in a fixed shapeDecoder plus constrained decodingKeeps the writer inside the rulesExact lookups, math or datesPlain codeFast and easy to check A CDM sits in the fourth row. Jev and Laya are closer to the encoder idea. Laya, for example, is built on a ModernBERT encoder with a decision layer on top (Laya model card). What Is Constrained Decoding? A decoder picks its next token by scoring every token it knows. Constrained decoding adds a step. Before the model picks, a rule engine blocks the tokens that would break your rules. The XGrammar paper describes this as restricting the available tokens at each step (arXiv 2411.15100). How CDM works The loop has five steps: Score: the model scores every possible next token.Check: the rule engine works out which tokens are legal at this spot.Mask: illegal tokens are blocked.Pick: the model chooses from the legal tokens only.Repeat: the token is added, and the loop runs again until the answer is complete. The values in the close-up part of the picture are made up to show the idea. What matters is the shape of the process: blocked tokens disappear, and the remaining choices share the probability. What Kinds of Rules Can You Use? The common ones are a JSON schema, a regular expression, a fixed list of choices, or a grammar. vLLM, a popular serving engine, lists all four (vLLM docs). The Outlines library shows how simple this can be. For a yes-or-no answer, you use a Literal["Yes", "No"] type. For a sentiment label, you use Literal["Positive", "Negative", "Neutral"] (Outlines on GitHub). The model can no longer return anything else. That means a choice constraint turns a general LLM into a classifier that can never invent a label. One Thing to Remember Masking controls what the model is allowed to write. It does not change what the model understands. Everything in the research section below comes back to this point. Tools You Can Use Today The tools fall into three groups: engines that compute the mask, servers that run them, and hosted APIs that hide them. ToolWhat it isStart hereOutlinesA Python library that turns types, schemas or regex into a state machine that masks tokensPaper, GitHubXGrammarAn engine that prepares most of the mask ahead of time to keep overhead low. XGrammar-2 targets rules that change at runtimePaper, XGrammar-2llguidance (Guidance)A fast engine written in Rust that handles full grammars and JSON schemasGitHubllama.cpp grammarsGrammar support for local models, using GBNFGrammar guidevLLMA serving engine that uses XGrammar or Guidance as backendsDocsNVIDIA NIMContainerized serving with JSON schema, regex, grammar, and choice supportDocsOpenAI Structured OutputsA hosted option that enforces a JSON schema in strict modeAnnouncementClaude structured outputsA hosted option that compiles your JSON schema into a grammarDocs Two things to watch for. Hosted APIs support a part of JSON Schema, not all of it. For example, Google's documentation for Claude on Vertex AI says some constraints, such as numeric and string-length rules, are not supported (Vertex AI docs). If a rule is not enforced by the mask, check it in your own code. Your backend can change under you. vLLM's configuration notes say the automatic backend choice can change between releases (vLLM config). If you need repeatable behavior, set the backend yourself and pin the version. What the Research Says I read the main papers so you do not have to. Several 2026 papers are preprints, which means they have not been peer-reviewed yet. Here are the five points that matter most. 1. A valid format becomes a guarantee. A 2026 preprint on small open models found that constrained decoding removed format failures such as wrong types and missing fields (arXiv 2609.23742). If a program must parse the output, this is the main reason to use it. 2. Valid is not the same as right. The same paper found that the content can still be wrong or thin. A model that skips a required step still passes the schema if the schema allows it. The authors describe rescued answers that look complete but carry little meaning. The short version: constraints fix form, not understanding. 3. Real schemas are harder than demos. JSONSchemaBench tested several engines, including Guidance, Outlines, llama.cpp, XGrammar, OpenAI, and Gemini, on a large set of real-world schemas. Engines differ in which schema features they support (arXiv 2501.10868). That benchmark is from early in this wave of tools, and engines have improved, so test your own schemas on the engine you plan to use. 4. Strict formats can hurt reasoning. The paper "Let Me Speak Freely?" found that format restrictions can weaken reasoning, and stricter formats can do more harm. It also found that constraints can help classification (arXiv 2408.02442). In reasoning tasks, JSON mode often did not keep the order of "think first, answer second." So if the model must think, let it think first. 5. Masking can nudge what the model prefers. Research on grammar-aligned decoding shows that blocking tokens can distort the model's own probabilities, so some outputs are valid but lower in quality (arXiv 2405.21047). A 2026 preprint on draft-conditioned decoding targets a related problem, where masking pushes the model down a path that is valid but wrong (arXiv 2603.03305). FindingWhat it meansWhat to doFormat failures disappearOutput always fits the rulesUse it wherever a program parses the outputValid can still be wrongMeaning is not checkedTest the content, not only the formatEngines support different schema features"Supports JSON schema" is not one yes or noTest your schemas on your engineStrict formats can hurt reasoningEarly constraints take away thinking roomLet the model think, then constrain the final answerMasking can bend probabilitiesOpen-ended content can lose qualityWatch quality on long or creative output CDM vs. LLM vs. Jev and Laya Now we can compare the designs. All three can give you a valid answer from a fixed list. They get there in different ways. LLMCDMDecision modelCore designDecoderDecoder plus maskEncoder plus decision layerHow it answersWrites token by tokenWrites token by token, legal tokens onlyScores every allowed answer in one passOutput always validNoYes, inside the rules the engine supportsYes, it picks from your listProbabilitiesNot by defaultToken level, not trained to be calibratedTrained to be calibratedCan write or explainYesYes, inside the shapeNoOpen shapes like SQL or nested JSONYesYesNo, typed answers onlyMain riskBroken formatValid but wrong, reasoning lossConfident wrong picks, forced answers What Are Jev and Laya? They are decision models, sometimes called System One models, after the fast, gut-level mode of thinking. You give them an input, a question, and the allowed answers. They return probabilities for each answer in one pass. They do not write text. Jev comes from TypeSafe AI and is offered as a hosted service. Its evaluation paper describes typed answers: a choice, a position on a rubric, or a true-or-false probability (arXiv 2609.37647). Laya comes from Convai Innovations and is open, so you can host and fine-tune it yourself. Its model card explains that it is trained so that its probabilities are honest, a method it calls RLCD (Laya model card). I covered both in Classification Never Left. It Just Got a New Home in LLMs. How Do They Differ From a CDM? Think of a quiz with four answers. A CDM is a writer who is only allowed to write one of the four answers. It still goes through the writing loop, and it can also handle open-ended formats like SQL. A decision model reads the question once and hands you a score for each of the four answers. It cannot write anything else, and it is built for calibrated scores. ScaleDown's CDMs, covered in the next section, are a third variant. They are small models trained for one task, so the limit on what they can return is built in during training instead of added by a mask at run time. What About a Head-to-Head Test? I looked for a direct comparison of a constrained LLM against Jev or Laya on the same data and did not find one. So I will not name a winner. If you are choosing between them, test both on a few hundred of your own inputs. One weakness is shared. A closed list forces an answer. If a message fits none of your categories, both a decision model and a constrained LLM will still pick one. This follows from how the list works, and it is why you should always add an "other" option and test it on purpose. When to Use What When to use what - LLM vs CDM vs Decision Model Walk the questions from left to right and stop at the first yes. JobBest fitWhyTool call arguments for an agentCDMStrict JSON that must parseSQL, regex or a small languageCDM with a grammarOpen shape with clear rulesPulling fields out of text into a schemaCDMType and structure errors disappearTicket routing at high volumeDecision model, or a trained encoder classifierOne pass, cheap, probabilitiesUrgency score or yes or no flagDecision modelTyped scoresReasoning, then a structured answerLLM thinks freely, then a constrained last stepAvoids the reasoning lossSummaries, replies and explanationsLLMYou need wordsExact rules, lookups, math and datesPlain codeFast and easy to check Two rules of thumb: If the model must think, let it think first. Give it a free reasoning step, then constrain only the final answer. If you must use one schema, put the reasoning field before the answer field.If the answer is a pick, constrain it tightly. Picks are where constraints help most. How ScaleDown Brought CDMs Into Agents A recent post from LangChain shows where this idea is heading. Long-running agents spend a lot of effort on context management: summarizing old history, shrinking large tool results, and pulling the right values out of long inputs. Each of those is a small decision, and making them with a large LLM on every turn is slow and costly. ScaleDown's answer is a set of CDMs. Instead of writing any string one token at a time, each model is trained for one job and only returns answers that job allows, such as a label from your list or pieces of the input text. ScaleDown offers models to summarize, compress, extract, and classify, and the langchain-scaledown package wraps them as middleware for LangChain and Deep Agents. The big LLM keeps the planning and writing, and the CDMs handle the high-volume steps around it (LangChain blog). Its classify model will feel familiar if you have read about Jev. Both return a typed answer with a probability, and the post says ScaleDown also adds a short reason, which helps when you debug why an agent stopped or routed a task. The post also notes that ScaleDown is now available through the LangSmith LLM Gateway, so its calls are traced like your other model calls. One caution. This is a vendor-written post, and the speed and cost claims come from ScaleDown's own benchmarks, so test them on your own workload before you rely on them. Using a CDM Inside an Agent The two designs work well together. A CDM keeps the agent's tool call valid. A decision model, or a second choice constraint, decides whether that call is safe to run. CDM inside an agent Here is how it flows: A user or system sends a request.The LLM agent plans the next step and proposes a tool call.Constrained decoding makes sure the call matches the schema, so it always parses.A decision gate scores the call as allow, review, or block.Allowed calls run, unsure calls go to a person, and risky calls are rejected and logged. Three habits keep this honest: Test two things. Check that the output is valid, and check that it is right. They are not the same.Log every step. Keep the proposed call, the schema it matched, the gate's scores, and who approved it.Treat the gate as one control. Give the agent only the access it needs, and require a human for high-impact actions. I wrote more about these controls in AI-Era Cybersecurity, Part 2, and about the tools that sit beside them in Part 1. A Simple Checklist StepAction1List every place your system needs a fixed shape or a pick from a list2Move anything plain code can decide out of the model3Choose a rule type: choice, regex, JSON schema or grammar4Stay inside what your engine supports, and check the rest in code5Add an "other" or "unknown" option to every closed list6Put reasoning before the answer, or reason in a separate step7Pin the engine, backend and model version8Test format and meaning on real inputs, including ones that fit no category9Cache compiled schemas, or use an engine built for rules that change per request10Log decisions and recheck on a schedule Limits of This Article Some of the newest evidence is in preprints and small studies. Treat it as a signal, not a final answer.Hosted APIs do not publish their full mask engines, so I rely on their documentation.I found no same-data comparison between a constrained LLM and a decision model, so the comparison here rests on design differences and separate sources.Tools in this area change quickly. Check each official page before you decide. Related Articles From Me ArticleHow it connectsClassification Never Left. It Just Got a New Home in LLMsThe decision model side of this comparisonShipping Production-Grade AI Agents (Refcard #403)Guardrails and evaluation, where a CDM fitsA Field Guide to AI Agent FrameworksWhere structured output plugs into frameworksGraph Engineering: The Layer After Loop EngineeringA constrained choice can pick the next node in a graphAI-Era Cybersecurity, Part 2Agent identity, approval gates and measuring riskAI-Era Cybersecurity, Part 1The bug-fixing tools these gates sit besideHow AI Is Actually Changing SRE Tools, Part 2Alert triage is a pick-from-a-list problemFrom Prompts to Harnesses: How AI Engineering Has Grown UpThe harness around the model is where constraints live Conclusion A CDM is a decoder with guardrails. It gives you a promise that is easy to state and easy to test: the output will always fit the rules. It does not promise that the answer is right, and strict rules can cost you reasoning if you apply them too early. Decision models like Jev and Laya make a different trade. They give up writing in return for fast, one-pass scoring with probabilities built to be honest. They have their own risks, such as confident wrong picks and forced answers. So match the tool to the shape of the job. Constrain tightly when the output must parse, or the answer is a pick. Leave the model free when it needs to think. And test both the format and the meaning on your own data before you trust either one. If you try this in your own system, I would like to hear which part you tried first. Share it in the comments. References Papers Efficient Guided Generation for Large Language Models (Outlines)XGrammar: Flexible and Efficient Structured Generation EngineXGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMsJSONSchemaBenchLet Me Speak Freely? Format Restrictions and LLM PerformanceGrammar-Aligned DecodingThe Hidden Cost of Structured Generation: Draft-Conditioned Constrained DecodingConstrained Decoding Eliminates Structural Failures in Small LLMsFrom LLMs to LLM-based Agents for Software Engineering: A SurveyJurEE not Judges: Small, Specialized Encoder EnsemblesEvaluating and Benchmarking the System One Model Jev Tools and Documentation OutlinesXGrammarllguidancellama.cpp grammarsvLLM structured outputsNVIDIA NIM structured generationOpenAI Structured Outputs announcementClaude structured outputsClaude structured outputs on Vertex AI ScaleDown and LangChain Building Long-Running Agents with ScaleDown and Deep Agents, LangChain blogScaleDown Decision Models Laya model card, Hugging FaceJev: Decision Models on Trial, Cisco Blogs

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE