Microservices Resources

DZone's Featured Microservices Resources

Designing Scalable Containerized Backend Services

By Estefanio Fernando

Modern enterprise software design has fundamentally shifted away from monolithic, single-threaded runtimes toward decoupled, containerized architectures. When building systems that handle high throughput — such as fintech services, automated reporting pipelines, or real-time distributed platforms — engineers must address two core infrastructure vectors: high-concurrency connection management and deterministic relational state execution. A common anti-pattern in backend systems engineering is assuming that containerization automatically scales an application. In reality, wrapping a poorly optimized, blocking database service inside a Docker container simply shifts the performance bottleneck from local computing hardware to network sockets and thread pools. This technical guide breaks down the implementation of a decoupled, high-concurrency microservice engine built on an asynchronous Python core, using localized relational persistence layer mechanics, automated connection pooling, and multi-tier containerized deployment. The Concurrency Bottleneck in State Management In traditional synchronous web gateways, each incoming network transaction is mapped directly to an operating system (OS) thread. If an application needs to process a complex relational query, generate a structural file, or compute algorithmic abstractions, that thread remains blocked until the underlying process returns a code sequence. Under heavy enterprise concurrency patterns, thread starvation occurs. The OS spends more execution cycles executing thread context switches than processing actual transaction payloads. Textile [Blocking Monolith] ───► Thread 1 ───► [DB Read / Long IO Block] ───► (Thread Starvation) [Asynchronous ASGI] ───► Async Worker ───► Loop Delegation ───► (Thread Free to Serve Next Connection) To achieve true horizontal scaling within a decoupled microservices architecture, we must exploit non-blocking Asynchronous Server Gateway Interface (ASGI) frameworks combined with explicit, non-blocking asynchronous drivers for relational storage layers. This structural change ensures that database connection limits do not become a systemic point of failure. Architectural Blueprint: The Asynchronous Core Engine The following implementation showcases an advanced, high-performance transactional microservice core engine. The software structure uses asynchronous runtime loops to manage localized transactional persistence dynamically without blocking the main worker loop. Python import asyncio import logging import uuid from typing import AsyncGenerator from pydantic import BaseModel, Field from contextlib import asynccontextmanager # Setup structured system tracking logs logging.basicConfig(level=logging.INFO, format="[%(asctime)s] %(levelname)s: %(message)s") logger = logging.getLogger("MicroserviceCore") # Define strict, type-validated data transmission contracts class FinancialPayload(BaseModel): transaction_id: str = Field(default_factory=lambda: str(uuid.uuid4())) account_source: str amount_tokens: float = Field(gt=0.0) system_metadata: dict = Field(default_factory=dict) class ServiceInfrastructureEngine: """ Manages isolated transactional operations, simulating high-performance relational database pool connections under non-blocking event loops. """ def __init__(self, database_connection_string: str, max_pool_size: int = 20): self.connection_string = database_connection_string self.max_pool_size = max_pool_size self._execution_semaphore = asyncio.Semaphore(max_pool_size) async def initialize_state_store(self) -> None: """Simulates automated schema migration and table validation.""" logger.info(f"Connecting to data storage system: {self.connection_string}") await asyncio.sleep(0.5) # Simulate non-blocking I/O connection handshake logger.info("Relational schemas and operational metadata verified successfully.") @asynccontextmanager async def acquire_pooled_session(self) -> AsyncGenerator[str, None]: """Context manager enforcing connection pooling constraints asynchronously.""" await self._execution_semaphore.acquire() session_id = f"DB_SESSION_{uuid.uuid4().hex[:8].upper()}" try: yield session_id finally: self._execution_semaphore.release() async def execute_isolated_transaction(self, payload: FinancialPayload) -> bool: """Executes a strict transaction block simulating multi-stage persistence loops.""" async with self.acquire_pooled_session() as session: logger.info(f"[{session}] Opening isolated database transaction context for ID: {payload.transaction_id}") # Simulate a continuous analytical lookup pattern or sub-query sequence await asyncio.sleep(0.2) # Verify pseudo ledger balances if payload.amount_tokens > 50000.0: logger.warning(f"[{session}] Risk flags raised. Transaction {payload.transaction_id} requires multi-signature validation.") return False await asyncio.sleep(0.1) # Simulate final commit state save logger.info(f"[{session}] Transaction successfully written and finalized in relational engine.") return True # Entry point simulation for a highly distributed parallel transaction load async def main(): # Instantiating our microservice engine layer infra_engine = ServiceInfrastructureEngine(database_connection_string="sqlite+aiosqlite:///production_ledger.db") await infra_engine.initialize_state_store() # Generate an explicit array of simultaneous simulated client event payloads mock_requests = [ FinancialPayload(account_source=f"ACC_USR_{i:03d}", amount_tokens=150.0 * i) for i in range(1, 15) ] # Map requests into non-blocking concurrent tasks execution_tasks = [infra_engine.execute_isolated_transaction(req) for req in mock_requests] # Execute all database loops concurrently across the async thread gateway results = await asyncio.gather(*execution_tasks) logger.info(f"Engine batch processing sequence finished. Successful operations: {results.count(True)}/{len(results)}") if __name__ == "__main__": asyncio.run(main()) Containerization Strategy: Engineering Production Docker Environments To guarantee predictable horizontal execution layers across hybrid cloud network topologies, we deploy the engine utilizing highly optimized multi-stage containerization blueprints. A common operational failure is building bulky production images that include build tools (compilers, development headers, package managers) in the final image. This significantly expands the system's attack surface and increases cold-start image download latency over orchestrated node servers. Below is the optimized, multi-stage Dockerfile blueprint designed to ensure image portability and minimal security overhead: Dockerfile # Stage 1: The Build/Compilation Environment FROM python:3.11-slim AS build-compiler WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ libpq-dev \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt . # Compile and dump dependencies natively into a local virtual deployment tree RUN python -m venv /opt/venv ENV PATH="/opt/venv/bin:$PATH" RUN pip install --no-cache-dir -r requirements.txt # Stage 2: The Final Operational Secure Runtime FROM python:3.11-slim AS runtime-layer WORKDIR /app # Install localized system runtime libraries required by standard database interfaces RUN apt-get update && apt-get install -y --no-install-recommends \ libpq5 \ && rm -rf /var/lib/apt/lists/* # Pull exclusively compiled executable components from the isolated build stack COPY --from=build-compiler /opt/venv /opt/venv COPY . . # Set immutable system paths and configuration markers ENV PATH="/opt/venv/bin:$PATH" ENV PYTHONUNBUFFERED=1 ENV ENVIRONMENT=production EXPOSE 8000 # Enforce secure operational constraints by bypassing root execution privileges USER 1001 CMD ["python", "main.py"] Key Takeaways for Production Systems Explicit resource bound manipulation: As seen in the asynchronous script implementation, utilizing asyncio.Semaphore allows developers to hard-code a strict ceiling over concurrent processing paths, mitigating database exhaustion errors.Deterministic data contracts: Leveraging schema wrappers (like Pydantic or native object-relational models) ensures invalid or corrupted objects fail at the interface boundary before hitting core internal relational storage pipelines.Multi-stage build standards: Decoupling the development environment from the final execution layer results in production-ready, low-footprint containers, making the backend infinitely more resilient and ready for automated container management layers. Building enterprise-grade web backends requires careful alignment between non-blocking application loops and resource constraints. By implementing asynchronous architectures and optimized container multi-staging, software engineers can design resilient systems capable of sustaining high-throughput data pipelines efficiently. More

When AI Agents Call Your Microservices: 5 Assumptions That No Longer Hold

By Maryna Klochkova

Microservices were designed around a simple contract: a known caller sends a predictable request, expects a typed response, and moves on. That contract held for years. Then AI agents arrived. An agent doesn't call your service once. It might call it three times in a single reasoning loop, fan out to five services simultaneously, retry on ambiguous output, or decide mid-flight that a different endpoint is more appropriate. The distributed systems assumptions baked into your architecture rate limiting, idempotency, circuit breakers, auth flows were never built for a non-deterministic, autonomous caller. This article walks through the five core assumption breaks and how to address each one practically. 1. Requests Are Predictable Traditional microservice clients have a defined call pattern. A checkout service calls inventory once per order. A reporting job calls analytics on a schedule. Capacity planning is straightforward. Agents operate differently. A single user prompt can cause an agent to call the same service multiple times as it refines its reasoning, compare results from parallel tool calls, or loop back after a downstream failure. The call volume becomes a function of model behavior, not request count. Fix: decouple throughput from upstream intent Rate limiting per user session rather than per request becomes essential. Async queues absorb burst traffic without cascading pressure into downstream services. Python # Instead of synchronous fan-out: results = await asyncio.gather( inventory_service.check(item_id), pricing_service.get(item_id), supplier_service.availability(item_id) ) # Use a task queue with back-pressure: import asyncio from asyncio import Queue async def agent_tool_dispatcher(tasks: list[dict], concurrency: int = 3): semaphore = asyncio.Semaphore(concurrency) queue = Queue() for task in tasks: await queue.put(task) async def worker(): while not queue.empty(): task = await queue.get() async with semaphore: await dispatch(task) queue.task_done() await asyncio.gather(*[worker() for _ in range(concurrency)]) await queue.join() This pattern puts a ceiling on concurrent outbound calls regardless of how the agent reasons through its plan. 2. Idempotency Is a Nice-to-Have For human-driven workflows, duplicate requests are edge cases. A user double-clicks, a network retry fires. Idempotency keys handle it. For agents, duplicate calls are part of normal operation. The model may re-invoke a tool to confirm an earlier result, or retry after receiving an ambiguous response. Without strict idempotency at the service level, you get duplicate orders, double-charged transactions, or conflicting writes. Fix: idempotency keys scoped to the agent session Python import hashlib import json def make_idempotency_key(session_id: str, tool_name: str, payload: dict) -> str: """Stable key for a given agent session + tool call + payload combo.""" content = json.dumps({ "session": session_id, "tool": tool_name, "payload": payload }, sort_keys=True) return hashlib.sha256(content.encode()).hexdigest() # Usage in a tool wrapper async def safe_tool_call(session_id: str, tool_name: str, payload: dict): key = make_idempotency_key(session_id, tool_name, payload) # Check cache first cached = await redis_client.get(f"idem:{key}") if cached: return json.loads(cached) result = await call_tool(tool_name, payload) # Cache result with TTL matching the agent session lifetime await redis_client.setex(f"idem:{key}", 3600, json.dumps(result)) return result The key insight is scoping idempotency to the agent session, not just the request. The same tool call with the same arguments inside one reasoning loop should return a cached result rather than re-executing. 3. Auth Tokens Belong to a Human OAuth flows, JWT tokens, and API keys were designed around human sessions or long-lived service accounts. When an agent acts on behalf of a user, the auth picture gets complicated fast. The agent needs permissions to call your services. But it should not have the full permissions of the user it represents. An agent tasked with "summarize my open tickets" shouldn't also be able to close, delete, or reassign them just because the user can. Fix: scoped agent tokens with explicit capability grants Python from dataclasses import dataclass from typing import FrozenSet @dataclass(frozen=True) class AgentToken: user_id: str session_id: str allowed_tools: FrozenSet[str] allowed_actions: FrozenSet[str] # e.g. {"read", "comment"} not {"delete", "write"} expires_at: float def create_agent_token(user_id: str, task_description: str) -> AgentToken: # Derive minimum required permissions from task scope # In production, use a policy engine here if "read" in task_description.lower() or "summarize" in task_description.lower(): actions = frozenset({"read", "list"}) tools = frozenset({"tickets_read", "comments_read"}) else: actions = frozenset({"read", "write", "comment"}) tools = frozenset({"tickets_read", "tickets_write", "comments_write"}) return AgentToken( user_id=user_id, session_id=generate_session_id(), allowed_tools=tools, allowed_actions=actions, expires_at=time.time() + 3600 ) def enforce_agent_permissions(token: AgentToken, tool_name: str, action: str): if tool_name not in token.allowed_tools: raise PermissionError(f"Agent not authorized to use tool: {tool_name}") if action not in token.allowed_actions: raise PermissionError(f"Agent not authorized for action: {action}") This is the Principle of Least Agency, the agentic equivalent of least privilege. Scope the token to the task, not the user. For further reading on building secure agent permission layers, see Architecting Zero-Trust AI Agents: How to Handle Data Safely and Identity Security in the Age of Agentic AI. 4. Circuit Breakers Protect You From Downstream Failures Circuit breakers were designed to stop cascade failures when a downstream service degrades. They work on the assumption that failures are transient and rare. Agents introduce a different failure mode: the model retries not because of a network error, but because the response was ambiguous or incomplete. A circuit breaker that opens after three failures will open constantly when the agent is probing service behavior as part of its reasoning loop. Fix: distinguish agent retries from fault retries Python from enum import Enum import time class RetryReason(Enum): NETWORK_ERROR = "network_error" TIMEOUT = "timeout" AMBIGUOUS_RESPONSE = "ambiguous_response" # agent reasoning VALIDATION_FAILURE = "validation_failure" # agent reasoning class AgentAwareCircuitBreaker: def __init__(self, failure_threshold: int = 5, recovery_timeout: int = 30): self.failure_threshold = failure_threshold self.recovery_timeout = recovery_timeout self.fault_failures = 0 # Only infrastructure failures count self.last_failure_time = None self.open = False def record_failure(self, reason: RetryReason): # Only count infrastructure failures toward circuit state if reason in (RetryReason.NETWORK_ERROR, RetryReason.TIMEOUT): self.fault_failures += 1 self.last_failure_time = time.time() if self.fault_failures >= self.failure_threshold: self.open = True # Agent reasoning retries are tracked separately for observability # but do not trip the breaker def can_proceed(self) -> bool: if not self.open: return True if time.time() - self.last_failure_time > self.recovery_timeout: self.open = False self.fault_failures = 0 return True return False Agent reasoning retries should be tracked for observability (they signal prompt or schema issues), but they should not influence circuit state. For a deeper treatment of building reliable pipelines around non-deterministic callers, Build Self-Managing Data Pipelines With an LLM Agent is worth reading alongside this. 5. Structured Logs Are Enough for Debugging When a human-driven request fails, you have a stack trace, a request ID, and a timestamp. That's usually enough to reconstruct what happened. When an agent-driven call fails, you need the entire reasoning trace: what the model decided, which tools it called in what order, what intermediate outputs looked like, and where the logic diverged. A standard log line doesn't carry that context. Fix: structured agent traces as first-class observability Python import uuid from datetime import datetime, timezone from typing import Any class AgentTracer: def __init__(self, session_id: str): self.session_id = session_id self.trace: list[dict] = [] def log_tool_call(self, tool_name: str, input_payload: dict, output: Any, duration_ms: float, reason: str = ""): self.trace.append({ "event": "tool_call", "session_id": self.session_id, "trace_id": str(uuid.uuid4()), "timestamp": datetime.now(timezone.utc).isoformat(), "tool": tool_name, "input": input_payload, "output_summary": str(output)[:500], # truncate for log size "duration_ms": duration_ms, "agent_reason": reason # capture model's stated rationale if available }) def log_decision(self, decision: str, context: dict): self.trace.append({ "event": "agent_decision", "session_id": self.session_id, "timestamp": datetime.now(timezone.utc).isoformat(), "decision": decision, "context_keys": list(context.keys()) }) def export(self) -> list[dict]: return self.trace The session ID ties the entire reasoning chain together across service boundaries. Every tool call, decision, and retry is logged with its agent-level context, not just its network-level context. For production-grade observability patterns on this, Production Checklist for Tool-Using AI Agents in Enterprise Apps covers instrumentation, eval gates, and cost monitoring in detail. Putting It Together These five fixes share a common thread: they treat the agent as a distinct caller type, not a faster human. Your services don't need to know about LLMs. They need to handle non-deterministic, high-frequency, session-scoped callers that have constrained permissions and generate reasoning traces, not just request logs. The patterns above are not exotic. Idempotency keys, scoped tokens, semaphore-bound queues, and structured traces are standard distributed systems tools. What changes is when and why you apply them. For agents, they move from edge-case handling to baseline requirements. If you're moving agents into production and want a complete pre-launch checklist, the Shipping Production-Grade AI Agents Refcard published by DZone covers the full lifecycle from state management to cost controls. The underlying infrastructure doesn't change. The assumptions about who calls it do. More

Anti-Patterns of Microservices Architecture From Real Production Experience

By Ivan Balashov

Architecting Autonomous Network Ecosystems: From Reactive Monitoring to Agentic AI Orchestration

By Daniel Oh

CORE

Your Agent Trusts That Wiki. Should It?

By Abhinav Srivastava

Getting Started With RabbitMQ in Spring Boot

RabbitMQ is an enterprise-grade open-source messaging and streaming broker. In this blog, you will learn some basic concepts of RabbitMQ and how to use it in a Spring Boot application. Enjoy! Introduction Before diving into the programmatic details, first some concepts need to be explained. Do realize that in this blog, only the surface is scratched from what is possible with RabbitMQ. A detailed overview can be found in the official RabbitMQ documentation. Several protocols are supported by RabbitMQ. In this blog, the AMQP 0-9-1 protocol will be used. AMQP stands for Advanced Message Queuing Protocol. RabbitMQ receives messages from a publisher, a producing application, and routes them to consumers, applications that process the messages. A publisher publishes messages to an exchange (like a mailbox). The exchange then routes the messages to queues using bindings. RabbitMQ then delivers the messages to the consumers who are subscribed to the queues. The process is shown in the figure below. In the examples in the remainder of this blog, you will make use of a Topic Exchange. There are different exchange types, but for the sake of simplicity, only one will be used. A topic exchange routes messages to one or many queues, based on a message routing key. Topic exchanges are commonly used for multicast routing of messages. Sources used in this blog are available on GitHub in module topics. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose. Create Spring Boot Application In order to get started, you navigate to the Spring Initializr and add the following dependencies: Spring Web: in order to be able to send messages via an http request.Docker Compose Support: in order to start a RabbitMQ container when the application starts.Spring for RabbitMQ: in order to integrate Spring Boot with RabbitMQ. You will build the following: One Exchange with one Topic.Publish a general message to the topic which will be consumed by consumer A and consumer B.Publish a specific message to the topic which will be only consumed by consumer B. In order to send a general and a specific message, two HTTP endpoints are created in the MessageController. Java @RestController public class MessageController { private MessageService messageService; public MessageController(MessageService messageService) { this.messageService = messageService; } @RequestMapping( method = RequestMethod.POST, value = "send-general" ) public ResponseEntity<Void> sendGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } @RequestMapping( method = RequestMethod.POST, value = "send-specific" ) public ResponseEntity<Void> sendSpecificMessage(@RequestBody String message) { messageService.sendMessage("event.specific.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } } The requests are forwarded to a MessageService.sendMessage method, which takes a routingKey and the message as arguments. The message is taken from the http request body, the routingKey is hardcoded. Remember that the routingKey determines to which queue the message will be routed. In the service, you make use of Spring Boot's RabbitTemplate in order to send the message to RabbitMQ. Java @Service public class MessageService { private RabbitTemplate rabbitTemplate; public MessageService(RabbitTemplate rabbitTemplate) { this.rabbitTemplate = rabbitTemplate; } public void sendMessage(String routingKey, String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.TOPIC_EXCHANGE_NAME, routingKey, message); } } Bind Consumer A Consumer A will consume general messages. The queue needs to be bound to the Topic Exchange with the routing key. Create a RabbitMqConfig class with: A TopicExchange bean with name events.exchange.A Queue bean for consumer A with name consumer-a.queue.A binding bean for consumer A connecting the queue of consumer A to the TopicExchange with the routing key for the general messages. Do note that the name of the queue in method bindingConsumerA needs to match the queueConsumerA bean name. Java Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; @Bean TopicExchange eventsExchange() { return new TopicExchange(TOPIC_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } } Create Consumer A Next thing to do is to consume the messages from queue A. Create a Component named ReceiverA. Annotate the method for processing the messages with @RabbitListener and connect it to queue A. When receiving the message, just print it to the console. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } Run the Application In order to run the application, you will need RabbitMQ. Since you have added Docker Compose Support to the project earlier, you can just add a compose.yaml in the root of the repository. YAML services: rabbitmq: image: rabbitmq:3.13-management-alpine # Stable, lightweight, includes management UI container_name: rabbitmq ports: - "5672:5672" # AMQP - "15672:15672" # Management console environment: RABBITMQ_DEFAULT_USER: secret RABBITMQ_DEFAULT_PASS: myuser Also add the connection parameters for RabbitMQ to the application.properties file. Properties files spring.rabbitmq.host=localhost spring.rabbitmq.port=5672 spring.rabbitmq.username=secret spring.rabbitmq.password=myuser Start the application. Shell mvn spring-boot:run You will notice that RabbitMQ is started automatically. Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The console log will print the following. Plain Text Queue Consumer A received <This is a general message> Stop the application. Bind Consumer B Consumer B will process general messages, but also specific messages. Add to the RabbitMqConfig the queue for consumer B, and bind it to the exchange with respectively the general message routing key and the specific message routing key. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String QUEUE_CONSUMER_B = "consumer-b.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; public static final String ROUTING_KEY_SPECIFIC_MESSAGE = "event.specific.*"; ... @Bean public Queue queueConsumerB() { return new Queue(QUEUE_CONSUMER_B, false); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } } Create Consumer B Consumer B is created just like consumer A. Create a ReceiverB class in order to receive the queue B messages. Java @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_B) public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Run the Application Start the application. Shell mvn spring-boot:run Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The message is now received by Consumer A and Consumer B. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Send a specific message. Shell curl -X POST http://localhost:8080/send-specific \ -H "Content-Type: text/plain" \ -d "This is a specific message" The message is only received by Consumer B. Plain Text Queue Consumer B received <This is a specific message> Management Console Also take a look at the RabbitMQ management console, which is accessible at http://localhost:15672/. Here you can see the exchanges, the queues, the bindings, etc. Conclusion In this blog, you learned some basics of RabbitMQ using the AMQP 0-9-1 protocol. You learned how easy it is to integrate this within your Spring Boot application.

By Gunter Rotsaert

CORE

Resilience Lost in the Stack: How Abstraction Layers Silently Mask Distributed Systems’ Topology Awareness

Distributed coordination services exist for a reason, and they are the CPUs of distributed systems that give them their high availability. When it's in your stack, you assume failover is handled. Some services that operate in this layer include Apache Zookeeper, Redis Sentinel, etcd, etc. These services are mathematically engineered for HA. Protocols such as Raft/Paxos/ZAB guarantee this. We know that the DCS itself cannot go wrong as long as a quorum of nodes exists. Here, we want to explore one specific problem that makes this high availability subjective. It is an issue where individual layers hold this promise, while as we go to higher-level abstractions, the intelligence silently dies. The article focuses on how topology awareness needs to be preserved mindfully as we move up the stack, and that, when using smart clients and drivers, we should inherit the responsibility not to silence their intelligence. We take a use case in the Java ecosystem. In a production-grade system, we have microservices distributed across multiple regions that take care of specific components. We discuss a rate-limiting use-case here to demonstrate the underlying problem. This is one area where the problem manifests itself. A similar architecture can still have the same problem under the hood. Deconstructing the Stack The use case: A rate limiter that is implemented in production-grade using Bucket4j. A distributed rate limiter needs to keep track of token buckets across multiple instances. We assume the application runs in 3 instances and want to rate-limit requests to 5 req/sec. To enforce this strict global token limit across 3 instances, a centralized state coordinator becomes mandatory. So we introduce Redis, which centrally stores the token bucket state. This way, we decouple the bucket state from application instances and do not make every instance accept 5 tokens each, making it 15 req/sec. Redis is a distributed caching layer. In Java, if you're using Redis, you're likely talking to it through Lettuce, Redisson, or Jedis. Since Lettuce is the widely adopted Redis client, it acts as the asynchronous engine that bridges the Java application layer with the Redis database infrastructure. A sample boilerplate for the connection configuration would look something like: Java @Bean public ProxyManager<String> proxyManager() { RedisURI.Builder builder = RedisURI.builder().withSentinelMasterId(masterId).withTimeout(Duration.ofSeconds(12));; for (String node : nodes) { String[] hostPort = node.split(":"); builder.withSentinel(hostPort[0], Integer.parseInt(hostPort[1])); } RedisClient redisClient = RedisClient.create(builder.build()); RedisCodec<String, byte[]> bucket4jCodec = RedisCodec.of(StringCodec.UTF8, ByteArrayCodec.INSTANCE); StatefulRedisConnection<String, byte[]> redisConnection = redisClient.connect(bucket4jCodec); return LettuceBasedProxyManager.builderFor(redisConnection) .withClientSideConfig(ClientSideConfig.getDefault()) .build(); } This code looks straightforward in terms of establishing the connection. We build the Redis cluster nodes along with the Sentinel configuration.Feed this to the Redis client and plug this code into the Bucket4j instance.Create a StatefulRedisConnection wrapper available as part of the Bucket4j library.Configure a LettuceProxyManager that interacts with the Redis cluster. This code complies properly, passes all integration tests, and handles throttling by rate limiting, yet silently harbors a topology blindness that will cause the application to stall during a failover. The Failure Mode: What It Looks Like At the time of failover, when the primary Redis master suddenly drops, the Redis sentinel infrastructure kicks in and promotes a healthy replica to master. It does so by holding a rapid quorum vote, and high availability is achieved. But at the application side, we experience the following: Thread Stalls and Application Freeze The app does not throw any obvious connection errors; instead, it just freezes and waits indefinitely for Redis to recover. The executing thread stalls in the process and never recovers. On the application side, Bucket4j continuously starts receiving requests, and its internal token buckets are routed to a broken Redis connection. The Illusion of Network Failure Internally, Lettuce has methods and wrappers to handle this type of failure, and it transparently buffers and queues commands while trying to reconnect. But Bucket4j does not have this information and keeps waiting for Lettuce. Even a generous timeout from the application side is not going to help us in this situation. The Rate Limiter Paradox While some threads are stalled, other parts of the application may continue; for instance, the endpoint might still receive requests and route them to Bucket4j, but because this exception is swallowed, no actual rate limiting occurs. Bucket4j keeps talking to a broken connection and doesn't ideally keep track of tokens. The Wrapper Deficit While Lettuce does have a StatefulMasterReplicaConnection that comes with topology awareness, Bucket4j never exposes wrappers to use this StatefulMasterReplicaConnection. So a user using Bucket4j may or may not be aware of internal wrappers available in Lettuce. In the case where this is not known to the user, engineers naturally instantiate static connections and can easily overlook this. This results in code that seems to handle failovers but is completely devoid of master-replica awareness. System architecture with the topology awareness blindspot The Abstraction Blindspot This becomes hard to catch at some level precisely because every layer seems to work and does its job correctly. Redis-Sentinel experiences a failover and successfully recovers. Meanwhile, lettuce, the client that interacts with Redis, also has a MasterReplicaConnections that is capable of knowing that this event has occurred. Bucket 4j is responsible for token buckets and rate limiting, and it is also doing its job well. The problem happens in the composition. This brings us to a broader principle: abstraction layers do not just simplify complexity but also inadvertently suppress capabilities silently. The HA awareness exactly gets broken at this point in the stack Why did testing not expose this? Unit tests almost always don't uncover these types of errors. Load/Performance focus on high throughput under normal conditions to ensure the rate limiter is functioning correctly and is handling throttling.Health checks and readiness probes target the wrong layer, namely Redis, in ensuring availability. The Solution Blueprint To solve the thread stall and force the application layer to inherit The High Availability of Redis, we have to preserve the topology at every layer of the stack. The fix requires choosing the exact Lettuce connection interface that tracks topology shifts while preserving the raw command execution engine. Navigating Lettuce’s Topology refresh options connecTion interfacetarget redis infrause case StatefulRedisConnection Standalone Node General single-node use; blind to topology changes. StatefulRedisClusterConnection Sharded Cluster Data partitioning across many nodes. StatefulRedisPubSubConnection Messaging Channels Real-time pub/sub event listening. StatefulRedisSentinelConnection Sentinel Nodes Directly Administrative tracking and master discovery. StatefulRedisMasterReplicaConnection Primary + Replicas via Sentinel Dynamic health tracking, automatic failover, and read/write splitting. Why We Choose StatefulRedisMasterReplicaConnection over StatefulRedisSentinelConnection When working with Redis and when the need is to explicitly inherit Sentinel properties, the intuitive solution is to reach StatefulRedisSentinelConnection. However, a closer look at the source code for RedisSentinelConnection extends the basic StatefulConnection interface, and its async command blocks only expose Sentinel APIs. Java public interface StatefulRedisSentinelConnection<K, V> extends StatefulConnection<K, V> { RedisSentinelAsyncCommands<K, V> async(); } // Sneak peek inside RedisSentinelAsyncCommands: RedisFuture<List<Map<K, V>>> slaves(K key); RedisFuture<String> failover(K key); RedisFuture<String> monitor(K key, String ip, int port, int quorum); RedisFuture<Long> reset(K key); A point to note here is that the standard data manipulation commands like GET, SET, HGET are absent in this abstraction. We would require evaluation scripts for the wrapping client (Bucket4j) to execute Lua scripts. This shows that the interface was built to manage the cluster but not to read and write app data. On the other hand, StatefulRedisMasterSlaveConnection directly extends StatefulRedisConnection, inheriting the complete data manipulation layer. Java public interface StatefulRedisMasterReplicaConnection<K, V> extends StatefulRedisConnection<K, V> { void setReadFrom(ReadFrom readFrom); RedisAsyncCommands<K, V> async(); // Exposes GET, SET } By choosing StatefulRedisMasterReplicaConnection instantiated via a Sentinel-backed MasterReplica builder, we inherit: Topology awareness: As it hooks directly into the Sentinel Pub/Sub event stream to automatically reroute trafficAsynchronous engine preservation: Which exposes the RedisAsyncCommands necessary for Bucket4j to asynchronously execute thread-safe token The Wrapper Integration To cleanly connect our new topology-aware connection with the rate-limiter in our example (Bucket4J), we introduce a dedicated wrapper to the integration layer. Java public static <K> LettuceBasedProxyManagerBuilder<K> casBasedBuilder(StatefulRedisMasterReplicaConnection<K, byte[]> statefulRedisMasterReplicaConnection) { return casBasedBuilder(statefulRedisMasterReplicaConnection.async()); } A word on CAS: Compare-And-Swap (CAS) is a builder that uses a non-blocking database pattern to update data safely without heavy locks. It reads the token bucket value, does the math, and writes it back only if another thread hasn't changed it in the meantime. If the value did change, it safely retries the operation automatically. Bucket4j exposes similar builders for CAS. This builder expects a standard Lettuce asynchronous command interface. By including the above builder, we enable Bucket4j’s proxy manager to accept a StatefulRedisMasterSlaveConnection. Validation Through a Little Chaos Engineering The Test Stack Since standard testing frameworks won’t expose this issue, we needed a real-world setup to simulate the production environment. The test stack included: Docker and Docker Compose: To manage a multi-node Redis cluster (1 Master, 2 replicas, 3 sentinels)Java/Springboot: The host-side sample application integrating the Bucket4j logic to rate limit an endpointLettuce/Bucket4j: The libraries that we want to test Apache Benchmark: A command-line utility used for injecting the load Test Setup The following docker-compose.yml served as the baseline configuration for the Redis setup. YAML version: '3.8' services: redis-master: image: redis:7-alpine container_name: redis-master # Network mode host maps directly to your machine's ports, bypassing docker bridge DNS network_mode: "host" command: redis-server --port 6379 redis-replica: image: redis:7-alpine container_name: redis-replica network_mode: "host" # Since we are on host mode, the replica connects to localhost 6379 and binds its own engine to 6380 command: > redis-server --port 6380 --replicaof 127.0.0.1 6379 --replica-announce-ip 127.0.0.1 --replica-announce-port 6380 depends_on: - redis-master redis-sentinel: image: redis:7-alpine container_name: redis-sentinel network_mode: "host" command: > sh -c " echo 'port 26379' > /sentinel.conf && echo 'sentinel monitor mymaster 127.0.0.1 6379 1' >> /sentinel.conf && echo 'sentinel down-after-milliseconds mymaster 3000' >> /sentinel.conf && echo 'sentinel failover-timeout mymaster 6000' >> /sentinel.conf && redis-server /sentinel.conf --sentinel " depends_on: - redis-master - redis-replica Here, we use a Redis master-replica configuration and a portable built-in sentinel.conf. This makes it easier and starts the service using the configs provided in the file. Architectural Parameters down-after-milliseconds mymaster 3000: Sentinel waits for 3 seconds before deciding the master is unreachable. The host is marked “subjectively down” (SDOWN) if the master does not continuously respond in this window.failover-timeout mymaster 6000: The window for the promotion of a new master and the reconfiguration of the cluster. At this point, it is marked “objectively down” (ODOWN) and starts the failover. Step 1: Validating the Infrastructure Baseline and Sanity Before testing how it performs with different abstractions, a standard failover was executed to see if redis-sentinel was working as expected and proceeded with the leader election. Shell docker stop redis-master The logs where Sentinel performs a leader election process: Shell redis-replica-1 | 1:S 17 May 2026 19:44:23.894 # Unable to connect to MASTER: Success sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +sdown master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +odown master mymaster redis-master 6379 #quorum 1/1 sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +try-failover master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +vote-for-leader e1db4435d0770f294d0d13835729c5102cb5a4cd 1 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +elected-leader master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +failover-state-select-slave master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.850 # +selected-slave slave 172.18.0.3:6379 172.18.0.3 6379 @ mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.850 * +failover-state-send-slaveof-noone slave 172.18.0.3:6379 172.18.0.3 6379 @ mymaster redis-master 6379 Step 2: The Naive Connection and Indefinite Freeze We now use the StatefulRedisConnection and expose a test endpoint from our SpringBoot application. This endpoint is now sent 10,000 concurrent requests, and mid-stream we kill the master node to allow for the re-election of the master. Java @GetMapping("/test") public ResponseEntity<String> handleRequest() { Bucket bucket = proxyManager.builder().build("reproduction-key", () -> bucketConfiguration); // Under high traffic concurrent loads, this execution point will freeze solid // the moment the master container is stopped! if (bucket.tryConsume(1)) { return ResponseEntity.ok("SUCCESS"); } else { return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS).body("RATE_LIMITED"); } Observation: The application logs revealed a blind reconnection loop. The client was stuck knocking on the door of the dead port 6379, oblivious to the new Master on 638. The infrastructure healed at this point; however, the application remained in an indefinite freeze. Even when we increased the command timeout to 12 seconds, the results were the same. The Apache Benchmark test did not complete and timed out. Below are the results: Plain Text rithraravikumar@Rithras-MacBook-Air redis-sentinel-lab % ab -n 10000 -c 10 http://localhost:8080/test This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests apr_pollset_poll: The timeout specified has expired (70007) Total of 5460 requests completed Step 3: The Master-Replica Connection Pivot The final step of the process was to use the MasterReplicaStatefulConnection and observe if the application bounced back. The code change looks like the below: Java RedisClient redisClient = RedisClient.create(builder.build()); RedisCodec<String, byte[]> bucket4jCodec = RedisCodec.of(StringCodec.UTF8, ByteArrayCodec.INSTANCE); RedisURI sentinelUri = RedisURI.builder() .withSentinelMasterId(masterId) // Looks up "mymaster" .withSentinel("127.0.0.1", 26379) // The host and port where Sentinel is listening .build(); StatefulRedisMasterReplicaConnection<String, byte[]> redisConnection = MasterReplica.connect(redisClient, bucket4jCodec, sentinelUri); return LettuceBasedProxyManager.builderFor(redisConnection) .withClientSideConfig(ClientSideConfig.getDefault()) .build(); Observation: With this topology-aware connection, when we killed the master node at the 5,000-request mark. While there was a noticeable stall, the connection was able to recognize there was a new master and resumed processing. Plain Text rithraravikumar@Rithras-MacBook-Air redis-sentinel-lab % ab -n 10000 -c 10 http://localhost:8080/test This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests .... Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Time taken for tests: 24.522 seconds Complete requests: 10000 Failed requests: 5983 (Connect: 0, Receive: 0, Length: 5983, Exceptions: 0) Non-2xx responses: 5983 Total transferred: 1814793 bytes HTML transferred: 656334 bytes Requests per second: 407.80 [#/sec] (mean) Time per request: 24.522 [ms] (mean) Time per request: 2.452 [ms] (mean, across all concurrent requests) Transfer rate: 72.27 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 2.5 0 251 Processing: 0 24 671.0 1 21235 Waiting: 0 24 671.0 1 21234 Total: 0 24 671.0 1 21235 Analyzing the Metrics The test recorded 5,983 failed requests (non-2xx responses). Those 5,983 failures represent the exact window of time when the master went down, and Bucket4j rejected traffic or failed fast instead of hanging. The longest request took a massive 21.2 seconds (21235 ms) to process. The critical difference was that the application bounced back. The Bigger Lessons — Not Just Topology Framework abstractions can inadvertently mask underlying driver capabilities, turning a standard infrastructure-level database failover into an application-level thread stall.When configuring stateful caching or synchronization wrappers, developers must ensure that connection pools explicitly utilize master-replica topology providers rather than static endpoints.High-availability verification cannot rely on static environment tests; engineers must actively simulate node terminations under concurrent load using tools like Apache Benchmark to uncover edge-case race conditions.Designing frameworks with extensible builder patterns allows downstream developers to inject custom infrastructure topologies without altering the core business logic of the library.Generous timeouts during a network failure are not an effective strategy if your client driver is blind to network routing shifts, as it merely forces application threads to spend more time waiting on a dead address.Under high-concurrency workloads, a client's internal memory buffer will saturate within milliseconds, rapidly cascading into total application thread pool exhaustion.

By Rithra Ravikumar

One Stolen Key, One Stolen Token: Why Machine Identity Is Cloud-Native's Quietest Crisis — and the Only Fix That Actually Holds

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be.

By Igboanugo David Ugochukwu

CORE

Building Production-Safe Agentic Remediation With Docker MCP Gateway: Lessons From 43% to 100% Accuracy

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

By Mohammad-Ali Arabi

Who Owns the Data Stack?: How AI Is Reshaping Ownership, Architecture, and Accountability Across Teams

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads. For years, some of us have argued that the data stack is part of the product and should be engineered like the application layer: as code and as a service. The market matured toward it, and the data mesh has been the clearest recent expression. AI has eclipsed those debates and settled the matter. The data stack is now product-facing, shaping what users see, what AI answers, and which automated decisions and workflows fire. That makes one question unavoidable: When an answer depends on data across many systems and teams, who is accountable for accuracy? An AI answer is assembled at request time from corporate data. The data stack is inside the response. AI Turns Data Infrastructure Into Product Behavior AI makes the data stack part of product behavior, but raw infrastructure should not leak into the product. The goal is to abstract the stack behind durable, governed interfaces. An AI feature should consume meaning, relationships, permissions, and context. Following data mesh and data contracts, the API layer has to evolve from returning data to exposing capabilities. A consumer, including an AI model, should depend on a contract that carries: Metadata – origin, lineage, meaningQuality – freshness, completeness, confidenceRelationships – how entities compose and traverseSecurity – authorization applied consistently across operational, analytical, and vector stores When meaning lives in the contract, infrastructure becomes interchangeable, and a misbehaving AI feature is no longer an opaque failure — it’s a question with an owner. Where Ownership Breaks First Ownership does not break at the edges of systems but much earlier, in how the organization is designed. Most technology organizations still distribute teams around components and technical specialization: applications, databases, pipelines, governance, indexing, and analytics. Each team owns its layer, though no one owns the end-to-end meaning of the data. That worked when data only fed analytics. It fails in AI-native products, where data is product behavior, and the two lifecycles are inseparable. AI composes its behavior across every layer at once, inheriting each inconsistency in semantics, freshness, permissions, and relationships. So this is not a handoff problem; it is a Conway’s Law problem. Architecture mirrors the organization, and AI makes the organizational seams visible to the user. Platform teams remain essential for shared abstractions, governance primitives, and standards. But product teams need to own both their features and APIs and their data end to end: its lifecycle, meaning, quality, and governance. Splitting teams by technical layer scatters one business entity across many disconnected owners, and AI inherits that fragmentation. AI-native organizations give product teams end-to-end ownership of the data, with platform teams providing shared standards. Accountability Follows Product Behavior When data only fed dashboards, accountability could stay narrow: Did the pipeline run, and did the report match? AI moves that boundary. Once retrieval, copilots, and agents start making decisions and generating answers from data, a correct pipeline, a healthy index, and a valid access policy still don’t guarantee a correct user-facing result. Accountability can’t be pinned to technical layers. It has to follow the behavior the user experiences. The product team that owns an AI capability is responsible for the end-to-end correctness, freshness, explainability, and safety of the data behind it. Its job is to own the contract that defines what the AI may know and retrieve. Platform teams provide the standardized primitives that make this accountability structure possible: semantic contracts, lineage, quality signals, access enforcement, observability, and governance-aware retrieval. The question shifts from “which team owns this layer?” to “which product team owns this behavior, and which platform capabilities guarantee it?” In AI-native systems, accountability rests with the team that owns the behavior, not the system that happened to fail. Table: Accountability Differences Between the Layer-by-Layer and AI-Native Models arealayer-by-layerai-nativeSource of truthEach system decides locallyThe product team owns the authoritative semantic contractQualityThe data team checks pipelinesThe product team owns user-facing correctness; the platform provides quality signalsRetrievalThe platform team owns indexes as infrastructureA governed product capability with explicit SLOsAccessThe security team owns policies separatelyEnforced consistently across product, data, and AI layersIncidentsRouted to whichever layer failedThe product team leads; the platform, data, and security teams support as capability owners Architecture Choices Are Also Operating Model Choices Architecture decisions also decide how an organization governs and evolves meaning. AI-native systems raise the stakes here because copilots and agents consume meaning — entities, relationships, metrics, and permissions — rather than tables. Semantic consistency becomes part of how the product behaves. No central team can own the meaning of every domain, so meaning has to live close to the domain that owns the capability. But decentralization alone backfires: Without platform-enforced standards, the old central bottleneck just turns into semantic fragmentation, with every domain exposing its own definitions and contracts. The fix is to split ownership cleanly: Domains own the meaningPlatform teams own the contracts that keep it consistent Underneath, storage and processing keep churning. What actually lasts is whether stable abstractions (e.g., “employee,” “payroll,” “entitlement”) survive above them. The principle is simple: Infrastructure should be replaceable, and meaning should not. So the real operating-model choice comes down to who owns meaning, and who keeps it consistent. Shared Data Contracts Make Accountability Concrete If organizational fragmentation is the root problem, contracts make ownership explicit. A classic data contract is necessary but insufficient. Schema validation catches a renamed column, but it misses semantic drift, stale meaning, or a changed business definition. Those failures don’t break a build. They break behavior. The contract has to grow from schema into semantics, carrying meaning, lineage, quality, and authorization. Crucially, it abstracts the capability and meaning a domain exposes, not the storage format underneath, so it behaves the same whether the source is a table, a document, an event, or an embedding. That makes the data contract both a producer-to-consumer check and a runtime semantic interface that retrieval, copilots, and agents all consume. Its real value is relocating accountability to the source so drift surfaces in the producing domain while context stays local, which accelerates interoperability rather than centralizing control. Governance Has to Travel With the Data Traditional governance sat beside the data in the form of periodic reviews, approvals, and access checks. AI breaks that model. Data now moves continuously through pipelines, caches, embeddings, indexes, and agents, recomposed at runtime faster than any review can observe. Governance must be part of the execution model itself. Governance travels with meaning, not storage. An embedding holds no raw rows yet reveals sensitive meaning, so policy must follow the semantic classification. The gap is sharpest in authorization. Identity systems stop at the API boundary, and AI doesn’t preserve security boundaries on its own, which turns every embedding, cache, and retrieval step into a new one to defend. Governance therefore becomes a runtime capability that decides what AI may retrieve, infer, expose, and act on. Solving that calls for composable, declarative governance primitives embedded in the platform so auditability becomes a property of the system rather than the outcome of a project. Accountability Gaps That Slow AI Data Work The real cost of fragmented accountability is the constant drag on every data-powered capability. Friction is never neutral, so when teams can’t trust the platform’s freshness, semantics, or governance, they route around it and build their own, resulting in shadow pipelines, local indexes, and duplicated transformations. Each workaround makes sense locally even as it corrodes the whole, fragmenting governance and eroding trust in the very platform it was meant to replace. And piling on more central control only hides the problem — the fragmentation just migrates into those shadow systems. So the deeper gap was missing platform contracts. What Clear Ownership Looks Instead of adding more teams on more layers, clear ownership means aligning accountability with the single product experience the user meets. What you’re really investing in is the stable semantic abstractions that outlast whatever infrastructure comes and goes. And the hardest problem is how to make the organization understandable to its own AI systems. Additional resources: DAMA-DMBOK: Data Management Body of KnowledgeDAMA International – foundational guidance on data ownership, stewardship, and governance rolesOpen Data Contract Standard (ODCS) – an open spec for declaring schema, semantics, quality, and service levels between data producers and consumersOpenLineage – an open standard for collecting data lineage across pipelines and services, useful for tracing what AI features consumeNIST AI Risk Management Framework (AI RMF) – a vendor-neutral framework for accountability and governance of AI systemsCoral – exposes diverse data sources to agents through one declared SQL and semantic layer; an example of meaning being owned per source rather than centrallyGetting Started With Data Quality, DZone Refcard by Miguel García LorenzoData Pipeline Essentials, DZone Refcard by Sudip SenguptaOpen-Source Data Management Practices and Patterns, DZone Refcard by Abhishek GuptaReal-Time Data Architecture Patterns, DZone Refcard by Miguel García Lorenzo“Building Trusted, Performant, and Scalable Databases: A Practitioner’s Checklist” by Saurabh Dashora This is an excerpt from DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads.Read the Free Report

By Miguel Garcia

CORE

Your AI Coding Agent Can't Steal What It Never Had: The Docker Sandbox Isolation Story

I ran an AI coding agent against a broken Kubernetes deployment for five minutes. The agent called Anthropic's API dozens of times — reasoning about manifests, running kubectl commands, redeploying workloads. It made fully authenticated requests throughout the entire session. The API key was never in its environment. Shell env | grep -iE "anthropic|api_key|secret|token|password" # (empty) That is Docker Sandbox's credential isolation model in action. This article is about what that actually means — and what else the isolation holds, breaks, and surprises you with when you probe it properly. Key Takeaways Docker Sandbox uses a host-side proxy to inject API credentials without the agent ever seeing them — the agent makes authenticated calls without possessing the keySeven live isolation probes confirmed the boundary held throughout real AI agent activity, not just at restNetwork policy is hostname-scoped HTTP filtering — not a full network control plane — with three specific behaviors the documentation doesn't make clearDevOps agents can run docker build and kubectl inside the sandbox without any path to the host Docker daemon or cluster credentialsThe --branch parallel agent mode is Git-level isolation, not VM-level — important distinction for threat models requiring separate credentials per agent The Setup I manage eight AKS clusters for Fortune 500 clients. My laptop has Azure service principals, SSH keys, kubeconfig files with a dozen cluster contexts, and twenty-plus repos — some with .env files containing real API keys. Running an AI agent from this machine without guardrails means the agent inherits all of it. Docker Sandbox changes that. Each sandbox is a microVM — its own Linux kernel, its own Docker daemon, its own network stack. You mount one project directory. The agent sees one project directory. Everything else on the machine does not exist inside the sandbox. I spent two weeks testing this claim. Here is what I found. Test environment: What Detail sbx version v0.31.1 · commit e658be1 Host macOS Apple Silicon Network endpoints probed 13 Isolation probes 7 targeted commands Kubernetes scenario Real agent task, two bugs, timed All findings backed by real terminal output. Full repo: github.com/opscart/docker-sandbox-devops. How the Credential Isolation Actually Works The sandbox environment has no API keys. But the agent made authenticated API calls. Here is the mechanism: Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # http_proxy=http://gateway.docker.internal:3128 # JAVA_TOOL_OPTIONS=-Dhttp.proxyHost=gateway.docker.internal -Dhttp.proxyPort=3128 ... Every outbound request — HTTP, HTTPS, even Java tools — routes through a proxy at gateway.docker.internal:3128. That proxy runs on the Mac host, completely outside the microVM boundary. When the agent sends a POST to api.anthropic.com, there is no Authorization header — the agent does not have the key. The request reaches the host-side proxy. The proxy checks the allowlist — api.anthropic.com is in the default AI services group under the Balanced policy. Authentication is performed by the host-side proxy using credentials stored outside the sandbox boundary. The authenticated request is forwarded to Anthropic. The agent receives the response. It has no idea what key was used, where it came from, or how to find it again. Think of it like an OAuth gateway. The proxy holds the credential and vouches for the agent's requests. The agent gets access without ever possessing the key. You cannot steal what you never had. This is architecturally different from the standard setup where ANTHROPIC_API_KEY sits in the shell environment — one echo $ANTHROPIC_API_KEY away from being exfiltrated. What the Four Isolation Layers Actually Do Docker Sandbox stacks four layers: Hypervisor isolation. Separate Linux kernel per sandbox. Host processes invisible. Other sandboxes invisible. A compromised sandbox cannot escalate to the host kernel. This is the fundamental difference from a Docker container — a container shares the host kernel. The microVM does not. Network isolation. All outbound HTTP/HTTPS routes through the host-side proxy. Raw TCP, UDP, and ICMP are blocked at the network layer. Three policy tiers: allow-all, balanced (curated dev allowlist), deny-all. Set before starting your first sandbox: Shell sbx policy set-default balanced Docker Engine isolation. Each sandbox runs a private Docker daemon with its own socket. No path to the host Docker daemon. An agent can run docker build and docker run without socket mounting — which is the tradeoff that breaks isolation in plain container-based approaches. Credential isolation. Proxy-based injection as described above. The raw key never enters the microVM. macOS host with sensitive assets and proxy on the left, Docker Sandbox microVM in the center, network policy zones on the right. Seven Isolation Proofs — Run Live After a Real Agent Task The agent exited after completing the debugging task. The sandbox remained alive, and I executed the following commands from the same shell session the agent had used — to show exactly what was accessible throughout the entire run. 1. Filesystem Boundary Shell ls /Users/opscart/ # Source ls /Users/opscart/.ssh/ 2>&1 One directory. The workspace mount. SSH keys, other repos, credential directories — none of them exist inside the sandbox. Parent directories above the workspace are read-only stubs with no siblings. One critical implication: if your workspace is your home directory, your entire home is visible and writable. Always mount a project subdirectory, not your home. 2. No Credentials in Environment Shell env | grep -iE "anthropic|api_key|aws|secret|token|password" # (empty) Confirmed. The agent that just made dozens of API calls had no raw credentials anywhere in its environment. 3. Proxy Confirms the Injection Mechanism Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # no_proxy=localhost,127.0.0.1,::1,[::1],gateway.docker.internal Proxy address visible. Credentials it carries: not visible. The mechanism described above confirmed live inside the running sandbox. 4. Process Namespace Shell ps aux | wc -l # 13 A macOS host runs hundreds of processes. The sandbox shows 13 — all internal. The stack includes dockerd, containerd, socat bridging SSH agent forwarding, and the coding agent. Host processes completely invisible. No way to inspect or interact with anything running on the host. 5. Private Docker Engine Shell docker info | grep -E "Server Version|Operating System|ID" # Server Version: 29.4.3 # Operating System: Ubuntu 25.10 (containerized) # ID: e6934b23-368c-4259-a873-96f879f587e5 Ubuntu 25.10. A unique daemon ID that differs from docker info on the host — confirming the sandbox runs a fully isolated daemon. The agent deployed a full Kubernetes cluster using this daemon. No path to the host Docker socket existed. 6. Host Services Unreachable Shell curl -s --max-time 3 https://localhost:6443 2>&1 || echo "blocked" # curl: (7) Failed to connect to localhost port 6443: Connection refused Port 6443 — my minikube cluster on the Mac host. From inside the sandbox, localhost is the sandbox's own loopback. Host clusters, host SSH, host services — unreachable by default. Eight AKS contexts on this machine. Zero is reachable from inside the sandbox without an explicit policy rule. 7. What the Agent Had vs. What It Didn't During the entire debugging task, the agent had full access to one project directory, kubectl to the sandbox-internal Kubernetes cluster, and full Docker capabilities against the private daemon. It could not reach any other directory, cloud credentials, other kubeconfig contexts, the host Docker daemon, or any cluster not running inside the sandbox. All seven proofs held throughout the session without exception. Three Network Policy Findings That Change How You Think About It Network policy is not a full network control plane. It is hostname-scoped HTTP filtering. Three findings define the actual scope: Finding 1: Blocking returns HTTP 403, not TCP rejection. Plain Text probe "example.com" "https://example.com" # example.com | exit=0 | http=403 Exit code 0. The curl command succeeded. The proxy returned 403 directly. An agent that retries on 403 will retry blocked requests indefinitely. It cannot distinguish a blocked domain from a legitimate server-side error by exit code. For DevOps workflows — an agent hitting a blocked container registry will keep retrying silently rather than failing fast. Finding 2: HTTP CONNECT established a tunnel to port 22 on an allowed host. Plain Text # Port 22 — SSH port curl -s --max-time 5 telnet://github.com:22 # Connected to github.com port 22 # Port 9999 — non-standard port curl -s --max-time 5 telnet://github.com:9999 # Connected to github.com port 9999 github.com is on the Balanced allowlist. HTTP CONNECT established TCP tunnels to github.com on both port 22 and the non-standard port 9999 — both succeeded. Port-based restrictions are not enforced at the proxy layer. The Balanced policy is hostname-scoped only. Any port to an allowed host is reachable via HTTP CONNECT. Finding 3: DNS is not filtered. A common assumption is that all outbound traffic routes through the HTTP proxy — including DNS. Lab results show DNS resolution occurs independently: Plain Text dig example.com +short # 172.66.147.243 A blocked domain resolved. The microVM has an internal stub resolver that forwards DNS independently of the HTTP proxy. An agent can resolve any hostname regardless of the active policy. DNS cannot serve as a secondary enforcement layer. These findings do not break the isolation model. They define its actual boundary. Network policy controls HTTP/HTTPS access by hostname. It does not control DNS, TCP tunnels to allowed hosts on arbitrary ports, or how agents interpret 403 responses. The Agent Scenario: Isolation Under Real Load The real test of isolation is not seven probe commands — it is whether the boundary holds while an agent is actively working, making API calls, running kubectl, deploying containers. I gave an AI agent a broken Kubernetes deployment: a payments-service with memory limits set to 64Mi on a service that needs ~150Mi at peak. The agent received a task file and a set of manifests. No other context. The agent completed the task in under five minutes. It found two bugs — one planted, one discovered independently by reading the manifest and noticing health check probes targeting port 8080 on an nginx container that only serves on port 80. The task said nothing about probes. Result: both pods 1/1 Running, 0 restarts. The seven isolation proofs above were verified immediately after — throughout the entire debugging session, the boundary held without exception. Full article and complete repo at opscart.com/docker-sandbox-devops. What This Means for DevOps Engineers Specifically Most Docker Sandbox articles target software developers running Claude Code on a single codebase. The DevOps case is different and more demanding. A DevOps engineer running an AI agent faces a broader attack surface: multiple cluster contexts, infrastructure credentials, IAM roles, service accounts, kubeconfigs that grant production access. The blast radius of a compromised or manipulated agent is not one repo — it is potentially every system those credentials touch. Docker Sandbox addresses this at the architecture level rather than the prompt level. You are not relying on the agent being well-behaved. You are relying on the microVM boundary, the proxy, and the private Docker daemon. The agent can be fully autonomous inside the sandbox because the guardrail is the environment, not the agent's behavior. The private Docker Engine is particularly significant. DevOps agents need to build and test containers. Every other local isolation approach that allows container operations requires socket mounting — which gives the agent direct access to the host Docker daemon and every image and volume on the host. Docker Sandbox eliminates this tradeoff. What Is Still Rough The image iteration cycle is the primary friction point. Adding a tool requires editing a Dockerfile, rebuilding, pushing to a registry, and recreating the sandbox. For a stable toolchain, this is acceptable. For rapid experimentation, it is not. The --branch parallel agent mode is Git isolation, not VM isolation. Both agents run in one microVM with shared Docker and network. For separate credentials or separate network policies per agent, you need separate workspace directories. The network policy CLI has non-obvious syntax in several places — sbx policy deny does not remove an allow rule, and external cluster access requires two policy rules not one. Neither behavior is documented. The CLI changes between minor versions. v0.31.1 changed login flow, renamed policy tiers, and introduced --clone mode. Pin your version. When Not to Use Docker Sandbox Docker Sandbox is the right tool for a specific set of problems. It is not the right tool when: You need raw UDP or ICMP. Network tracing tools (traceroute, mtr), some mTLS configurations, and anything relying on ICMP will not work — the sandbox proxy only handles HTTP/HTTPS. Your toolchain requires host-device access. USB devices, GPU passthrough beyond basic forwarding, and hardware security keys are not accessible from inside the microVM. You are on a memory-constrained machine. Each sandbox runs a full microVM plus its own Docker daemon. On a machine with 8GB RAM, running multiple sandboxes simultaneously alongside Docker Desktop and a browser will cause pressure. You need production-grade audit logging. Docker Sandbox is Experimental. Audit trails, compliance logging, and enterprise controls are not mature yet. For regulated environments, evaluate accordingly. Your agent needs to coordinate across multiple repositories simultaneously. The one-sandbox-per-workspace model means cross-repo agent work requires careful orchestration. The --clone mode helps but adds git workflow overhead. Conclusion The credential isolation model is the headline: the agent made authenticated API calls throughout the session without the API key ever entering the sandbox. Authentication was performed by the host-side proxy using credentials stored outside the sandbox boundary. The agent could use the credential — it could never see, copy, or exfiltrate it. Seven isolation proofs confirmed the boundary held under real active load. One directory visible. No credentials. No host processes. No host clusters. No host Docker daemon. The network policy findings add important nuance. The --branch mode reality is different from what the documentation implies. Docker Sandbox is Experimental, and the CLI is moving. Use it knowing what it is — and what it is not.

By Shamsher Khan

CORE

The Rise of Microservices Architecture in Scalable Applications

In recent years, building modern applications has changed from what has been seen historically. Usually, in the past, systems were developed with a single, large block of code (referred to as a monolithic design) and would operate fairly well for smaller applications, but with time, as they got larger and more complex, the method of writing software became more of a hindrance to the applications as they required more users and increased speed. Now, companies need their applications to be able to grow quickly, adapt to changes quickly, and be able to support millions of users without any impact on performance, and that is where microservice architecture is so relevant. Microservice architecture has become the way to design scalable applications because applications can be broken into smaller, individual services that can work independently from each other. The trend towards microservice architecture in developing applications that can scale indicates to me that there is a shift in value towards being flexible, quick, and resilient in the highly competitive digital environment we live in today. What Is Microservices Architecture? Microservice architecture is a method of designing an application as a set of distinct parts that operate independently and perform specific tasks. Each microservice communicates with the others via APIs. With a microservice architecture, as opposed to a traditional monolithic system where all of the application’s components are dependent upon one another, developers can modify/update/deploy/scale a single microservice without impacting any of the other microservices in the application. In an e-commerce application, the components include user authentication, product catalog, payment processing, and order processing (each of these services exists as a microservice). Why Microservices Are Gaining Popularity Microservices are more than just a trend; they are the answer to increased demands for scalable, flexible, and high-performing applications. As digital-first business models grow, traditional architectures simply can't keep up, driving a preference for microservices. 1. Scalability Requirements Modern applications often deal with unpredictable user traffic, especially during peak times such as high-volume sales, new product launches, or virally driven surges in user traffic. In a monolithic architecture, scaling means replicating your entire application on expensive resources over a long period, which is inefficient. 2. Quick Development Cycles With the rapid pace of change in the marketplace, speed is key to success in competitive industries today. The use of a microservices architecture enables development teams to develop different services simultaneously without affecting one another’s progress. 3. Technology Flexibility The flexibility of technology is one of the greatest benefits of microservices architecture. Unlike Monolithic systems that typically use only one tech stack, each microservice can be built using the best programming language, framework, or database. For example, a data-intensive microservice can use a high-performance programming language as its primary language, while the UI microservice can use a more flexible front-end framework. 4. Enhanced Fault Containment Failure is a fact of life for big programs. What you do when it happens can make a difference. In a monolithic program, a single bug or failure can shut down an entire application. Microservices provide better fault containment by isolating faults to independent services. When an individual service fails, the failure won't automatically affect the rest of the program. This results in higher overall system availability and an improved user experience. 5. Agreement With DevOps Microservices architecture aligns well with DevOps practices, which focus on automation, collaboration, and continuous delivery. With microservices, teams can develop CI/CD pipelines for each of their services so that they can deploy frequently and reliably. Automated testing, monitoring, and deployment allow them to release updates efficiently with minimal risk. The Benefits of Microservice Architecture The rise in popularity of microservices aligns with current trends in the enterprise landscape; however, many organizations are beginning to realize significant value in microservice architecture for application development and performance. Through the use of microservices, an organization can break down large, complex systems into smaller parts (components). By creating applications using smaller components or microservices, organizations can develop highly scalable, resilient, and efficient systems. 1. Services Can Be Deployed Independently Deployment of one or more services can occur independently using a microservices-based architecture. In traditional applications, deploying even a small change would require deploying the entire application (which could take a long time and add significant risk). 2. Improved Scalability Since microservices inherently have scalability as a key design feature, software development companies can concentrate just on scaling those parts of their applications that require more resources rather than scaling an entire application as was done with Monolith-type applications. 3. Greater Agility Agility is extremely important in today’s digital market that changes rapidly. By allowing multiple teams that consist of members from different functional areas to independently develop their own services using microservices, microservices allow us to increase development speed and decision-making speed. 4. Easier to Manage Codebase It is common for large codebases to become challenging to manage over time. One of the advantages of using a microservices architecture is the ability to create smaller codebases that can be easily managed. 5. Increased Reliability Reliability is one of the most important aspects of any system, especially those with a large number of users. Microservices can help improve reliability by isolating faults between services. Conclusion The increase in the use of microservice architectures within scalable applications has led to a change of focus to properly design systems that are flexible, durable, and can grow with an organization's business needs. By breaking down large, complex applications into smaller independent services, organizations can take advantage of better speed of development, increased scalability, and greater reliability of their systems. Although there are some challenges associated with implementing microservices, the long-term benefits will more than justify any upfront investment required to adopt this architectural style in a modern enterprise. Businesses that have a plan, the right tools, and the right people can quickly realize the full benefits of the microservice architecture while providing high-quality digital experiences to their customers.

By Mitchell Jhonson

Runtime Formula Evaluation With MVEL Library in Spring Boot

In our software development processes, business units constantly want to update discount rates, loyalty points, or salary calculation logic. If this logic is within the code, between when-or-if-else blocks, every change means a new unit test process, code analysis, CI/CD pipeline work, and ultimately a "deployment." In this article, we will separate the business logic from the code, making it manageable in the database and reliably interpretable at runtime. By increasing flexibility, we will ensure the system's stable operation continues without interruption. To do all this, we will examine how to use the MVEL (MVFLEX Expression Language) library below. The Cost of Static Code: Why Should We Avoid It? Generally, point calculations are as follows: Kotlin fun calculatePoints(pointType: String, factor: Int): Long { return when (pointType) { "INITIAL" -> 100L "BIRTHDAY" -> 50L "TENURE_5_10" -> factor * 10L "TENURE_10_20" -> factor * 20L "TENURE_20_PLUS" -> factor * 30L else -> 0L } } When looking at the code, what appears is more of a maintenance burden than a simple function. If the factors change or a new rule is added, the code is triggered from the beginning. However, these values are actually data, not code. Architectural Approach Below, you will find how it works when we add the Formula engine. Kotlin import org.mvel2.MVEL val formula = "factor * 20" val vars = mapOf("factor" to 5) val result = MVEL.eval(formula, vars) In this architecture, the code does not know "how to calculate"; It only knows how to call the 'Formula engine.' Database Design Converting Rules to Data We can store business rules in a flexible table. This ensures manageability. PLSQL CREATE TABLE t_point_type ( point_type_id NUMBER PRIMARY KEY, point_type_name VARCHAR2(100), point_formula VARCHAR2(500), description VARCHAR2(1000) ); Sample data: Plain Text | point_type_id | point_type_name | point_formula | |:-------------:|:---------------:|:-----------------------:| | 1 | INITIAL | `100` | | 2 | BIRTHDAY | `50` | | 3 | TENURE_5_10 | `factor * 10` | | 4 | TENURE_10_20 | `factor * 20` | | 5 | TENURE_20_PLUS | `factor * 30` | | 6 | PROMOTIONAL | `factor * multiplier` | Application Layer The most critical point to consider in MVEL integration is performance and error management. 1. Entity Definition Kotlin @Entity @Table(name = "t_point_type") data class PointTypeEntity( @Id @Column(name = "point_type_id") val pointTypeId: Long? = null, @Column(name = "point_type_name") val pointTypeName: String? = null, @Column(name = "point_formula") val pointFormula: String? = null ) 2. MvelUtil: Performance-Oriented Helper Class Considering the CPU cost of parsing strings in every request, we should use compiled expressions and caching mechanisms. Kotlin @Component class MvelUtil { fun evaluateFormula(formula: String, factor: Int): Long { return try { val variables = mapOf("factor" to factor) val result = MVEL.eval(formula, variables) when (result) { is Number -> result.toLong() else -> 0L } } catch (e: Exception) { throw BusinessException( errorCode = ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED, errorDesc = "Formula evaluation failed: $formula, factor: $factor — ${e.message}" ) } } fun evaluateFormulaAsString(formula: String, factor: Int): String { return try { val variables = mapOf("factor" to factor) MVEL.eval(formula, variables).toString() } catch (e: Exception) { throw BusinessException( errorCode = ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED, errorDesc = "Formula evaluation failed.: $formula — ${e.message}" ) } } } 3. Service Layer and Business Logic Therefore, our service layer simply receives the data and triggers the formula engine. Kotlin @Service class PointCalculationService( private val pointTypeRepository: PointTypeRepository, private val mvelUtil: MvelUtil ) { fun calculatePoints(pointTypeId: Long, factor: Int): Long { val pointType = pointTypeRepository.findById(pointTypeId) .orElseThrow { BusinessException(ErrorCodes.POINT_TYPE_NOT_FOUND) } val formula = pointType.pointFormula ?: throw BusinessException(ErrorCodes.POINT_FORMULA_NOT_DEFINED) val points = mvelUtil.evaluateFormula(formula, factor) if (points <= 0) { log.info("The formula gave a score of 0 or negative: type=$pointTypeId, factor=$factor") return 0L } return points } } Call service: Kotlin val factor = inputData.factorSpecificForPoint ?: 1 val points = calculatePoints(inputData.pointTypeId, factor) if (points > 0) { savePointDetail(points, subscriptionId, inputData.pointTypeId, inputData.operationId) } Advanced Usage: Multivariable and Conditional Formulas MVEL has the ability to decode complex strings. Its true power lies in this. For example, the formula in the database might look like this: SQL UPDATE t_point_type SET point_formula = 'factor * multiplier + bonus' WHERE point_type_id = 6; Kotlin fun evaluateWithMultipleVars(formula: String, vars: Map<String, Any>): Long { return try { val result = MVEL.eval(formula, vars) (result as? Number)?.toLong() ?: 0L } catch (e: Exception) { throw BusinessException(ErrorCodes.MVEL_FORMULA_EVALUATION_FAILED) } } val vars = mapOf("factor" to 5, "multiplier" to 3, "bonus" to 10) evaluateWithMultipleVars("factor * multiplier + bonus", vars) Conditional Statements MVEL supports ternary expressions and Boolean logic: Plain Text factor > 10 ? factor * 20 : factor * 10 (factor >= 5 && factor < 10) ? 50 : (factor >= 10 ? 100 : 25) This provides truly dynamic rules without any code changes. We must not ignore these three rules, as everything is necessary; Strict validation: The formula must be validated with MVEL.compileExpression() before being saved to the database. An incorrect syntax error can disrupt the entire flow at runtime.Sandbox and security: MVEL is robust; it can access Java classes. Therefore, formula entry should only be done from authorized (admin) panels, and if necessary, MVEL's secure mode should be configured.Default value: There can always be a fallback mechanism. We determine how the system will behave if the formula receives an error or the result returns null (e.g., 0 points). Conclusion MVEL makes it easy for us to dynamically implement business rules in Spring Boot projects. It reduces code complexity while allowing you to respond to business unit requests within minutes (without deployment!). XML Dependency (Maven): XML <dependency> <groupId>org.mvel</groupId> <artifactId>mvel2</artifactId> <version>2.5.0.Final</version> </dependency>

By Erkin Karanlık

Parallel Kafka Batch Processing With Kotlin Coroutines in Spring Boot

Managing high-volume message traffic in distributed architectures is crucial. Efficient use of database and CPU resources is also very important. There are structures that allow us to receive messages in batches. The default Spring Kafka "BatchMessageListener" structure addresses this need. However, the processing of these messages often goes through a sequential bottleneck. This article will discuss the structure and usage of Kotlin Coroutines in detail. We will examine how to maximize Kafka message processing performance using Structured Concurrency principles and Resource Throttling techniques. Architectural Bottleneck: Sequential I/O Blocking On the current Kafka listener: Database or external service calls made for each message directly increase total processing times. If the processing speed of a message lags behind the message arrival speed and the max-poll-interval-ms time is exceeded, the consumer is removed from the consumer group. Rebalancing is triggered, and the partitions of that consumer are redistributed to other consumers in the group. Kotlin @KafkaListener(topics = ["usage-pool-topic"]) fun usagePoolListener(records: List<ConsumerRecord<String, String>>) { records.forEach { record -> processRecord(record) // Network latency + DB I/O blocking } } Solution 1. Batch-Fetch and In-Memory Map Structure Before any concurrent code is entered, data is retrieved collectively from all necessary entities. Multiple separate queries are converted into a batch query before data processing begins. The N+1 query problem is solved at the application layer. All data is cached once before being broken down into concurrent operations. Having the data cached significantly reduces our reliance on the database. Using the associateBy function, we transform the data into a map structure with X access times. This allows us to read the data safely from the maps instead of reading each concurrent operation from the database. Kotlin val messages = records.map { objectMapper.readValue(it.value(), UsagePoolRecord::class.java) } val usagePoolEntities = usagePoolRepository .findByIds(messages.map { it.usagePoolId.toBigInteger() }) .associateBy { it.usagePoolId } val lockEntities = lockRepository .findByUserIds(messages.map { it.userId }) .associateBy { it.userId } 2. Structured Concurrency Memory Management With Chunking The chunk structure serves two purposes. It prevents the creation of coroutines simultaneously. This prevents unnecessary memory usage. Each chunk writes to the database after all coroutines have completed their operations. Unnecessary connection pool consumption is avoided. Kotlin messages.chunked(150).forEach { chunk -> // Each chunk of 150 records is processed concurrently } Resource Isolation With limitedParallelism Why limitedParallelism? If the database connection pool has, for example, X connections, keeping the parallelism limit below X prevents "Connection Timeout" errors. Kotlin messages.chunked(150).forEach { chunk -> val deferredResults = chunk.map { record -> CoroutineScope(Dispatchers.IO.limitedParallelism(15)).async { try { processRecord(record, usagePoolEntities, lockEntities) } catch (e: Exception) { log.error("Operation error: ${record.key()}", e) buildErrorRecord(record, e) } } } val results = deferredResults.awaitAll() // Structural waiting collectAndAggregate(results) } The Dispatchers.IO.limitedParallelism(X) command limits the number of concurrent coroutines to X, preventing the DB connection pool from being exhausted.Each coroutine returns a result with the async command. The awaitAll() command waits for all coroutines in the chunk to finish before proceeding to the next step. runBlocking This function blocks callers until all concurrent operations are complete. This is the correct approach here because: It ensures that the Kafka consumer remains blocked to maintain its offset commit structure until all records in the batch are processed. We still benefit from concurrent operation parallelism within the runBlocking block. 3. Thread-Safe Result Structure After the awaitAll() operation, all results are collected in thread-safe queues. Then a single batch write operation takes place. Using MutableList structures to combine results returned from parallel processed coroutines can lead to data loss. At this point, lock-free data structures should be preferred. ConcurrentLinkedQueue uses CAS (Compare-And-Swap) algorithms instead of synchronized blocks. This provides superior performance in high-content write operations. Why Shouldn't We Use ConcurrentLinkedQueue? Concurrent operations (concurrent functions) perform simultaneous write operations to a shared collection of results. Using MutableList leads to race conditions. It performs well in secure and concurrent write operations. Kotlin data class AggregatedRecords( val processedSave: ConcurrentLinkedQueue<ProcessedEntity> = ConcurrentLinkedQueue(), val toDelete: ConcurrentLinkedQueue<UsagePoolEntity> = ConcurrentLinkedQueue(), val retryQueue: ConcurrentLinkedQueue<RetryEntity> = ConcurrentLinkedQueue() ) The DataIntegrityViolationException return is important. When two consumer instances are processing the same record, one of them falls into a unique constraint violation. Instead of making the entire batch fail, record-by-record deletion is performed. Kotlin AggregatedRecords.processedSave .chunked(150) .forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { batch.forEach { record -> try { processedRepository.save(record) } catch (e: DataIntegrityViolationException) {} } } } 4. Error Tolerance in Write Operations Batch write (saveAll) operations are performant. However, a "Unique Constraint" error in a single record can cause the entire batch to fail. The following structure is critical to meet Optimistic Locking or Idempotency requirements. Kotlin aggregatedRecords.processedSave.chunked(150).forEach { batch -> try { processedRepository.saveAll(batch) } catch (e: DataIntegrityViolationException) { // Fallback: Try one by one if batch fails batch.forEach { record -> try { processedRepository.save(record) } catch (innerException: DataIntegrityViolationException) { log.warn("Duplicate record skipped: ${record.id}") } } } } 5. Data Flow Diagram Ingress: The Kafka batch is caught with runBlocking.Preparation: All necessary context data is retrieved bulk from the DB.Execution: Coroutines are started asynchronously in chunks.Synchronization: The completion of all coroutines is awaited as a barrier point with awaitAll().Egress: Collected results are made permanent with saveAll. Performance Analysis and Results Conclusion Processing Kafka messages in Spring Boot with Kotlin Coroutines not only increases speed but also improves code readability and makes resource management deterministic (predictable). The use of runBlocking allows us to build a bridge between the blocking Kafka consumer thread and the suspended world without disrupting Kafka's offset management mechanism. Dependencies XML <dependency> <groupId>org.jetbrains.kotlinx</groupId> <artifactId>kotlinx-coroutines-core</artifactId> <version>1.7.3</version> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency>

By Erkin Karanlık

Building a Multi-Agent Orchestration Capability: Architecture and Code Walkthrough

Artificial intelligence (AI) is quickly changing from simple conversation models to systems that can tackle complex problems through teamwork. As products become smarter, one key approach that is gaining traction today is multi-agent orchestration. A single AI model can handle straightforward tasks like answering questions or generating content. Yet, modern product features increasingly need: Multi-step reasoningSpecialized expertiseTool integrationsDynamic decision makingExecution of actionsContinuous feedback Trying to manage all of these with one model often leads to complexity, decreased accuracy, and limited growth potential. Multi-agent orchestration solves these issues by establishing a system where multiple specialized agents work together within a coordinated framework. This article explains how to create a general multi-agent orchestration capability and shows a practical example using code. One example we could think of, where we would require multi-agent orchestration, is: Intelligent Travel Assistant A user says, "Plan my trip to New York for three days under $1500." An intent agent understands the needs. The weather agent checks for the best time to visit. A search agent finds flights and hotels. A planning agent creates the itinerary while an execution agent makes the reservations. All this happens without the user being aware of it in the backend. Understanding the Architecture A multi-agent system generally consists of four major components: Agents Agents are specialized AI units designed for particular responsibilities. Examples: Intent agent, planning agent, search agent, recommendation agent, execution agent, and validation agent Tools Agents require access to external systems. Examples: APIs, databases, search engines, knowledge repositories, and workflow systems Shared Context Agents need access to common information: Python { "user":"User", "goal":"Recommend an action", "history":[], "constraints":[] } This prevents agents from operating independently without awareness. Orchestration Layer The orchestrator acts as the central coordinator. Responsibilities include: Task decompositionAgent selectionContext managementWorkflow executionResult aggregation The orchestrator acts as the "brain." Example: Suppose users interact with a product capability using: "Help me find and recommend the best option based on my needs." The workflow might involve "Understand user intent", "Retrieve information", "Analyze findings", "Generate recommendations," and "Execute actions." Step 1: Define Base Agent Structure Create a generic agent abstraction. Python from abc import ABC, abstractmethod class Agent(ABC): @abstractmethod def execute(self, context): pass All agents inherit from this class. Step 2: Create Specialized Agents Intent Agent Responsible for understanding user objectives. Python class IntentAgent(Agent): def execute(self, context): query=context["query"] print("Intent Agent running...") context["intent"]=f"Intent identified from: {query}" return context Search Agent Responsible for retrieving information. Python class SearchAgent(Agent): def execute(self, context): print("Search Agent running...") context["results"]=[ "Option A", "Option B", "Option C" ] return context Recommendation Agent Generates recommendations. Python class RecommendationAgent(Agent): def execute(self, context): print("Recommendation Agent running...") recommendations=context["results"][:2] context["recommendations"]=recommendations return context Step 3: Create Tool Integrations Tools provide external capabilities. Example: Python class SearchTool: def search(self,query): return [ "Data 1", "Data 2", "Data 3" ] Modify the search agent to use tools. Python class SearchAgent(Agent): def __init__(self): self.tool=SearchTool() def execute(self,context): query=context["query"] data=self.tool.search(query) context["results"]=data return context Agents now become capable of interacting with external systems. Step 4: Build the Orchestrator The orchestrator coordinates the execution flow. Python class Orchestrator: def __init__(self): self.agents=[ IntentAgent(), SearchAgent(), RecommendationAgent() ] def run(self,query): context={ "query":query } for agent in self.agents: context=agent.execute(context) return context Step 5: Execute the Workflow Run the orchestration system. Python orchestrator=Orchestrator() response=orchestrator.run( "Recommend something useful" ) print(response) Output: Python Intent Agent running... Search Agent running... Recommendation Agent running... { 'query': 'Recommend something useful', 'intent': 'Intent identified from: Recommend something useful', 'results':[ 'Data 1', 'Data 2', 'Data 3' ], 'recommendations':[ 'Data 1', 'Data 2' ] } The user sees a single interaction, while multiple agents collaborate behind the scenes. Adding Dynamic Agent Selection Real systems should not execute every agent for every request. The orchestrator can dynamically decide which agents participate. Example: Python class DynamicOrchestrator: def get_agents(self,query): agents=[IntentAgent()] if "search" in query: agents.append(SearchAgent()) if "recommend" in query: agents.append( RecommendationAgent() ) return agents def run(self,query): context={ "query":query } agents=self.get_agents(query) for agent in agents: context=agent.execute( context ) return context Now execution becomes adaptive. Parallel Execution Many tasks can run simultaneously. Python supports parallel processing: Python from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor() as executor: futures=[] futures.append( executor.submit( searchAgent.execute, context ) ) futures.append( executor.submit( recommendationAgent.execute, context ) ) results=[ f.result() for f in futures ] Parallelism significantly reduces latency. All in all, multi-agent orchestration marks a significant shift in how intelligent systems are designed and operated. As product capabilities evolve from separate interactions to complex, goal-driven workflows, depending on a single AI component becomes harder to scale and maintain. Sharing responsibilities among specialized agents leads to systems that are more modular, flexible, and able to handle complicated reasoning and execution patterns. From an engineering viewpoint, the real benefit goes beyond just connecting multiple models. Success relies on creating a strong orchestration layer that can manage context, route tasks wisely, integrate with tools, coordinate workflows, and monitor the entire execution process. Production-grade systems must also tackle important issues like state management, fault tolerance, security boundaries, minimizing latency, and controlling costs. The future of AI-powered products will probably look more like distributed systems than traditional applications. Just as microservices changed software architecture by breaking down monolithic systems into specialized services, multi-agent orchestration is bringing a similar change for intelligent systems by separating generalized intelligence into collaborative, specialized abilities. Organizations that focus on building strong orchestration capabilities now are not just adding AI features; they are laying the groundwork for adaptable systems that can understand goals, coordinate actions, and consistently deliver valuable results at scale.

By Narendra Lakshmana gowda

Microservices

DZone's Featured Microservices Resources

Top Microservices Experts

The Latest Microservices Topics