Performance Resources

DZone's Featured Performance Resources

How to Contain Partial Failures in Distributed Systems

Distributed systems rarely fail as a single unit. More often, one dependency slows down, one shard becomes unavailable, one region loses capacity, or one pool saturates while neighboring components remain healthy. The engineering challenge is not simply availability; it is preventing a local defect from becoming a system-wide event. Google’s SRE guidance describes cascading failure as positive feedback in which lost capacity shifts load onto surviving capacity until additional components fail. Resilience design must break that loop early, cheaply, and at a boundary narrow enough to preserve useful work elsewhere. Containment Starts With Explicit Failure Domains A failure-containment framework begins by defining what is allowed to fail independently. Service names are usually too coarse. A single service may contain independent regions, tenants, shards, endpoint families, connection pools, or workload classes. Circuit breakers, concurrency limits, queues, and caches should align with those real failure domains rather than an arbitrary process boundary. Microsoft’s circuit-breaker guidance warns against combining independent resources, such as multiple data shards, behind one breaker because failure in one resource can incorrectly block healthy resources. This changes the design question from whether a service is healthy to which partition, operation, or dependency is unhealthy and what scope must be isolated. A breaker keyed by destination shard can contain a hot partition. A bulkhead keyed by traffic class can prevent batch work from consuming interactive capacity. A separate connection pool for an optional dependency can stop socket exhaustion there from starving critical calls. Resilience4j reflects this separation: circuit breakers control whether calls are permitted based on recent failures or slow calls, while bulkheads independently cap concurrent executions. Time Is the First Resource Budget Partial failure often presents as latency before an explicit error. A caller that waits indefinitely converts remote slowness into local resource retention: threads remain occupied, queues grow, memory pressure increases, and upstream deadlines begin expiring. gRPC recommends explicit realistic deadlines, and Google SRE recommends propagating the original request deadline through downstream RPCs so deeper services do not continue work after the result has become useless. A downstream timeout should fit inside the remaining end-to-end latency budget rather than being configured independently at every hop. For a request with 800 milliseconds remaining, a dependency call cannot safely claim a fresh two-second timeout. Deadline propagation preserves the original budget and cancellation propagation stops abandoned fan-out work. A concise Java boundary can make the budget explicit: Java public Inventory reserve(ReserveRequest request, long remainingMillis) { return inventoryStub .withDeadlineAfter(Math.min(remainingMillis, 250), TimeUnit.MILLISECONDS) .reserve(request); } The 250-millisecond cap is dependency-specific rather than universal. The containment rule is broader: downstream work must not outlive the request that justified it. Retries Must Consume a Bounded Failure Budget Retries are useful only for failures likely to be transient. Unbounded or synchronized retries create extra traffic precisely when capacity is impaired. Google documents how retries can amplify overload into cascading failure and recommends randomized exponential backoff, per-request retry limits, and retry budgets. AWS similarly recommends backoff for throttling, temporary network loss, and temporary service unavailability, while warning that retried operations should be idempotent to avoid duplicated state changes. A retry policy belongs inside the same time and load budget as the original call. It should distinguish transient statuses from permanent failures, leave enough deadline for another attempt, and stop when aggregate retry pressure becomes unsafe. gRPC implements configurable attempt limits, exponential backoff, jitter, retryable status codes, and retry throttling. Java @Retry(name = "catalog") @CircuitBreaker(name = "catalog", fallbackMethod = "catalogFallback") public ProductView loadProduct(String productId) { return catalogClient.fetch(productId); } private ProductView catalogFallback(String productId, Exception ex) { return ProductView.withoutCatalogEnrichment(productId); } The fallback is intentionally degraded rather than deceptive. Noncritical enrichment can disappear while the core response remains valid. A failed payment authorization, by contrast, should not become a fabricated success. Failure containment preserves service by removing optional work, not by weakening correctness. Resilience4j’s Spring integration supports composed retry, circuit-breaker, bulkhead, and fallback annotations for this boundary pattern. Admission Control Protects the Healthy Remainder Circuit breakers answer whether a dependency should receive calls; bulkheads answer how many calls may enter concurrently. Those controls solve different failure modes and should often be composed. Resilience4j notes that a closed circuit breaker does not limit concurrency, while its semaphore and fixed-thread-pool bulkheads provide explicit concurrency isolation. The next containment layer is admission control at saturation. Google SRE recommends early rejection, small queues where appropriate, load shedding, and graceful degradation because overloaded servers should preserve useful throughput rather than accept work until latency and resource exhaustion collapse the instance set. An HTTP 503 or equivalent overload signal can be healthier than a long queue that guarantees deadline misses. This creates a deliberate sequence under stress: optional work is removed, excess work is rejected, unhealthy dependencies are short-circuited, and concurrency remains partitioned. Capacity degrades in controlled steps instead of falling off a cliff. The design target is not zero errors, but a bounded error surface with stable latency and preserved capacity for higher-value operations. Recovery Behavior Matters as Much as Failure Detection Containment mechanisms can cause a second outage during recovery if suppressed traffic returns at once. The half-open circuit-breaker state probes recovery with limited traffic rather than immediately restoring full load. Microsoft describes this as protection against flooding a recovering service, while Resilience4j permits a configurable number of half-open trial calls before closing the breaker. Recovery also requires warm capacity. Google notes that restarted processes and cold caches can be more expensive than steady-state instances, so traffic should often be restored gradually. The same principle applies after regional failover, database restart, or large deployment: recovery is a controlled ramp, not a binary switch. Observability should expose breaker transitions, rejected bulkhead calls, timeout rates, retries, queue depth, saturation, and degraded responses. Microsoft recommends circuit-breaker monitoring and distributed tracing, while gRPC exposes retry-attempt metrics. Containment Is Proven Under Pressure A failure-containment framework is incomplete until overload and dependency failure are tested beyond the normal operating envelope. Google recommends load testing components until failure, observing whether excess demand produces bounded rejection instead of crashes, and testing recovery after load is reduced. It also recommends testing abrupt capacity loss and backend blackholing because gradual load tests alone can miss cascade behavior. The strongest distributed systems are not those that assume every dependency remains available; they are those that define exactly how much time, concurrency, retry traffic, and functional degradation each dependency may consume when it fails. Explicit failure domains, propagated deadlines, bounded retries, circuit breakers, bulkheads, load shedding, truthful fallbacks, controlled recovery, and failure testing form a single containment discipline. Partial failure then remains partial: damage stops at designed boundaries, healthy capacity continues doing useful work, and recovery proceeds without recreating the conditions that caused the incident. More

9 Practical Ways to Fix Laravel Performance Issues in Production

By Ankit Khoiwal

A Laravel application can feel instant on a developer laptop and painfully slow after a few months in production. The framework is rarely the real problem. More often, the database has grown, a relationship is being loaded inside a loop, a report pulls far more rows than anyone expected, or a third-party API has quietly become part of the request's critical path. If you are trying to learn how to fix Laravel performance issues in production, resist the urge to begin with Redis, Octane, or a larger server. First, find the work the request shouldn't be doing. Removing wasted work usually produces a bigger, safer improvement than adding infrastructure around it. Here is the production-focused sequence I would use. 1. Measure the Slow Request Before Changing Code “The dashboard is slow” is not a diagnosis. Pick one route, record its response time, database query count, cumulative SQL time, memory usage, and any external calls. Then reproduce it with production-like data. Laravel Telescope and Pulse can help during controlled investigation. An APM trace is even more useful when the request crosses PHP, Redis, MySQL, queues, and third-party services. For a narrow database check, listen for unusually slow queries: PHP use Illuminate\Database\Events\QueryExecuted; use Illuminate\Support\Facades\DB; DB::listen(function (QueryExecuted $query) { if ($query->time >= 200) { logger()->warning('Slow database query', [ 'sql' => $query->toRawSql(), 'time_ms' => $query->time, ]); } }); Keep this targeted. Logging every query on a busy server creates noise, consumes disk I/O, and may record values that should not live in application logs. One more detail matters: compare the same request before and after each change. Otherwise, a warm cache or quieter database can make an ineffective fix look successful. 2. Stop N+1 Queries Before They Multiply N+1 queries are easy to miss because the page still returns the correct data. This innocent-looking code can execute one order query followed by one customer query for every order: PHP $orders = Order::latest()->take(100)->get(); foreach ($orders as $order) { echo $order->customer->name; } Load the relationship deliberately: PHP $orders = Order::query() ->select(['id', 'customer_id', 'total', 'created_at']) ->with('customer:id,name') ->latest() ->take(100) ->get(); That usually turns 101 queries into two. It also avoids retrieving columns the response never uses. Remember to keep the primary and foreign keys Eloquent needs to match the models. For the edge cases involving Blade, API resources, accessors, withCount(), and regression tests, use this complete guide to detecting and fixing Laravel N+1 problems. Laravel's official Eloquent relationship documentation is a useful reference for constrained and nested eager loading. During development and CI, make accidental lazy loading visible: PHP use Illuminate\Database\Eloquent\Model; public function boot(): void { Model::preventLazyLoading(! app()->isProduction()); } Do not respond by eager-loading everything. Pulling thousands of related models into memory can replace a query problem with a hydration problem. 3. Fix the Query Plan, Not Just the Eloquent Syntax Once the query count is under control, inspect the slow SQL itself. A clean Eloquent chain can still force the database to scan a large table. Suppose a production report repeatedly filters paid orders by date: PHP Order::query() ->where('status', 'paid') ->whereBetween('created_at', [$from, $to]) ->orderByDesc('created_at') ->get(); A composite index may fit that access pattern: PHP Schema::table('orders', function (Blueprint $table) { $table->index(['status', 'created_at']); }); “May” is important. Run EXPLAIN with realistic values and inspect the chosen index, estimated rows, join order, and sorting work. The MySQL guide to query execution plans explains what the optimizer is showing you. Do not collect indexes as insurance. Every unnecessary index consumes storage and adds work to inserts and updates. 4. Stop Loading Entire Tables Into PHP Memory Exports and maintenance commands often work during testing and fail later because get() loads the complete result set. Process large tables in stable batches: PHP Order::query() ->where('status', 'pending') ->chunkById(500, function ($orders) { foreach ($orders as $order) { ProcessPendingOrder::dispatch($order->id); } }); Use paginate() for normal screens, cursorPaginate() for large ordered feeds, and lazyById() or chunkById() for background processing. A cursor is memory-friendly, but it is not a magic solution when every record needs a large relationship graph. 5. Cache Expensive Results With an Invalidation Plan Caching helps when the underlying result is expensive and safe to reuse. It does not make a bad query good. PHP $stats = Cache::remember( "organization:{$organizationId}:dashboard-stats", now()->addMinutes(10), fn () => $this->buildDashboardStats($organizationId) ); Before shipping this, answer three questions: What event makes the value stale? Who is allowed to see it? What happens when many requests rebuild it at once? Include tenant, permission, locale, and other relevant dimensions in the cache key. Forget or version the key when its source data changes. For a value that expires under heavy traffic, use a lock or stale-while-revalidate strategy to prevent a cache stampede. Laravel's cache documentation covers atomic locks and the available stores. On a multi-server application, use a shared store such as Redis. A local file cache gives each server a different answer. 6. Move Slow Work Out of the Web Request Email delivery, PDF generation, video processing, large imports, webhook fan-out, and search indexing normally belong in queues. The controller should validate the request, commit the required state, dispatch the work, and return. PHP GenerateInvoicePdf::dispatch($invoice->id)->afterCommit(); afterCommit() prevents a fast worker from reading records before the surrounding transaction is committed. Queues do not remove work; they move it. Monitor queue depth, wait time, run time, failed jobs, retries, and worker availability. A one-second job is still late when it waits 15 minutes behind an under-provisioned queue. When retries behave strangely, check the original connection, queue name, worker configuration, timeout, backoff, and retry_after values. This Laravel queue retry and failed-job troubleshooting guide walks through those production cases without turning this performance article into a queue manual. 7. Put Boundaries Around External APIs A payment, CRM, shipping, or AI service can make your Laravel route slow even when your own code is efficient. Set explicit connection and response timeouts: PHP $response = Http::connectTimeout(2) ->timeout(5) ->retry(2, 200) ->get($url); Retry only operations that are safe to repeat. For optional information, cache the last successful value or update it asynchronously. A controlled fallback is better than letting one unhealthy vendor consume every PHP worker. 8. Apply Laravel and PHP Production Optimizations A deployment should not leave Laravel rebuilding framework metadata on live requests. Install production dependencies and cache the application bootstrap files: Shell composer install --no-dev --prefer-dist --optimize-autoloader php artisan optimize The optimize command caches configuration, events, routes, and views. Laravel documents the individual commands and environment caveats in its official deployment optimization guide. Keep env() calls inside configuration files. Once configuration is cached, the .env file is not loaded in the normal way for direct calls elsewhere in the application. Production should also have: Plain Text APP_ENV=production APP_DEBUG=false Enable PHP OPcache and size it for the deployed codebase. OPcache reuses compiled PHP bytecode, but an undersized cache can churn and lose much of the benefit. The relevant settings are listed in the PHP OPcache configuration reference. Finally, reload long-running processes after deployment: Shell php artisan queue:restart php artisan octane:reload Run the Octane command only when the application uses Octane. Horizon-managed workers should be terminated through Horizon so the process monitor starts them with the new release. 9. Reduce Logging, Payload, and Serialization Waste Debug logging inside a loop can become surprisingly expensive. Keep logs structured and actionable, rotate them, and avoid recording large request or response bodies unless an investigation genuinely requires it. Look at the response too. API resources that append unused attributes, serialize deep relationships, or return thousands of records waste database time, PHP memory, network bandwidth, and client rendering time in one move. Smaller responses are often the cheapest performance win in the entire request. A Fast Production Diagnosis Order When you need to fix a slow Laravel application without making random changes, use this order: Trace one slow route with realistic data.Count queries and find repeated SQL shapes.Remove N+1 queries and unnecessary columns.Run EXPLAIN on the remaining slow statements.Paginate, stream, or chunk oversized results.Cache only stable, expensive values.Queue work the user does not need immediately.Add timeouts and fallbacks to external calls.Cache Laravel bootstrap files and enable OPcache.Repeat the same measurement and compare the result. Conclusion The reliable way to fix Laravel performance issues in production is to remove unnecessary work before adding more capacity. Keep query counts bounded, make the database's job index-friendly, control how much data enters PHP, and stop slow dependencies from occupying request workers indefinitely. Fix one measured bottleneck at a time. That keeps the change reviewable, proves which optimization worked, and prevents a rushed “performance pass” from becoming a collection of permanent workarounds. More

Moving 900+ GB of Prometheus Without a Cutover Weekend

By Daniil Romashov

Columnar Formats for AI Telemetry: Parquet vs Vortex With DuckDB

By Bhala Ranganathan

CORE

Consumer Lag Is the Wrong Finish Line

By Ishan Shah

Production-Grade LLM Regression Testing: Contracts, Evals, and Quality Gates

Conventional unit tests remain essential around an LLM application, but they do not answer the most important production question about whether a change preserved useful behavior. A parser can still return valid JSON while the answer becomes less grounded. A tool call can satisfy its schema while selecting the wrong tool. A retrieval pipeline can return documents successfully while omitting the evidence required for the final answer. This gap exists because an LLM application is not a deterministic function whose correctness can always be represented as actual == expected. Empirical work on code generation has shown substantial output variation across repeated calls, including at temperature zero, so single-run assertions can mistake sampling noise for either success or failure. Regression testing therefore has to evaluate behavior statistically and semantically, not only execution paths. Unit Tests Prove Contracts, Not Behavior The first layer should still look familiar. Deterministic properties deserve deterministic tests like JSON must parse, required fields must exist, tool arguments must conform to a schema, forbidden operations must remain blocked, and latency or token budgets can be checked numerically. Anthropic’s evaluation guidance explicitly separates success criteria from the mechanisms used to measure them and includes exact-match, code-based, human, and model-based grading as different options rather than treating one metric as universal. A contract test for a support-answering endpoint can remain intentionally narrow: Python def validate_contract(result): assert result["status"] in {"answered", "abstained"} assert isinstance(result["answer"], str) assert len(result["answer"]) <= 2_000 assert all(c["source_id"] for c in result["citations"]) Passing this function proves that the response is consumable by downstream software. It does not prove that the answer is correct, that every material claim is supported, or that an abstention occurred when evidence was missing. Those are behavioral properties and need evaluators matched to the application’s actual failure modes. Research on LLM-based judging also shows why a single generic "quality" score is insufficient as judges can exhibit position, verbosity, and self-enhancement biases, even though strong judges can correlate well with human preferences under controlled evaluation. Build the Regression Corpus From Failures A useful regression suite begins with representative tasks rather than prompts invented solely for testing. Production traces, manually verified examples, support escalations, retrieval misses, tool selection failures, malformed outputs, and previously fixed incidents should become durable evaluation cases. LangSmith’s current evaluation documentation describes this feedback pattern where production traces can be added to datasets so that a failure observed in live traffic becomes a repeatable offline test, while experiments compare application versions on the same dataset. Each case should preserve enough context to reproduce the behavior that matters. A question alone is often insufficient for RAG or agent systems because the retrieved documents, tool state, permissions, conversation history, and expected policy can affect the result. The expected value also should not always be a canonical sentence. A stronger case stores assertions about required facts, forbidden claims, acceptable citations, expected tool choices, and whether abstention is mandatory. Python case = { "input": "Can an expired license be renewed online?", "required_facts": {"renewal_window": "30 days"}, "forbidden_claims": {"automatic_extension"}, "must_cite": {"policy-17"}, "expected_action": "answer" } The corpus also needs slices. A global average can improve while an important category regresses. Long queries, multilingual requests, ambiguous requests, high-risk tool actions, low-retrieval-confidence cases, and specific product domains should retain explicit tags so candidate performance can be compared within those populations. Google’s production ML guidance similarly recommends monitoring real-world metrics and inspecting data slices because aggregate quality can conceal skew or deterioration in subgroups. Score Properties Instead of Strings Exact matching works for classifications, IDs, tool names, and other canonical outputs. Open-ended language requires property-based evaluation. For RAG, retrieval and generation should be measured separately. Ragas formalized this decomposition with metrics targeting retrieval quality, answer relevance, and faithfulness rather than collapsing the entire pipeline into one score. That separation matters operationally because an unsupported answer and a retrieval miss require different fixes. A production-like evaluator can combine deterministic checks with semantic scoring: Python def evaluate(case, result): return { "contract": contract_score(result), "groundedness": groundedness_score( result["answer"], result["evidence"] ), "task_quality": rubric_score( case["input"], result["answer"], case ), "tool_correctness": tool_score(case, result), } The evaluator should expose dimensions rather than immediately averaging them. A groundedness score of zero must not be hidden by excellent style. A wrong financial action must not pass because the explanation is fluent. Hard safety and contract requirements should act as vetoes, and softer qualities such as completeness or tone can be aggregated only after those constraints pass. LLM as a judge is useful when the desired property cannot be expressed with code, but the judge itself becomes part of the test infrastructure. G-Eval demonstrated that rubric-driven LLM evaluation can align better with human judgments than older reference-based metrics for some generation tasks, while later judge research documented systematic biases and sensitivity to evaluation setup. The practical consequence is straightforward with the judge model, rubric, prompt, sampling settings, and parser should be versioned like any other dependency. Borderline cases should be sampled repeatedly or routed to human review rather than converted into false precision by a single score. Gate Changes Against a Baseline, Then Close the Production Loop Regression testing is most informative when a candidate is compared with a pinned baseline on identical cases. Absolute thresholds such as "quality must exceed 0.85" can hide a meaningful drop from 0.94 to 0.86. A gate should therefore combine non-negotiable case failures with relative movement in quality, cost, and latency. Python def release_allowed(baseline, candidate): if candidate["critical_failures"] > 0: return False quality_delta = candidate["quality"] - baseline["quality"] latency_ratio = candidate["p95_ms"] / baseline["p95_ms"] return quality_delta >= -0.01 and latency_ratio <= 1.10 The tolerances in this example are application policy, not universal constants. More important is the comparison model where the baseline and candidate run against the same versioned corpus, results remain inspectable by case and slice, and repeated samples are used when output variance is material. Recent research on judge reproducibility reinforces this requirement, finding that temperature control can reduce variability without eliminating it and arguing that grader disagreement should be treated as an evaluation signal rather than ignored. CI can then apply evaluation at different depths. Pull requests can execute a compact, high-signal corpus covering critical behaviors, while scheduled or pre-release runs can execute broader datasets and repeated samples. Official LangSmith guidance supports offline evaluation for comparing versions before deployment and provides CI/CD integration patterns around evaluation runs. The important design principle is not the specific platform, as evaluation results must be release evidence rather than a dashboard consulted only after a failure. Offline evaluation still cannot fully represent production traffic. Query distributions change, documents change, tools return new states, and upstream services evolve. Production monitoring therefore completes the regression loop. Google’s production ML guidance emphasizes live quality measurement, drift detection, and ongoing monitoring rather than treating validation as a one-time pre-deployment event. Sampling real traces, reviewing low-confidence or high-impact outcomes, and promoting confirmed failures back into the regression corpus turns operational incidents into permanent test coverage. AI regression testing becomes reliable when deterministic software tests and behavioral evaluations are treated as complementary rather than interchangeable. Unit tests should enforce contracts and invariants, curated datasets should preserve real failure modes, evaluators should measure task-specific properties, baseline comparisons should detect meaningful degradation, and production traces should continuously refresh the suite. An LLM application is ready for release not when every generated sentence matches a fixture, but when measured behavior remains within explicit quality, safety, cost, and latency tolerances under representative conditions. That shift turns evaluation from an ad hoc prompt-checking exercise into a disciplined software engineering control for probabilistic systems.

By Uthej Mopathi

CORE

Building High-Performance Time-Series Applications With Java and QuestDB

QuestDB is well-suited for applications necessitating high-performance ingestion and SQL access to time-oriented data. It is especially useful in financial markets, real-time analytics, observability, telemetry, and operational systems where data arrives continuously and must be queried with low latency. Unlike general-purpose databases, QuestDB is purpose-built with time as a core element of both storage and query processes. QuestDB appeals to enterprise developers by combining a time-series architecture with a familiar SQL interface. This technique reduces the learning curve for teams experienced in relational querying while providing a database optimized for append-heavy, chronological workloads. For systems needing recent-state queries, historical analysis, trend detection, and rapid ingestion, QuestDB delivers a practical solution. Why QuestDB Matters QuestDB excels when applications require both high ingestion rates and fast analytical queries on recent and historical data. This is essential for workloads with continuous data streams where rapid business response is critical, such as market prices, trading activity, telemetry, infrastructure metrics, clickstream data, operational events, and real-time business indicators. In these cases, the database must efficiently manage append-heavy data while supporting intuitive SQL queries across time ranges. As a result, QuestDB is well suited for financial platforms, observability systems, logistics, energy, IoT, and real-time analytics. For example, a trading system can query both the latest prices and historical intervals, an operations platform can analyze recent latency and throughput, and a logistics application can review vehicle or delivery telemetry over time. Its SQL-oriented model is especially valuable for enterprise teams, offering a familiar query language alongside an architecture designed for time-series workloads. Hands-On: Java With QuestDB We will build a simple Java application using QuestDB as the time-series database. For development, the easiest way to begin is with Docker: Shell docker run -d \ --name questdb-instance \ -p 9000:9000 \ questdb/questdb:10.0.1 QuestDB provides its Web Console and HTTP endpoint on port 9000. Once the container is running, you can open http://localhost:9000 in your browser to view the data. Configuring the Application Eclipse JNoSQL uses Jakarta APIs such as CDI and JSON-B, along with Eclipse MicroProfile Config. These APIs are available in popular Java runtimes including Helidon, Quarkus, Payara, and Open Liberty. Add the QuestDB driver dependency: XML <dependency> <groupId>org.eclipse.jnosql.databases</groupId> <artifactId>jnosql-questdb</artifactId> <version>${jnosql.version}</version> </dependency> Next, configure the Time Series database and QuestDB connection in microprofile-config.properties: Properties files jnosql.timeseries.database=qdb jnosql.questdb.url=ws::addr=localhost:9000 The qdb value specifies the logical database for the JNoSQL Time Series manager. The QuestDB URL uses the connection format required by the driver. Modeling Sensor Data For this example, we will use a simple sensor reading model: Java @Entity public class SensorReading { @Id private Instant id; @Column private String sensor; @Column private double temperature; @Column private double humidity; // constructors, getters, and setters } The Instant field records the measurement time, while the other fields describe the sensor reading at that moment. You can also expose the entity through Jakarta Data: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensorOrderByIdDesc( String sensor, Limit limit); } Using TimeSeriesTemplate You can now insert a sequence of readings and query both the latest state and recent history: Java public class App { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { TimeSeriesTemplate template = container.select(TimeSeriesTemplate.class).get(); template.insert(firstReading); template.insert(secondReading); template.insert(latestReading); var currentReading = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .limit(1) .singleResult(); System.out.println( "Current sensor reading: " + currentReading ); var history = template .select(SensorReading.class) .where("sensor") .eq("sensor-01") .orderBy("id") .desc() .skip(1) .limit(10) .result(); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } The first query retrieves the latest reading for sensor-01. The second retrieves a limited history, skipping the most recent observation. This approach better fits typical time-series use cases than retrieving records by identifier. Using Jakarta Data You can achieve the same functionality using a Jakarta Data repository: Java public class App2 { public static void main(String[] args) { var firstReading = new SensorReading( Instant.parse("2026-09-20T08:00:00Z"), "sensor-01", 21.4, 45.0 ); var secondReading = new SensorReading( Instant.parse("2026-09-20T09:00:00Z"), "sensor-01", 22.1, 46.5 ); var latestReading = new SensorReading( Instant.parse("2026-09-20T10:15:00Z"), "sensor-01", 23.6, 48.2 ); try (SeContainer container = SeContainerInitializer.newInstance().initialize()) { SensorReadingRepository repository = container.select(SensorReadingRepository.class).get(); repository.save(firstReading); repository.save(secondReading); repository.save(latestReading); var currentReading = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.of(1) ) .stream() .findFirst(); System.out.println( "Current sensor reading: " + currentReading ); var history = repository .findBySensorOrderByIdDesc( "sensor-01", Limit.range(2, 10) ); System.out.println("Recent sensor history:"); history.forEach(System.out::println); } } } Notably, QuestDB’s time-series capabilities are available through the same Java programming model as other NoSQL time-series drivers. This lets the application focus on temporal queries such as latest state, ordering, and recent history, while the driver handles database-specific details. Why SQL Matters for QuestDB Adoption QuestDB stands out for its time-series specialization and SQL-oriented approach. When organizations adopt specialized databases, teams often struggle to learn new query languages and mental models. By providing a familiar SQL interface, QuestDB helps reduce these adoption barriers. Teams are already familiar with filtering, ordering, grouping, aggregation, and limiting result sets. Keeping these concepts allows programmers to focus on time-based challenges instead of learning a new query system. This is especially valuable when multiple roles need access to the same data. Financial systems illustrate this well. Market data, exchange rates, trades, and price movements are time-oriented, and many financial professionals already use SQL extensively. QuestDB lets teams leverage existing SQL expertise while gaining from a database optimized for high-frequency ingestion and time-based analysis. This advantage also applies to observability and operational analytics. Teams often need to query metrics for example request latency, throughput, error rates, or customer activity over specific intervals. Using familiar SQL constructs makes it easier to access application data for analysis. QuestDB does not function exactly like a traditional relational database. Its architecture and optimizations target time-series workloads. The result is an approachable query experience combined with a storage engine designed for specialized performance. For enterprises, this combination is powerful: specialized time-series performance without requiring teams to abandon the familiar query model. Conclusion QuestDB is a strong fit for applications that need fast ingestion, recent-state queries, historical analysis, and SQL over continuously changing data. With Eclipse JNoSQL 1.1.18, Java developers can use these capabilities through familiar Jakarta APIs, keeping the application model consistent while QuestDB handles the time-series specialization underneath.

By Otavio Santana

CORE

Why Your MCP Server Loses Session State Behind a Load Balancer

The Problem The Model Context Protocol has an initialize handshake. A client connects, the server hands back an Mcp-Session-Id header, and every request after that carries the same ID. That's fine with one server. It falls apart with three. A load balancer doesn't know or care about Mcp-Session-Id. It's an application-layer detail, and by default, the load balancer routes based on whatever algorithm it uses: round robin, least connections, or source IP hash. The client's second request can land on a completely different instance than the one that issued the session. That instance has never heard of the session ID, so it returns an error or, worse, silently starts a new session with no memory of the tools the client already listed or the state it had built up. Teams hit this the same way every time. Local development works because there's one process. Staging works because there's one pod. Production breaks the first time the deployment scales beyond a single replica, and the failure looks like a flaky client rather than an infrastructure problem, so it takes a while to trace. The standard fixes were the same ones every stateful HTTP service has used for 20 years: sticky sessions at the load balancer (route by session ID or client IP, accept the uneven load distribution), a shared session store like Redis so any instance can pick up any session, or deep packet inspection at the gateway to route on the Mcp-Session-Id header specifically. All three work. All three add an operational dependency to what should be a stateless API call. What Changed in the Spec The 2026-07-28 MCP specification release candidate removes protocol-level session management. The Mcp-Session-Id header is gone, and so is the session it represented. Connection metadata that used to be exchanged once at initialize time (protocol version, client info, client capabilities) now travels _meta on every request instead. The practical effect: a remote MCP server can sit behind a plain round-robin load balancer, route traffic on the Mcp-Method header if you want method-aware routing, and let clients cache tools/list responses for as long as the server's ttlMs value says they're valid. No sticky sessions. No shared session store just to keep the protocol working. This doesn't mean your server has to be stateless. It means the protocol stopped assuming state lives at the transport layer. If your server genuinely needs to remember something between calls (a shopping basket, an open browser tab, a half-finished multi-step operation), you handle it the way HTTP APIs have always handled it: mint an explicit handle from a tool call and have the model pass that handle back as an ordinary argument on the next call. Migrating a Stateful Server Say you have an MCP tool that opens a browser session and later needs to make calls to act on that same browser. Before the spec change, you'd have been tempted to key that off the transport-level session ID. After that, you do it explicitly: Python from fastmcp import FastMCP import uuid mcp = FastMCP("browser-tools") class BrowserHandle: """Minimal stand-in for a real browser automation session (e.g. Playwright).""" def __init__(self, start_url: str): self.start_url = start_url def click(self, selector: str) -> None: # Real implementation would drive an actual browser page here. pass # In-memory for the example; use Redis or a DB in production _sessions: dict[str, BrowserHandle] = {} @mcp.tool() def open_browser(start_url: str) -> dict: """Open a browser and return a handle for later calls.""" handle_id = str(uuid.uuid4()) _sessions[handle_id] = BrowserHandle(start_url) return {"browser_id": handle_id, "status": "opened"} @mcp.tool() def click_element(browser_id: str, selector: str) -> dict: """Click an element in a previously opened browser.""" session = _sessions.get(browser_id) if session is None: return {"error": f"No browser session for id {browser_id}. Call open_browser first."} session.click(selector) return {"status": "clicked", "selector": selector} The model now carries browser_id as an ordinary tool argument, the same way it would carry a basket_id or an order_id. That handle can live in Redis with a TTL, in a database row, wherever makes sense for your durability requirements. It's no longer the protocol's job to keep it alive. If you're running behind a load balancer today with sticky sessions configured specifically to work around the old MCP session model, this is your cue to check whether you can drop that configuration once your server and client both support the 2026-07-28 spec, or its final released version. Check the actual spec version your SDK negotiates before you rip out sticky sessions. Older clients still speaking the pre-07-28 protocol will still expect Mcp-Session-Id to work, and mixed-version fleets are exactly the kind of thing that turns a clean migration into a bad on-call week. What to Check Before You Migrate A few things worth confirming before you touch the load balancer config: Check your MCP SDK version and whether it implements the stateless spec or still assumes session pinning. Not every SDK moved at the same pace.Check whether any of your tools rely on server-side state that is implicitly tied to the session lifecycle (e.g., an open file handle, a database transaction, or a lock). Those need an explicit handle now, not an assumption that "the session" will still be around.Check your load balancer's health check and routing rules for anything referencing Mcp-Session-Id specifically. If your ops team added rules to route on that header, they can likely come out.Test with a client that is legitimately routed to different instances across consecutive calls, not just a local single-instance setup. That's the scenario the old model broke on, and it's the one you want to confirm the new model handles. References The 2026-07-28 MCP Specification Release CandidateSEP-2567: Sessionless MCP via Explicit State HandlesSEP-1442: Make MCP Stateless (by default)Scaling AI Agent Infrastructure with the MCP Stateless updates (Google Developers Blog)

By Eshaan Jain

OpenSearch Heap Sizing: Swap, Page Cache, and the 50% Rule

OpenSearch is an open-source, distributed search and analytics suite derived as a fork of Elasticsearch and maintained under the Apache 2.0 license. When it comes to memory configuration, the guidance is often reduced to a few rules of thumb: swapoff -a, vm.swappiness=1, or bootstrap.memory_lock, and allocating 50% of available memory to the JVM heap while leaving the rest for Lucene and the filesystem page cache, OpenSearch off-heap caches, network buffers, and other system needs. These recommendations are repeated throughout documentation, blog posts, and operational guides, yet their origins and the mechanisms that justify these specific values are rarely examined. Undoubtedly, they provide a reasonable and safe starting point or a safe upper bound in most of the cases, but a safe default is not necessarily an optimal configuration. All of this raises even more questions. How do these defaults affect cluster performance? What is the optimal JVM heap ratio? Does memory given up by the JVM actually become filesystem page cache, and at what point does that trade-off stop paying off? How do read/write latency correlate with the heap ratio? These questions become particularly important in resource-constrained environments and in the cloud, where long-term contracts may make existing instances significantly cheaper, making horizontal or vertical scaling a difficult decision. In this article, we'll try to answer these questions through benchmarking. This is Act 1 of a two-act series. Act 1 focuses on identifying the cause of the latency problems we observed with the current defaults. Act 2 will explore what other heap-ratio values might look like for read/write loads. Along the way, I'll share the tools and commands used throughout the investigation, making this article a practical reference as well for you and for myself when I inevitably need to retrace the investigation months later. Knowledge Context The story also crosses several boundaries, such as the Kernel VM, the JVM, and Lucene. So, it’s important to outline the concepts mentioned in this part of the article beforehand, both for the context and, optionally, to enrich the AI context if you'd like to summarize everything. AreaWhere memory livesWhy it matters hereLinux page cacheFile-backed RAMLucene relies heavily on it for index data; under memory pressure, these pages can be reclaimed and read again later.Linux swapDisk-backed anonymous memoryAnonymous process memory can be swapped out under pressure. vm.swappiness influences this decision but does not prohibit it.Linux PSIKernel pressure signalShows time tasks spend stalled due to CPU, memory, or I/O pressure. We'll use I/O PSI while investigating latency.JVM heapAnonymous memoryControlled by Xms/Xmx; contains Java objects and several OpenSearch data structures.JVM native memoryAnonymous/file-backed memory outside XmxIncludes code cache, metaspace, stacks, direct buffers, and native allocations. Heap metrics do not account for all of it.OpenSearch cachesHeap/off-heap, depending on cacheTheir sizes may depend on heap size, which becomes important when we change jvm_heap_ratio.OpenSearch indexing bufferHeapIts size depends on heap and therefore becomes an important variable in Act 2.Lucene mmapFile-backed/page cacheLucene index files mapped into the process do not consume JVM heap; resident pages compete for physical RAM. Environment I used Aiven for OpenSearch on Azure, with cluster metrics exported to Thanos. The OpenSearch Benchmark metrics don't provide everything we need to answer our questions, particularly host-level metrics such as Linux PSI and swap activity. Exporting the cluster metrics to Thanos allows us to use PromQL queries later to retrieve the additional metrics needed for the investigation. The cluster consists of 3 nodes: CPUAMD EPYC 7763v (Milan)vCPU / RAM2 vCPU, 8 GiBDisk Size175 GiB per nodeAzure Regionazure-westeuropeAzure DiskPremiumV2_LRSAzure SKUStandard_D2as_v5 OpenSearch Version3.6.0JDKjava-21-openjdk-headlessGCG1GC Act 1. The Latency and an Extra GB The symptom: elevated query latency across the cluster, first reported by the customer after a kernel and Azure image upgrade. The load pattern on OpenSearch itself remained unchanged, as did the cluster configuration and settings. The monitoring panels give us the first clue. In the screenshots below, the green vertical line marks the moment of the upgrade. After that point, the page cache grows by roughly a gigabyte, while I/O PSI, previously close to zero, starts showing significant spikes. Nothing crashed, no alert fired, and from the JVM point of view everything looked normal. So where did that extra gigabyte of page cache come from? Nothing was actually freed. It moved. The interesting part isn't just that memory went to swap; it's which memory. When you have thousands of running clusters, there is always a small fraction of them operating close to the edge: relatively stable, yet sensitive enough that even a small change can noticeably affect performance. Like a star nearing the end of its lifetime, they may look stable right up until something disturbs the balance. The immediate cause of the page-cache change was identified fairly quickly: Azure applies tuning parameters that differ from the Linux kernel defaults, and those parameters were not applied by the older image. Once applied, the larger buffers and read_ahead increased the filesystem cache footprint, putting additional pressure on anonymous memory and eventually pushing some of it to swap. But rather than stopping there, let's use this incident as an opportunity to experiment with the heap ratio and make the behavior of OpenSearch instances explicit and less dependent on such environmental changes. Evidence It Is on Swap; None of It Locked First, find what is going on on a node itself: Shell PID=$(pgrep -f 'org.opensearch.bootstrap.OpenSearch') grep -E 'VmRSS|RssAnon|RssFile|RssShmem|VmSwap|VmLck' /proc/$PID/status Shell VmRSS: 4366408 kB # resident RssAnon: 3329896 kB # heap + anonymous native RssFile: 1036496 kB # resident mmap'd Lucene pages RssShmem: 16 kB VmSwap: 2679496 kB # on swap VmLck: 0 kB # bootstrap.memory_lock=fasle, none locked Shell grep -E 'MemFree|MemAvailable|Cached|SwapFree' /proc/meminfo Shell MemFree: 258216 kB MemAvailable: 3479264 kB Cached: 3403196 kB SwapCached: 854396 kB SwapFree: 4745444 kB The Swap Device Is dm-crypt Then confirm there is somewhere for it to go, and on what kind of device: Shell swapon --show Plain Text NAME TYPE SIZE USED PRIO /dev/dm-4 partition 8G 3.5G -1 The dm-* swap device is the interesting detail to catch and to keep in mind. This is an encrypted device, so once a page is requested it could drive more I/O -> more dm-crypt allocations -> more high-order pressure. A self-reinforcing loop and a good example of read amplification. Paging Is Live, Not Historical The next logical step is to check whether it is live paging or just a stale historical tail. The vmstat 1 5 the Linux Virtual Memory Statistics Tool should give us an exact answer for this, where non-zero swap blocks in and out (marked as si, so): Shell vmstat 1 5 Plain Text procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu------- r b swpd free buff cache si so bi bo in cs us sy id wa st gu 1 0 3620928 159716 5864 3704212 113 80 3361 577 3147 12 8 7 84 1 0 0 0 0 3625240 189036 5860 3705260 0 5228 24268 5228 4841 4072 14 12 69 4 0 0 0 0 3625240 166196 5860 3710696 0 0 32 0 3572 2556 21 5 74 0 0 0 0 0 3625240 162668 5860 3716200 8 0 164 0 3729 2619 20 7 73 0 0 0 12 0 3625312 147692 5860 3724288 0 140 9207 524 3679 4024 22 14 63 1 0 0 Non-zero si/so in 4 of 5 samples show the live paging process, swpd also climbing across five seconds, which is good proof. The Swapped Pages Are Anonymous, Not File-Backed Let's also check swap memory consumption for each of the process's mappings, to confirm that swap is heap-related: Shell PID=$(pgrep -f org.opensearch.bootstrap.OpenSearch) awk '/^[0-9a-f]/{h=$0} /^Swap:/{if($2>0)print $2" kB "h}' /proc/$PID/smaps | sort -rn | head -20 Plain Text 1160380 kB 708400000-7ffe00000 rw-p 00000000 00:00 0 62296 kB 7f30c0000000-7f30c3f4b000 rw-p 00000000 00:00 0 61452 kB 7f1fc8000000-7f1fcbc03000 rw-p 00000000 00:00 0 61448 kB 7f1fa8000000-7f1fabc02000 rw-p 00000000 00:00 0 61444 kB 7f2f38000000-7f2f3bc01000 rw-p 00000000 00:00 0 61444 kB 7f24b4000000-7f24b7c01000 rw-p 00000000 00:00 0 61444 kB 7f22ac000000-7f22afc01000 rw-p 00000000 00:00 0 61444 kB 7f2238000000-7f223bc01000 rw-p 00000000 00:00 0 61444 kB 7f216c000000-7f216fc01000 rw-p 00000000 00:00 0 61444 kB 7f2168000000-7f216bc01000 rw-p 00000000 00:00 0 61444 kB 7f2148000000-7f214bc01000 rw-p 00000000 00:00 0 61444 kB 7f1fec000000-7f1fefc01000 rw-p 00000000 00:00 0 61444 kB 7f1fe8000000-7f1febc01000 rw-p 00000000 00:00 0 61444 kB 7f1fcc000000-7f1fcfc01000 rw-p 00000000 00:00 0 61444 kB 7f1fac000000-7f1fafc01000 rw-p 00000000 00:00 0 60772 kB 7f1fc0000000-7f1fc3c14000 rw-p 00000000 00:00 0 59952 kB 7f311e000000-7f3123f41000 rw-p 00000000 00:00 0 59340 kB 7f30b4000000-7f30b7c9b000 rw-p 00000000 00:00 0 57180 kB 7f3110000000-7f3113e3e000 rw-p 00000000 00:00 0 43084 kB 7f3130800000-7f3133d70000 rwxp 00000000 00:00 0 The Largest Swapped Region Is But why is what we are seeing above a heap-related area? There are a few clues for that. The region size 0x708400000 - 0x7ffe00000 is exactly 4,154,458,112 bytes = 3,962 MiB as we use -Xms == -Xmx and the whole thing is committed at startup, and nothing else in a JVM process is a single contiguous ~4 GB anonymous rw-p mapping. Second, It's the lowest mapping in the address space, smaps_rollup [rollup] line starts at exactly 708400000: Shell cat /proc/$PID/smaps_rollup Shell 708400000-7ffd3bb56000 ---p 00000000 00:00 0 [rollup] Private_Dirty: 3213788 kB Swap: 2680532 kB SwapPss: 2679412 kB Locked: 0 kB The JIT Code Cache Is Swapped Too Decoding the top swapped regions: 1160380 kB at 708400000 – is the JVM heap, the compressed‑oops heap base and matches the [rollup] start from smaps_rollup.The dozens of 61444 kB regions – these areas are probably related to native/off‑heap: Netty, JNI, Lucene native, etc.43084 kB marked rwxp – the JIT code cache, also swapped out, a bad sign. Together, these regions account for almost exactly the ~2.5 GB of swapped memory we observed earlier: cold heap regions, native/off-heap allocations, code cache, and possibly thread stacks. Practically, this means two consequences: GC can amplify swap latency. 1 GB of the JVM heap was swapped out. G1 does not necessarily touch all of those pages during a mixed collection, but any GC phase that accesses a swapped page incurs a major fault and has to bring it back through the dm-crypt device. Hence, short GC work can produce significantly longer pauses.A swapped page can also contain executable code. The next call into a swapped-out compiled method can trigger a major fault before the code can run. The resulting latency may land on an otherwise random request and be difficult to attribute directly to GC, index I/O, or the query itself. Evidence of Sustained Anon Memory Churn workingset_refault_anon counts anonymous memory refault events after reclaim; it does not count unique pages. Together with pswpin and pswpout, it shows how much anonymous memory paging has accumulated since boot. Shell grep -E 'workingset_(refault|activate)_anon|pswpin|pswpout' /proc/vmstat Plain Text workingset_refault_anon 89061837 workingset_activate_anon 1856069 pswpin 86690374 pswpout 61618603 These counters are cumulative, so fetching them at 10-minute intervals clearly shows that this wasn't a one-time eviction. There was sustained process: nearly 89 million anonymous pages were repeatedly swapped out and faulted back in. vm.swappiness = 1 and GC Amplification In this story vm.swappiness=1 was set since the cluster's inception. It does what Linux defines it to do, but it doesn't provide the protection we wanted. I suspect this matters particularly in the most resource-constrained deployments. How do we know that? The entire result above is a counterexample. swappiness biases the kernel's choice between reclaiming file-backed and anonymous pages. It does not prevent anonymous pages from being swapped out. Even at 1, this can still happen under sustained memory pressure. On a resource-constrained node whose index is several times larger than its RAM, pressure on the filesystem cache is not an exceptional condition; this is the normal operating state. This has two important consequences: It is not self-healing. Nothing proactively pages anonymous memory back in on a schedule. A swapped-out page returns to RAM only when it is accessed again, and cold memory, as it's defined, may remain untouched for a long time. As a result, the cold JVM memory can remain in swap indefinitely.It may remain invisible until something touches it. At steady state, a cold tail of the heap can remain in swap without producing obvious symptoms. The problem becomes visible when those pages are touched again, causing major page faults and potentially amplifying GC and request latency. Key Takeaways So, the root cause of the latency problems is an oversized heap combined with page cache pressure (the cold heap tail has been swapped out). vm.swappiness=1 did not protect the heap. It biases what gets reclaimed; it does not prevent anonymous memory from being swapped out.vm.swappiness=1 should not be relied on with the other defaults in production. An oversized heap can lead to GC amplification that is difficult to detect.Most of the swapped-out memory wasn't heap at all, but malloc arenas and the JIT code cache, none of it inside Xmx, so heap metrics didn't show it.Swap on dm-crypt exacerbates the issue, resulting in longer GC pauses and random request latency spikes. dm-crypt may be unavoidable in production due to security requirements. The fix isn't another swappiness tweak. It's two things: stop committing heap you don't use, and make the heap you do commit non-evictable. Follow Up In Act 2, we'll answer the remaining questions raised at the beginning of this article and look more closely at the trade-off introduced by bootstrap.memory_lock. This setting makes the heap resident and swap-immune, but it also turns jvm_heap_ratio from a soft default into a permanent memory commitment. The question then becomes: what heap ratio best suits different read and write workloads? There is one more complication to mention in advance: in a resource-constrained environment, merge storms can distort benchmark results, making an otherwise good heap ratio appear poor. See the screenshot below.

By Maxim Muzafarov

Why Time Series Databases Matter for Modern Enterprise Applications

Time is now a critical dimension in enterprise data. Modern applications generate a constant stream of events, such as payments, sensor readings, infrastructure metrics, user activity, logistics movements, market data, and application telemetry. The value exists not only in what happened, but also when it occurred, what came before it, and how the data changes over time. While traditional relational and NoSQL databases can store this information, increasing data volume and frequency make it essential to efficiently query recent states, historical windows, trends, and ordered events. Time-series databases fulfill these needs by treating time as a primary dimension. They make it easy to retrieve the latest measurements, examine historical intervals, aggregate data over time, and handle high-frequency writes. For enterprise applications, time-series databases now support not only monitoring and IoT, but also observability, financial systems, industrial platforms, logistics, energy, and any architecture where tracking data evolution is as important as the data itself. Why Time Series Databases Matter At first glance, time-series data appears similar to data stored in relational or NoSQL databases. SQL tables can include timestamp columns, document databases can store dated events, and key-value stores can use time-based keys. The distinction appears when time becomes the primary access pattern rather than a secondary attribute. If typical queries include “what is the latest value?”, “what happened during this interval?”, or “how did this metric change over time?”, a time-series model is often a better fit. Relational databases can support these queries, but they frequently require complex indexes, partitions, retention policies, and aggregation logic as data volume increases. Document and key-value databases typically require the application to manage temporal structure. Time-series databases are designed for append-heavy workloads, ordered data, time-window queries, downsampling, retention, and time-based aggregation. They do not replace SQL or general-purpose NoSQL databases, but serve as a specialized solution when tracking data changes over time is essential. A useful way to think about the difference is this: Plain Text Relational database: "What is the current state of this entity?" Document database: "What does this aggregate/document look like?" Key-Value database: "What value is associated with this key?" Time Series database: "How has this value changed over time, and what is happening now?" Consider a temperature sensor. In a relational model, we could create a table like: SQL CREATE TABLE sensor_reading ( timestamp TIMESTAMP, sensor VARCHAR(100), temperature DOUBLE ); This approach is functional, but typical workloads often require queries such as: Plain Text Give me the latest value for sensor-01. Give me the last 100 measurements. Give me the average temperature every five minutes. Show me the period where the temperature exceeded 30°C. Compare today's measurements with yesterday's. Delete or compress measurements older than six months. These are not isolated queries; they represent the typical data lifecycle. Time-series databases are specifically designed to support this pattern. Beyond IoT IoT is a clear example, as sensors naturally generate timestamped measurements. However, this model is common across many enterprise systems. Observability and infrastructure monitoring are classic use cases. Measurements such as CPU usage, memory consumption, request latency, error rates, queue depth, database connections, and network throughput are sampled repeatedly over time. Rather than focusing on a single point, teams typically seek trends, spikes, rolling averages, anomalies, and interval comparisons. Financial systems also generate highly temporal data. Market prices, exchange rates, trades, account activity, risk measurements, and portfolio valuations are all chronological streams. Trading applications may require the latest price, recent ticks, or price ranges within specific windows. Banking systems regularly retain transactional data in relational databases while sending temporal metrics and activity streams to time-series stores for analysis. Logistics and transportation also benefit from this approach. Vehicle position, speed, fuel consumption, delivery status, warehouse temperature, and route telemetry are all continuously changing metrics. Relevant queries include: Plain Text Where was the vehicle during the last hour? How has fuel consumption changed during this route? When did the delivery temperature leave the acceptable range? What was the most recent location reported by each vehicle? Energy and utilities are also inherently time-oriented. Smart meters, electricity consumption, voltage, water usage, solar generation, battery charge, and grid load all generate continuous measurements. While billing may remain in a relational system, raw measurements and their aggregation suit time-series databases well. Application and business metrics further demonstrate that time-series data goes beyond infrastructure. Enterprise applications may record: orders per minutepayments approved per minutefailed logins per houractive userscheckout latencyinventory changesfraud scoresmessages processedAPI calls by customer These are business signals, not simply technical telemetry. Persisting them over time enables organizations to understand both system and business processes as they evolve. Event-driven and distributed architectures present another important use case. Microservices continuously emit events and measurements. While event stores and message brokers are effective for transporting and preserving events, they are not optimized for queries such as: Plain Text What was the average latency for this service during the last 30 minutes? What is the latest measurement from every region? How has throughput changed since the deployment? Which customers experienced the highest error rate over the last hour? A time-series database can complement Kafka, Pulsar, or other event backbones by providing a queryable temporal view of these streams. This distinction is important: selecting a time-series database does not require replacing relational databases, document stores, or event brokers. Modern enterprise architectures are increasingly polyglot. Relational databases may remain the system of record, document databases may manage flexible aggregates, key-value databases may provide fast lookups, and time-series databases may handle continuously changing measurements. The architectural advantage lies in choosing a data model that corresponds to the application's key questions. Java and Time Series Traditionally, using time-series databases in Java required working with database-specific drivers and APIs. Developers needed to learn unique programming models, configuration styles, query APIs, and object-mapping strategies for each database. This increased implementation complexity and made maintenance challenging, especially when multiple time-series technologies were used. Eclipse JNoSQL 1.1.18 addresses this complexity by introducing Time Series as a supported database type. This release adds support for InfluxDB 3, Apache IoTDB, and QuestDB, along with a new Jakarta NoSQL Template specialization: TimeSeriesTemplate. Java @Inject private TimeSeriesTemplate template; The mapping model remains consistent. Time-series entities continue to use Jakarta NoSQL annotations: Java @Entity public class SensorReading { @Id private Instant timestamp; @Column private String sensor; @Column private double value; // constructors, getters, and setters } As with other Jakarta NoSQL database types, the primary difference lies in the data model. In time-series models, the identifier usually represents the temporal dimension. In this example, @Id is an Instant marking when the measurement occurred. The programming model remains familiar: Java @Inject private TimeSeriesTemplate template; SensorReading reading = new SensorReading( Instant.now(), "sensor-1", 21.5); template.insert(reading); Optional<SensorReading> result = template.find(SensorReading.class, reading.getTimestamp()); List<SensorReading> readings = template.select(SensorReading.class) .where("sensor").eq("sensor-1") .result(); The same model applies to Jakarta Data repositories: Java @Repository public interface SensorReadingRepository extends BasicRepository<SensorReading, Instant> { List<SensorReading> findBySensor(String sensor); } Then the repository can be injected normally: Java @Inject private SensorReadingRepository repository; The key improvement is not Java’s ability to connect to time-series databases, as native drivers have always enabled this. The real value is that developers can now use a consistent Jakarta NoSQL and Jakarta Data programming model across different time-series databases, reducing database-specific code within applications. Conclusion Time-series databases matter because many modern systems are no longer only about storing the current state of data, but about understanding how that data changes over time. From observability and finance to logistics, energy, IoT, and business metrics, a time-oriented model can simplify both the architecture and the queries when temporal behavior is central to the problem.

By Otavio Santana

CORE

Part 3: End-to-End Tracing and Observability Across Goose, agentgateway, and Quarkus

Enterprise context — Acme FinServ. SOC 2 CC7 (system monitoring) requires that Acme can detect and investigate anomalous activity. When an agent-driven workflow touches customer data at 2 AM, "we have logs somewhere" is not an answer an auditor accepts. The distributed trace built in this part is the forensic evidence trail: a single trace ID that ties the Goose prompt to every agentgateway policy decision and every Quarkus tool call, so a post-incident review can reconstruct exactly which agent did what, in what order, and how long each governed hop took. The Core Problem In Part 1, we built a Quarkus MCP tool server. In Part 2, we secured it with agentgateway's JWT authentication, RBAC, and ExtMCP guardrails. The architecture works — but when something goes wrong in production, you're flying blind. Agentic workflows are fundamentally different from traditional request-response APIs. A single user prompt like "Debug customer CUST-4091" triggers a multi-round-trip loop: Goose calls tools/list to discover available toolsThe LLM selects getCustomerStatus and Goose sends tools/callThe LLM reads the response, sees primaryRegion: US-EAST-1, and chains a second tools/call to getZoneHealthLogsThe LLM correlates both results and generates a diagnostic summary Each of these hops crosses process boundaries: Goose → agentgateway → Quarkus. Without distributed tracing, you see four isolated HTTP requests in your access logs. You cannot tell they belong to the same agentic workflow. When step 3 takes 12 seconds instead of 200ms, you have no waterfall to pinpoint whether the latency came from agentgateway policy evaluation, Quarkus bean validation, or a slow downstream call. This creates black holes in telemetry dashboards — the exact gap that autonomous agents exploit to degrade silently. The Solution: W3C Trace Context Across All Three Layers The fix is standard distributed tracing, applied to the MCP transport layer: agentgateway exports spans for every proxied MCP request and propagates traceparent headers to the backend.Quarkus with quarkus-opentelemetry picks up the incoming traceparent, creates child spans for tool execution and bean validation, and exports them to the same Jaeger instance.Jaeger correlates both sides into a single trace waterfall — one view from agent prompt to tool result. Prerequisites Everything from Parts 1 and 2, plus: Podman – for running Jaeger (podman compose) Verify Podman is available: Shell podman --version Step 1: Launching the Observability Backend We use Jaeger v2 as both the OTLP collector and the trace UI. A single container accepts traces from agentgateway on port 4317 (OTLP gRPC) and from Quarkus on port 4318 (OTLP HTTP), and serves the query UI on port 16686. Shell cd part3-observability podman compose up -d This starts Jaeger v2 with OTLP collection enabled by default. Verify it's running: Shell curl -sf http://localhost:16686/ > /dev/null && echo "Jaeger UI is ready" Open http://localhost:16686 — you'll see an empty Jaeger UI. We'll populate it with MCP traces in the following steps. Production Alternative: Grafana Tempo For production deployments, replace Jaeger with Grafana Tempo backed by object storage (S3/GCS). The OTLP endpoint stays the same — only the compose.yml changes. Grafana provides richer dashboards, alerting, and long-term trace retention. Step 2: Enabling OpenTelemetry in Quarkus Add the quarkus-opentelemetry extension to Part 1's pom.xml: Properties files <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-opentelemetry</artifactId> </dependency> Configure the exporter in application.properties: Properties files # OpenTelemetry quarkus.otel.service.name=customer-tools quarkus.otel.exporter.otlp.traces.endpoint=http://localhost:4318 quarkus.otel.exporter.otlp.traces.protocol=http/protobuf quarkus.otel.traces.sampler=always_on quarkus.otel.traces.suppress-non-application-uris=false PropertyPurposeservice.nameIdentifies this service in Jaeger's service dropdowntraces.endpointOTLP HTTP receiver — Jaeger's port 4318 (base URL only; Quarkus appends /v1/traces)traces.protocolhttp/protobuf — Quarkus uses its Vert.x-based HTTP exportertraces.sampleralways_on — sample every span (reduce in production)suppress-non-application-urisfalse — include MCP endpoint spans (they'd be filtered otherwise) When no OTLP collector is running (Parts 1 and 2 without Jaeger), Quarkus logs a connection warning, but the MCP server works normally. When the collector IS running (Part 3), traces flow automatically. Zero code changes to the MCP tools. Rebuild Part 1: Shell cd part1-quarkus-mcp mvn package -DskipTests What Quarkus Auto-Instruments With quarkus-opentelemetry on the classpath and the SDK enabled, Quarkus automatically creates spans for: LayerSpan NameWhat It CapturesHTTP serverPOST /mcpInbound MCP request with method, status, latencyCDI beansCustomerServiceTools.getCustomerStatusTool execution time within the MCP handlerBean ValidationHibernateValidatorParameter validation before tool logic runsREST clientOutbound HTTP callsAny downstream API calls (future extensions) No @WithSpan annotations needed. The Quarkus OpenTelemetry extension instruments the reactive pipeline automatically. Step 3: Configuring W3C Trace Context in agentgateway agentgateway supports native OpenTelemetry trace export. Add a tracing block to the gateway configuration: Properties files config: adminAddr: localhost:15000 tracing: otlpEndpoint: http://localhost:4317 otlpProtocol: grpc randomSampling: 1.0 FieldPurposeotlpEndpointOTLP receiver — Jaeger's port 4317otlpProtocolgrpc for OTLP/gRPC (also supports http)randomSamplingSample 100% of traces (reduce to 0.01–0.1 in production) How Trace Propagation Works When agentgateway receives an MCP request: Creates a root span for the proxy operation (e.g., agentgateway.mcp.proxy)Injects a traceparent header into the forwarded request to Quarkus:traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01Quarkus reads the traceparent, creates a child span under the same trace ID, and records tool executionBoth spans export to Jaeger via OTLP, where they appear as a single correlated trace This is standard W3C Trace Context propagation — the same mechanism used across all OpenTelemetry-instrumented services. Configuration Files Part 3 provides two agentgateway configurations: ConfigUse Caseconfig-traced.yamlTracing only — proxy + OTLP export, no security layersconfig-traced-guardrails.yamlTracing + ExtMCP guardrails — observe the guardrail evaluation spans too Step 4: Running the Interactive Demo Start all services with the one-command script: Shell cd part3-observability ./start-all.sh The script starts Jaeger, Quarkus (with OTel enabled), and agentgateway (with trace export), then launches the demo SPA on :8890. Open the MCP Observability Console at http://localhost:8890/index.html and walk through the three demo steps: Initialize – Establishes an MCP session through agentgateway. The architecture diagram animates the trace propagation: root span creation in agentgateway, traceparent injection, child span in Quarkus, and OTLP export to Jaeger.List Tools – Discovers all 5 tools through the traced proxy. The trace waterfall panel shows the agentgateway proxy span and the Quarkus HTTP span side by side with timing.Multi-Tool Workflow – Simulates Goose's multi-turn reasoning: getCustomerStatus (finds region US-EAST-1) → getZoneHealthLogs (checks zone health) → getSLACompliance (correlates SLA metrics). Each step generates a full trace with waterfall visualization. The stat tiles track traces generated, spans collected, and Jaeger status. Click Open Jaeger to view the real trace waterfalls in the Jaeger UI at http://localhost:16686. Step 5: Generating Traces via CLI To generate additional traces manually, simulate a multi-turn agentic workflow: Shell # Step 1: Initialize MCP session export MCP_SESSION_ID=$(curl -s -D - http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}' \ | grep -i "mcp-session-id:" | sed 's/.*: //' | tr -d '\r') # Step 2: Discover tools curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 3: Agent calls getCustomerStatus (first tool invocation) curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"getCustomerStatus","arguments":{"customerId":"CUST-4091"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 4: Agent chains getZoneHealthLogs based on the region from step 3 curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"getZoneHealthLogs","arguments":{"zoneId":"US-EAST-1"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . # Step 5: Agent fetches SLA compliance for correlation curl -s http://localhost:3000/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "MCP-Protocol-Version: 2025-03-26" \ -H "mcp-session-id: $MCP_SESSION_ID" \ -d '{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"getSLACompliance","arguments":{"serviceId":"api-gateway"}}' \ | grep '^data: ' | sed 's/^data: //' | jq . Each of these requests generates a trace that flows through agentgateway into Quarkus and lands in Jaeger. Step 6: Visualizing the Trace Waterfall in Jaeger Open http://localhost:16686 in your browser. Finding Traces In the Service dropdown, select customer-tools (Quarkus) or agentgatewayClick Find TracesClick on any trace to open the waterfall view Reading the Waterfall A typical tools/call trace shows the following span hierarchy: Shell agentgateway.mcp.proxy [12ms] └─ POST /mcp [8ms] ← Quarkus HTTP server └─ CustomerServiceTools.getCustomerStatus [2ms] ← CDI tool execution SpanServiceWhat It Tells Youagentgateway.mcp.proxyagentgatewayTotal proxy overhead including policy evaluationPOST /mcpcustomer-toolsQuarkus HTTP handling time for the MCP requestgetCustomerStatuscustomer-toolsPure tool execution time (business logic) What to Look For Proxy overhead: The gap between the agentgateway span and the Quarkus span shows network + policy evaluation time. If this grows, check guardrail server latency.Validation time: Bean Validation spans appear before tool execution. Regex-heavy patterns like ^CUST-[0-9]{4,8}$ are fast, but complex validators on large payloads can add latency.Multi-turn correlation: When Goose chains multiple tool calls (e.g., getCustomerStatus → getZoneHealthLogs), each appears as a separate trace. The mcp-session-id tag lets you filter all traces belonging to one agent session.Error traces: Failed validations (invalid customer ID format) or guardrail rejections (blocked poison payloads) produce error spans with exception details. Connecting Goose for Real Traces Launch Goose pointed at agentgateway and prompt a multi-tool workflow: Shell goose session "Debug customer CUST-4091 — check their account status, then pull health logs for their region and SLA compliance for api-gateway." This generates a burst of correlated traces in Jaeger showing Goose's multi-turn tool orchestration from the proxy layer down to individual tool execution spans. What We Achieved Starting from the secured architecture in Part 2, we added full observability without changing any MCP tool code: LayerWhat We AddedConfig ChangeQuarkusquarkus-opentelemetry dependencypom.xml + application.propertiesagentgatewaytracing block in config YAMLconfig-traced.yamlObservability backendJaeger all-in-one via Podman Composecompose.yml The entire stack runs locally with a single ./start-all.sh command and produces end-to-end trace waterfalls in Jaeger. Production Considerations ConcernLocal (this tutorial)ProductionTrace backendJaeger all-in-one (in-memory)Grafana Tempo + object storageSampling rate100% (default: 1.0)1-10% or adaptive samplingTrace retentionContainer lifetimeDays/weeks in durable storageAlertingManual Jaeger inspectionGrafana alerting on span latency/error rateMetricsTraces onlyAdd Prometheus + quarkus-micrometer for RED metrics Coming Up in Part 4 With tracing in place, you can now see every MCP tool call flowing through the system. In Part 4, we will move beyond single-agent tool calls to multi-agent orchestration — using the Agent-to-Agent (A2A) protocol to coordinate autonomous agents that can delegate work, enforce governance via AGENTS.md, and call back into our MCP tool services.

By Daniel Oh

CORE

The Telemetry Tax: Architecting Zero-Allocation Event Observability at 15B+ Daily Event Scale

Throughout my career, observability and monitoring systems have played a very important role in system stability, availability, and scalability when architecting, whether a system handles millions or billions of events per day. Having architected core infrastructure across communication platforms, enterprise compliance pipelines, and high-throughput transaction systems, I have repeatedly seen how background instrumentation can silently become a bottleneck under heavy production load. Telemetry and observability come with their own challenges. Instrumenting telemetry without degrading live business application performance, increasing request latency, or inflating cloud costs is often deprioritized initially but becomes very important as the system scales. Furthermore, it is a much harder problem to solve than processing domain events themselves in production. When an event-driven platform scales past 15 billion events per day, peaking between 200,000 and 400,000 TPS (transactions per second), the infrastructure costs and performance overhead of logging and tracing libraries create a severe operational bottleneck known as the Telemetry Tax. This article explains why addressing this telemetry tax is important and also provides you with brief insight into how you can start handling it with architectural changes at the initial level. The Multiplier Effect In a very scalable, high-throughput microservices architecture, business transactions rarely execute as isolated operations. An incoming event arriving at an API gateway typically passes through five to ten downstream services, caching layers, database instances, queues, message brokers (e.g., Kafka), etc. If every microservice hop generates three telemetry events in addition to the core business logic events, assuming two metric updates and a single log line, the resulting metric data scales exponentially: Regular workload (Core business Logic events): 15,000,000,000 events per dayMetrics and logs: 100,000,000,000 signals per dayStorage estimation: 15+ TBs of uncompressed JSON payloads daily Without specialized memory management and proper architecture, this massive volume of background instrumentation introduces critical failures in live production systems. Example: Garbage collection: Short-lived heap allocation for JSON log stringsThread Lock Contention: Sync metric exporters hold mutex locks while completing the request to send telemetry metrics, adding 10-20ms of latencyCascading downstream failures: Failure in downstream telemetry collector or failure in processing existing logs and metrics causes issues or backlog in upstream transaction workers, which may result in queue back-pressure and API timeouts. Engineering a Zero-Allocation Telemetry Pipeline To eliminate garbage collection pauses and thread lock delays, high-throughput microservices must decouple metric and log generation from core business logic, utilizing a middleware concept with pre-allocated memory pools and asynchronous buffers. Under heavy execution loads, standard string concatenations from logging and JSON marshaling for telemetry signals escape to the heap during Go's escape analysis. When millions of goroutines allocate short-lived objects on the heap per second, Go's runtime is forced into frequent concurrent sweep phases, consuming 15-20% of available CPU purely on garbage collection. By utilizing sync.Pool, we pre-allocate backing byte arrays that survive across the request lifecycle. When a worker goroutine finishes formatting a payload, the buffer pointer is reset (buf[:0]) and returned to the pool without invoking runtime.newobject . This keeps memory allocation in the critical execution path at zero bytes. Go package telemetry import ( "context" "sync" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" ) var tracer = otel.Tracer("zero-alloc-telemetry") // BufferPool pre-allocates memory var bufferPool = sync.Pool{ New: func() interface{} { b := make([]byte, 0, 1024) return &b }, } type RingBufferPipeline struct { telemetryChannel chan *trace.Span } func NewPipeline(bufferSize int) *RingBufferPipeline { return &RingBufferPipeline{ telemetryChannel: make(chan trace.Span, bufferSize), } } func (p *RingBufferPipeline) InstrumentEvent(ctx context.Context, eventID string) { // Retrieve pre-allocated byte slice from pool bufPtr := bufferPool.Get().(*[]byte) buf := (*bufPtr)[:0] defer func() { *bufPtr = buf bufferPool.Put(bufPtr) }() // Non-blocking trace span initialization ctx, span := tracer.Start(ctx, "ExecuteTransaction", trace.WithAttributes(attribute.String("event.id", eventID))) defer span.End() buf = append(buf, []byte("event_processed:")...) buf = append(buf, eventID...) // Async non-blocking dispatch select { case p.telemetryChannel <- span: default: // Add metric here to protect API response SLOs } } How to Control Ingestion Costs With Tail-based Adaptive Sampling Collecting 100% of telemetry traces across 100+ billion daily events leads to unsustainable tool ingestion costs (Regardless of the tool you use, e.g., Datadog, OpenTelemetry, Splunk). Standard Head-based sampling (deciding whether to keep a trace at the start of the request) drops fatal error traces while keeping millions of redundant HTTP 200 success traces. By deploying OpenTelemetry collectors configured with Tail-based adaptive sampling, trace spans are held in a 500 ms sliding memory buffer prior to routing: HTTP 200 / Successful Executions: Sampled at 0.1% to maintain baseline latency metricsHTTP 5xx / Latency exceptions (>200ms) / Server Errors: Retained at 100% for debugging, or any other analysis Production Performance Benchmark (Approx.) Refactoring telemetry infrastructure from sync logging to zero-allocation ring buffers and tail-based adaptive sampling helps with substantial performance improvements. (Note: The numbers in the table below are rough estimations based on high-scale modeling and past operational experience. Actual metrics may vary depending on your system and other aspects of architecture) Performance MetricSync TelemtryZero-allocation Adaptive TelemetryIngestion Throughput35k events/sec400k events/secp99 API response latency200ms40msAverage Monthly Cost$35-45k+$15-20kCPU overhead15-20% total CPU time spent on GC2-3% total CPU time spent on GC These estimations illustrate that the telemetry tax is not an inevitable cost of scale, but a consequence of applying synchronous, allocation-heavy architectural patterns to hyper-scale architectures. By refactoring memory management at the application level and applying adaptive filtering at the collector layer, engineering teams can achieve deep operational visibility while protecting system performance and cloud infrastructure budgets. Key Takeaways for System Architects Decouple the path: Never allow telemetry exporters to execute synchronously on the core business logic pathPre-allocate memory: Use memory buffer pools to eliminate garbage collection pauses during high-throughput event processingSample at the tail: Evaluate trace retention based on the execution outcome rather than making static decisions at the request ingress.

By Brindal Patel

Gossips on Cryptography: Part 4

In this blog, we will continue our discussion from the previous parts. If you have not read them, please read them first. Parts 1 & 2 – Caesar Cipher, Vigenere Cipher, Symmetric Encryption, AES, Convergent Encryption, IVPart 3 – Hashing, Salting, Rainbow Table Attacks, Asymmetric Encryption, RSA In Part 3, we teased a few topics for Part 4 — Envelope Encryption, PKI, and more. Today we gossip about exactly those! Let's go. First, A Quick Revisit: Convergent Encryption We discussed Convergent Encryption back in Parts 1 & 2, but let's revisit it here because it connects beautifully to Envelope Encryption. Remember? Convergent Encryption means — if you encrypt the same plaintext with the same key and the same IV, you will always get the same ciphertext. So Why Is That Useful? Imagine you work at a big company. 500 employees all upload the same file — let's say the company's HR policy PDF. If you use regular encryption (different ciphertext every time), your storage system stores 500 different encrypted copies. That's 500x storage wasted! With Convergent Encryption, since the same file + same key = same ciphertext, the storage system realizes — "Hey, I already have this encrypted file!" — and stores only ONE copy. All 500 employees point to the same encrypted blob. This is called deduplication. This is exactly how Dropbox, Google Drive, and AWS S3 save enormous amounts of storage at their scale. Another Use Case — Searching Over Encrypted Data Here is another very powerful use case of Convergent Encryption that most people don't think about — searching. Imagine you have a database where all the data is encrypted. A user wants to search for records where the email is "[email protected]." With regular encryption, every time "[email protected]" is encrypted, it produces a **different** ciphertext (because of a random IV). So to search, you would have to: Decrypt every single record in the databaseCompare the plaintextReturn the matches That is insanely expensive! Imagine doing this on a database with 100 million records. Your server will cry. Now with Convergent Encryption — "[email protected]" always produces the **same** ciphertext. So to search, you just: Encrypt the search term "[email protected]" onceLook for that ciphertext in the database — just like a normal indexed search!Return the matches No decryption needed at all! The data stays encrypted at rest, and you can still do fast, exact-match searches on it. This is called searchable encryption, and it is used in scenarios like: Encrypted databases where you still need to support queriesHealthcare systems — searching patient records without ever exposing raw dataEmail systems — searching your encrypted inbox without the server ever seeing your emails in plain text Pretty powerful, right? Same property (deterministic output) — two completely different superpowers (deduplication + searchable encryption). But wait — there's a catch. If two people can produce the same ciphertext, can someone guess your file? Yes, this is called a confirmation attack. Someone could hash a known file, compare it with stored hashes, and confirm whether you uploaded that file. So Convergent Encryption is great for performance and deduplication but is used carefully in highly sensitive scenarios. Now Let's Talk About Envelope Encryption Okay, so now we know — encryption needs keys. And those keys need to be stored somewhere safely. But here's the problem — who encrypts the key itself? If your key is lying around in plain text, a hacker who gets access to your server gets everything. So the answer is — we encrypt the key too! This is the core idea of Envelope Encryption. The Two Keys in Envelope Encryption DEK — Data Encryption Key. This is the key that directly encrypts your actual data. Think of it as the key to your diary.KEK — Key Encryption Key. This is the master key that encrypts the DEK. Think of it as the key to your locker — inside which you keep your diary key. So the flow looks like this: Your Data → encrypted with DEK → Encrypted Data DEK → encrypted with KEK → Encrypted DEK You store both — the Encrypted Data and the Encrypted DEK — together. The KEK lives safely inside a highly secure system (like AWS KMS or Vault). Real-Life Example — The Bank Locker Imagine you have an important document (your data). You put it in a box and lock it with a small key (DEK). Now you don't want to carry this small key everywhere — so you put the small key inside your bank locker (encrypt DEK with KEK). The bank locker key (KEK) stays with the bank in a highly secure vault. To read your document: Go to the bank → get your small key out (decrypt DEK using KEK)Use the small key to open the box (decrypt data using DEK) Simple! And very secure. Why Not Just Encrypt Data Directly With KEK? Two very practical reasons: 1. Performance The KEK usually lives inside a secure hardware vault or cloud service (like AWS KMS). If you send your entire 10GB file to KMS every time you want to encrypt or decrypt — that's painfully slow and expensive. Instead, you only send the tiny DEK (a few bytes) to KMS. The heavy lifting of encrypting actual data is done locally with the DEK. 2. Key Rotation Say after 6 months you want to change your encryption key (key rotation is a security best practice). Without envelope encryption — you'd have to decrypt ALL your data and re-encrypt it with a new key. Imagine doing that for terabytes of data! With Envelope Encryption — you only re-encrypt the DEK with the new KEK. Your actual data stays untouched. Much faster, much cheaper. Where Is Envelope Encryption Used? Literally everywhere in the cloud world: AWS S3 – When you enable server-side encryption on a bucketAWS RDS – When you enable encryption on a databaseGCP Cloud Storage – Envelope encryption is the defaultAzure Key Vault – Same pattern Every time you see that little "encryption enabled" checkbox on a cloud service — envelope encryption is what's happening under the hood. Now Let's Talk About PKI PKI stands for Public Key Infrastructure. In Part 3, we discussed Asymmetric Encryption — where you have a Public Key and a Private Key. Sounds great in theory. But here's a real problem. The Trust Problem Imagine Rahul wants to send an encrypted message to Priya. Priya shares her public key with Rahul. Rahul encrypts the message with Priya's public key and sends it. But wait — how does Rahul know that the public key he received is actually Priya's? What if a hacker intercepted the communication and swapped Priya's public key with their own? Rahul encrypts with the hacker's public key → hacker decrypts → reads the message. This is called a Man-in-the-Middle (MITM) Attack. We need someone that both Rahul and Priya trust, who can say — "Yes, this public key truly belongs to Priya." That trusted someone is called a certificate authority (CA). PKI — The Complete Picture PKI is a system made up of several components that together solve the trust problem. Let's go one by one. 1. Certificate Authority (CA) A CA is a trusted organization whose job is to verify identities and issue digital certificates. Think of them like the government passport office — they verify who you are and give you an official identity document (passport). Well-known CAs in the real world: DigiCert, Let's Encrypt, GlobalSign, Comodo. Your browser/OS comes pre-loaded with a list of trusted CAs. That's how your browser automatically trusts websites — because their certificates were signed by a CA your browser already trusts. 2. Digital Certificate A Digital Certificate is like a government-issued ID card for websites (or people or servers). It contains: The owner's name (e.g., google.com)The owner's Public KeyThe CA's name (who issued it)Expiry dateA digital signature from the CA When you open https://google.com — your browser checks Google's certificate. It sees the CA that signed it. It checks if that CA is in its trusted list. If yes — green light, connection is secure! That's the lock you see. 3. Digital Signature We talked about hashing in Part 3 — that you cannot reverse a hash. Digital Signatures use this + asymmetric encryption together in a clever way. When a CA wants to sign a certificate, it: Takes the certificate content and hashes itEncrypts that hash with its own Private Key That encrypted hash = Digital Signature. Anyone can verify the signature using the CA's Public Key (which is publicly available). If the decrypted hash matches the actual certificate content → the certificate is genuine and untampered! Think of it like a wax seal on an envelope. Anyone can see the seal, but only the king's ring (private key) could have made it. 4. Certificate Chain (Chain of Trust) In the real world, CAs have a hierarchy: Root CA → Intermediate CA → Your Website Certificate The Root CA is the ultimate trusted authority. It signs Intermediate CAs. Intermediate CAs sign individual website certificates. This chain is called the Chain of Trust. Why this hierarchy? Security! Root CA private keys are kept in ultra-secure, air-gapped hardware. They are almost never used directly. Intermediate CAs do the day-to-day certificate signing. If an Intermediate CA is ever compromised, it can be revoked without affecting the Root CA. Where Is PKI Used in Real Life? PKI is literally everywhere. You just don't see it because it works silently in the background. 1. HTTPS Websites (SSL/TLS) Every https:// website uses PKI. When you open your bank's website — a PKI handshake happens in milliseconds: Your browser asks the bank's server for its certificateThe bank's server sends its Digital CertificateBrowser verifies the certificate using the CA's public keyIf valid → browser and server agree on a secret key (using asymmetric encryption)All further communication uses that secret key with AES (fast symmetric encryption) That's TLS in a nutshell. And PKI is the backbone of all of it. 2. Email Signing (S/MIME) When your company sends you a digitally signed email — PKI is involved. The sender signs the email with their private key. You verify it with their public key from their certificate. You can be sure the email is genuinely from them and was not tampered with in transit. 3. Code Signing When you download an app or a software update — how does your phone/OS know it's legit and not malware? The developer signs the app with their private key. Your phone verifies it using the developer's certificate. This is why on Android you see the "Install from unknown sources" warning — no valid certificate found! 4. Government and Banking Your Aadhaar card, digital signatures on GST filings, net banking OTPs — all of these use PKI infrastructure behind the scenes. India's government runs its own CA called CCA (Controller of Certifying Authorities) under the IT Act. 5. VPNs and Internal Company Networks When you connect to your company's VPN, PKI certificates are used to verify that you are connecting to the genuine company server and not an imposter. Terms We Have Learned So Far (All 4 Parts) Cryptography, Algorithm, Plain Text, Key, Cipher TextSymmetric Encryption, Convergent Encryption, Initialization Vector (IV), Searchable EncryptionHashing, Hash/Digest, Avalanche EffectSalt, Rainbow Table Attack, Confirmation AttackAsymmetric Encryption, Public Key, Private Key, RSADEK (Data Encryption Key)KEK (Key Encryption Key)Envelope EncryptionKey Rotation, DeduplicationPKI (Public Key Infrastructure)Certificate Authority (CA)Digital CertificateDigital SignatureChain of TrustTLS/SSL That's a solid vocabulary now! Coming in Part 5... (Part 5 is in progress — stay tuned!) In the next part, we will gossip about: SSL/TLS Deep Dive – The full step-by-step TLS handshake explained simplymTLS (Mutual TLS) – How microservices talk to each other securelyHSM (Hardware Security Module) – The physical vault where the most sensitive keys liveZero-Knowledge Proofs – Proving you know something without revealing what it isAnd more... Stay tuned for Part 5! If you liked this blog, do give it a like and share it with someone who you think should learn this. Let's spread the knowledge! Read the previous parts here: Part 1 & 2 Part 3

By Sahil Aggarwal

The Silent Container Death: A TCP Dial That Never Times Out

A pod goes into CrashLoopBackOff. You pull the logs expecting a stack trace, a panic, an error string - anything that points you somewhere. Instead, you get one line: Plain Text Loading config... And then nothing. No error. No exit message. The container is just gone, and a few seconds later it’s back, prints the exact same line, and disappears again. Magic. This is the story of chasing that silence to its root cause. TCP connection that was never going to succeed, and never going to fail either. At least not on any timescale a Kubernetes health check was willing to wait for. The Setup We were migrating backend services from a legacy message queue to Kafka. The new consumers ran side by side with the old ones in a “shadow mode.” That let us compare behavior before the real cutover. Part of that work meant pointing a dev environment at a managed Kafka cluster. (Think AWS MSK — the specifics don’t matter here.) We also updated the broker endpoint in config. In shadow mode, we send to both old and new queues, but only one of them processes the message. The other queue infrastructure just logs what it receives. The change looked trivial: swap one connection string for another, restart the pods, watch them come up. Instead, every pod that touched Kafka went straight into a crash loop. The only clue was that single “Loading config” line. Repeated forever. Why “No Error” Is the Error The instinct when a service crashes is to look for what it logged right before dying. Here that instinct is a trap. The absence of any further log output isn’t a hint — it’s the symptom itself. Two things had to be true simultaneously for this to happen: Something blocked the process long enough that Kubernetes’ health checks gave up on it and sent SIGKILL.Whatever the process wanted to log about being blocked never made it out of its internal buffers before the kill. That second point matters more than it looks. Go’s standard logger writes to os.Stdout. How a container runtime attaches to that stream determines whether output appears immediately or sits in a buffer. Buffering is common under load, or when the write target isn’t a real TTY. Consider a process blocked inside a library call, say dialing a broker. It never gets back to the point in its code where it would flush or print the next line. SIGKILL doesn’t give a process the chance to clean up. Whatever was sitting in a buffer is gone. From the outside, a service that’s actually deep in a hung network call looks identical to one that exited silently. Both just print “Loading config” and stop. The lesson here generalizes past Kafka. If a container’s logs stop dead with no error and no clean shutdown message, assume a hang-then-kill. Not a fast crash. Until proven otherwise. Reaching for the Network Layer Once “look at the application logs” stopped being useful, the next step was to get underneath the application entirely. Shelling into a node and watching the raw traffic (tcpdump) tells you what actually happened at the OS level. So does tracing the process’s syscalls with strace. Neither depends on whether the application ever got to log anything about it. What that showed: a TCP handshake that started and never finished. A SYN packet went out toward the broker; no SYN-ACK ever came back, and critically, no RST came back either. That distinction is the whole story. Connection refused is fast and loud. The remote host, or a firewall in front of it, actively sends back an RST packet. Your client’s connect() call fails almost immediately.Connection blackholed is slow and silent. Packets go out, and nothing comes back. The OS has no way to know if the remote end is down, unreachable, or just very far away. So it retransmits the SYN a few times with exponential backoff, then gives up. The kernel’s default TCP connect timeout can be well over a minute. In this case, the broker endpoint we’d configured was a private, VPC-internal address. It was reachable from some parts of the network, but not from the specific node group these pods landed on. No security group or routing rule was actively rejecting the connection – the packets were simply going nowhere. That’s the worst kind of network failure to debug from inside an application. Everything about it looks like the process is just slow, right up until it isn’t. Where the Health Check Made Things Worse None of this would have been quite so opaque if the failure had surfaced immediately. But the service’s startup path connected to Kafka before reporting itself healthy. On top of that, the Kubernetes startup probe carried a generous timeout, meant to avoid flapping on slow boots. That combination left the platform with no opinion about what was wrong. It just saw a container that hadn’t become healthy in time. So it did the only thing it can do here: kill it and try again. The pod restart count climbed. The backoff delay between restarts grew too - Kubernetes doubles it after repeated failures, up to roughly five minutes. Every fix we tried afterward seemed to take forever to take effect. That’s because we were still watching a container that hadn’t actually restarted yet. It was just waiting out its backoff window. Deleting the pod outright forced an immediate restart. That turned out to be the fastest way to test each hypothesis, rather than waiting for the backoff timer. The Fix, and the More Useful Part The actual fix was almost anticlimactic: switch to the broker’s public endpoint. In a real production setup, you’d instead fix the VPC routing or peering. That makes the private endpoint reachable from every node group that needs it. Once the TCP path was real, the connection succeeded instantly, and the crash loop stopped. The useful part isn’t the fix. It’s the checklist that could have saved us time: Handy Checklist Treat “one log line then silence” as a hang, not a crash. A clean crash logs an error. A silent one usually means something upstream killed the process mid-blocking-call.Go to the network layer early, not last. tcpdump or strace on the affected node will show you a stuck SYN in seconds. That’s far faster than adding print statements and waiting through several crash-loop cycles.Know the difference between “refused” and “blackholed” in your bones. An RST means someone answered and said no. Check credentials, ports, and application-level config. Silence means the packet never arrived. Check routing, VPC peering, security groups, and whether you’re using the right endpoint for the network you’re actually in.Set explicit, short connect timeouts in your client libraries. Don’t let a startup path inherit the OS’s default TCP connect timeout. The OS optimizes that default for general robustness, not for failing fast during a health check window.Make sure your logger flushes before anything that can block indefinitely. If a call to an external system can hang, log “attempting to connect to X” first. Then make sure that line is actually out the door, synchronously if necessary, before making the call. It costs you nothing when the call succeeds and saves you hours when it doesn’t.When you’re mid-debug, delete the pod instead of waiting out the backoff. Kubernetes’ exponential backoff on repeated CrashLoopBackOff restarts is helpful in production and actively annoying when you’re iterating on a fix. None of this is exotic — it’s TCP fundamentals and container basics that everyone technically knows. What makes it worth writing down is how convincingly a blackholed connection disguises itself as an application bug. Right up until you stop looking at the application and start looking at the wire.

By Alexander Fo

Predict, Repeat, Improve: Deterministic Simulation Testing Explained

It’s 2 AM. Your phone buzzes, the on-call alert flashes, and suddenly you are staring at a production outage that makes no sense. Following the logs, you get a hint: when you rerun the same scenario in staging, everything behaves perfectly. None of the quality gates/QA pipelines catch it, chaos experiments didn’t reproduce it, and now a ghost is chased that only appears when the system is under real-world pressure. Distributed systems are notorious for these “phantom failures” — rare timing-dependent bugs that surface unpredictably and vanish just as quickly. They are the kind of dreaded incidents that keep engineers awake at night because they are unreproducible. Take a real-world example: A service once crashed because two nodes tried to become leader at the exact same millisecond. In staging, the timing never aligned that way, so the bug remained invisible. But in production, under heavy load, it just happens, sending the system into chaos. Engineers spent days trying to recreate the failure, but without a deterministic replay, it's pure luck to get a reliable reproduction. Even when it happens, engineers may not be sure what caused it or how to reproduce it deterministically. Enter deterministic simulation testing (DST). DST builds a fully controlled, re-playable simulation of your system’s world — nodes, clients, clocks, network delays, partitions — so that even the most elusive bugs can be identified, captured, replayed, and studied. In this article, we will uncover how DST can transform those unpredictable 2 AM incidents into predictable, debuggable coordinates — giving you a new way to tame the chaos of distributed systems. Deterministic Simulation Testing Definition Deterministic simulation testing (DST) is a software testing methodology that places the system under test within a fully controlled, simulated environment. All sources of non-determinism — system clock, thread scheduling, network, disk — are intercepted and made deterministic. The key property is that, for a given initial seed and configuration, the entire execution is reproducible. The same sequence of events, faults, and outcomes will occur on every run with that seed. Key Concepts Breaking down the above definition, below are the key concepts for DST: Determinism → The system’s behavior is purely dependent upon its initial state and the seed. All non-deterministic sources are simulated to achieve determinism.Simulation → The system is run in a virtual environment that can simulate faults and control the passage of time. E.g., controlled clock skew introduced across various nodes, added network delays to achieve out-of-order event delivery.Reproducibility → Any failure or bug found during simulation can be reliably reproduced by rerunning the simulation with the same seed.Scenario Exploration → By varying the seed and/or simulation parameters, DST systematically explores a vast range of possible execution paths and failure scenarios. How DST Works Let's consider a simple scenario where two users update and read the same record in a very short interval. User 1 updates a record with a new value at time instance T0, and User 2 reads the same record at time instance T1. Note that the interval between T0 & T1 stays the same. In an ideal case (i.e., scenario 1), the new value is updated or written immediately, i.e., without any delay. Thus, when User 2 reads the same record at T1, it is able to read the latest value. In scenario 2 suppose the write is delayed due to network partitioning, disk write etc. User 2 thus sees the old value of the record even if it reads the value at the same time instance T1. Although a stale read may look trivial, it may lead to workflow halt, process crash, etc. in a complex real-world system. Imagine what could happen in a real-world system where: Multiple processes are scheduled for execution, within and across nodes.Multiple network calls are made between several nodes.Multiple operations are performed by several distributed processes on a single disk. Traditional testing strategies or frameworks are inherently constrained and thus can’t simulate such delays or faults. Because of this, it's nearly impossible to identify, catch, reproduce, or debug issues arising from such situations — rendering them unreliable or, at best, non-deterministic. To achieve determinism, the testing framework must take total control over the environment to intercept and manage all external interactions as described below: Controlled scheduling → Instead of relying on the operating system’s unpredictable thread/coroutine scheduler, the simulator provides its own deterministic scheduler. Thus ensuring various scheduling combinations are simulated.I/O mocking → All network calls, disk writes, and clock queries are routed through the simulator, allowing it to inject latency, drop packets, or change the time (e.g., clock skew).Single-threaded execution → Many DST frameworks run the entire distributed system stack within a single thread, completely stripping away the chaotic, unrepeatable nature of multi-threading. Thus, by eliminating real-world “flakiness,” DST allows developers to reproduce chaotic distributed system bugs with perfect precision, thanks to its inherent ability to replay any failing execution: Seed-based replay → The same seed reproduces the exact sequence of events, making debugging tractable.Time-travel debugging → Some platforms (e.g., Flashback) allow stepping backward and forward through execution, inspecting state at any point for a granular view of the system. DST Implementation Approaches and Architecture Patterns Below are two approaches for DST. Pluggable Non-Determinism Design the system so that all non-deterministic components (clocks, I/O, etc.) are pluggable. This strategy is used by TigerBeetle. This requires: Abstracting all system interactions behind interfaces.Providing both real and simulated implementations.Ensuring that the same codebase can run in both production and simulation by swapping implementations at startup. Pros: Deep control and minimal divergence between test and production code. Suitable for greenfield systems. Cons: Requires significant upfront design and is challenging to retrofit into existing systems. Deterministic Hypervisors and Emulation A more recent and flexible approach is to run unmodified binaries inside a deterministic hypervisor or emulation layer. This strategy is used by Hermit and Weave. Pros: Can test existing systems without code changes; language-agnostic; simulates the entire stack. Cons: May have performance overhead; some system behaviors may escape determinism if not fully intercepted. Benefits System employing DST benefits as below: Identify and reproduce rare failures → DST allows engineers to replay the exact sequence of events that led to a bug. This eliminates the frustration of “flaky” issues that appear inconsistently, making debugging far more reliable. Moreover, DST helps find bugs in execution paths unreachable by example-based tests.Improved developer productivity → Bugs are easier to reproduce, debug, and fix; less time spent on war rooms and emergency triage.Improved confidence in correctness → DST validates critical invariants (like consensus, failover, or transaction consistency) under controlled simulations. Engineers gain assurance that core distributed protocols behave as expected even under stress. Thus, preventing rare bugs from reaching production, increasing system uptime and user trust.Scalable debugging for complex systems → In microservice or event-driven architectures, DST helps tame the exponential growth of possible interleavings by focusing on deterministic seeds. This makes large-scale debugging more tractable. Challenges and Limitations While DST provides unparalleled confidence, it requires significant architectural investment. Retrofitting DST into existing systems may require significant refactoring. It can be highly intrusive, requiring developers to write custom code or frameworks, as production code often cannot rely on external third-party libraries that invoke un-mocked I/O or system calls. Moreover, DST requires careful modeling of external systems to avoid missing integration bugs. Ensuring sufficient coverage without combinatorial explosion is a major challenge — especially in modern systems with multiple integration points. Below are gaps in tooling that limit DST outcomes: Language and platform support → Not all languages and runtimes have mature DST frameworks.Hypervisor limitations → Deterministic hypervisors may not support all system calls or hardware features. DST Comparison and Applicability DST vs. Chaos Engineering DST is proactive and enables perfect reproducibility. It is best suited for development and pre-production, catching bugs before they reach users. It can simulate production chaos in minutes, and every failure is a permanent regression. Chaos engineering is reactive, non-deterministic, and validates the behavior of real deployments. It is essential for catching issues arising from real infrastructure, misconfigurations, or dependencies that simulation cannot model. However, it cannot guarantee coverage or reproducibility, and carries the risk of impacting users. In essence, both DST and chaos engineering are complementary to each other and are necessary for comprehensive reliability. DST in Functional vs. Performance Testing Functional Testing DST is ideally suited for functional testing. Validates correctness under all possible interleavings, failures, and workloads.Checks invariants, safety properties, and liveness under stress.Finds rare, timing-dependent bugs that are invisible to example-based tests. Performance Testing DST is not primarily designed for performance testing. The simulated environment does not reflect real hardware performance, network latency, or throughput.Time is virtualized and compressed; I/O is in-memory.Performance metrics (latency, throughput) measured in simulation may not correspond to real-world values. However, DST can be used to: Validate performance-related invariants (e.g., absence of deadlocks, progress under load).Simulate pathological scenarios (e.g., extreme contention, resource exhaustion) to observe system behavior. Recommendation Combine DST for functional correctness with real-world performance and benchmarking suites for comprehensive validation. DST Applicability to AI/ML and Agentic Systems AI/ML systems, especially those based on large language models (LLMs) and agentic workflows, are fundamentally non-deterministic. This makes traditional testing and debugging extremely difficult, with “heisenbugs” that vanish when observed. DST can be adapted to AI/ML systems by creating controlled, simulated environments for agents to operate in. Or using a hybrid approach of combining deterministic components (rule-based logic) with LLM-driven reasoning, using seeds to replay failures. Case Studies FoundationDB, with its deterministic simulator tool, achieved legendary reliability by running trillions of simulated CPU-hours, finding and fixing every known bug before production.TigerBeetle built a Viewstamped Operation Replication simulator (VOPR) to simulate financial transaction systems, catching subtle bugs in consensus and replication.Ethereum Merge used Antithesis to test the transition to Proof-of-Stake, simulating multiple client implementations in a deterministic environment. Conclusion Deterministic simulation testing (DST) represents a paradigm shift in the testing and validation of distributed systems. By enabling exhaustive, reproducible exploration of the vast state space of concurrent, failure-prone systems, DST empowers engineers to find and fix the rarest and most pernicious bugs before they reach production. Its integration with property-based testing, fuzzing, and fault injection, combined with advances in deterministic hypervisors and simulation frameworks, has made DST accessible to a growing range of systems and organizations. While DST requires significant engineering investment, careful system design, and ongoing maintenance, its benefits in reliability, developer productivity, and user trust are profound. As distributed systems continue to grow in complexity and AI/ML systems become more agentic and autonomous, the need for rigorous, deterministic validation will only intensify. The future of DST lies in deeper integration with formal methods, smarter state-space exploration, and broader applicability to AI/ML and hybrid systems. Organizations that embrace DST, alongside complementary techniques like chaos engineering and formal verification, will be best positioned to deliver robust, trustworthy, and resilient distributed systems in the years ahead. DST Tools and Frameworks Deterministic simulation testing (DST) tooling is still a niche but growing ecosystem. Each has a unique focus — ranging from language-level deterministic runtimes to full-stack hypervisor-based reproducibility. Based on the specific needs a single or combination of them can be picked up. References and Further Reads Taming Chaos — DSTSquashing the Heisenbug with DSTAntithesis — DSTPhil Eaton — DSTRedstone — DST FrameworkResonate — DSTJespen | TickLoom

By Ammar Husain

CORE

Performance

DZone's Featured Performance Resources

Monthly Top Performance Experts

The Latest Performance Topics