Architecting for <1s Latency: Managing Eventual Consistency in Distributed Search Platforms

At a logistics visibility company, the shipment state can change thousands of times per second due to the massive scale of operations. Such changes take place when, for instance, a carrier scans a package or a route changes. Within milliseconds, those events are ingested by Kafka into our search index. The key requirement for these logistics operations is the ability to see the current status, not the one from five seconds ago

For a long time, our engineering team couldn’t achieve that level of precision. The cluster appeared healthy, writes were acknowledged, yet search results became stale. The issue we faced and addressed is called an eventual consistency problem. The insidious part is that it does not trigger alert fires, and the problem remains invisible until a user reports it. The eventual consistency problem can show up at each layer of the stack, including the replication quorum, the indexing engine, the cache tier, and the background repair process. Each of them introduces its own staleness window. Total end-to-end lag can be pushed past 1 sec with all systems seemingly green. 

In this article, we will cover each layer to understand what causes the staleness, and cover the specific settings we use to keep our lag time under 1 second in production

Replication Quorum and Freshness Trade-Offs

The first consistency decision takes place on the write path. In a replicated system, a write can be acknowledged in three cases: 1) after one replica accepts it; 2) after a majority of replicas accept it; 3) after all replicas confirm it. These options decide how much freshness the system receives in exchange for write speed. For instance, if the platform waits for only one replica, the write is confirmed faster, yet other replicas may still be behind. If the system waits for more replicas, then the write takes longer, but also the odds of reading stale data are significantly lower. Hence, software engineers need to decide in which cases the trade-off is less harmful to the system. 

Some shipment data is more time-sensitive than other data. These are the current statuses, locations, route deviations, and delay reasons; they need to be up to date because fresh information builds user trust in the logistics product. On the other hand, historical metadata can tolerate a longer status delay. Because there are different levels of urgency for keeping the data fresh, using the same consistency rule isn’t productive. Hence, our approach is based on a fresh classification of requirements. Critical shipment-state fields are handled strictly, while less urgent ones follow a slower freshness path. 

Event Ordering in Kafka

Once the write has been accepted, the next issue is event ordering. It occurs because logistics systems are updated by various sources, such as carriers, GPS devices, partner APIs, and internal services. Due to this variation, some events are duplicated, while some are received out of order.

For instance, let us analyze the following delivery status breakdown: 

T1: status = IN_TRANSIT
T2: status = DELAYED
T3: status = OUT_FOR_DELIVERY

If T3 reaches the index before T2, and the indexer applies both, the document can move backward to an older state. As a result, the platform has processed the event, but the indexed shipment is wrong. In real life, it can look like this: in a tracking app, the delivery status shows DELAYED even though the order is OUT FOR DELIVERY. The client received incorrect information and, as a result, did not receive the order at the specified time. 

To prevent this issue, every event needs a timestamp. The indexer should compare the incoming version with the version already stored in the index. From the code point of view, the process is as follows: 

if incoming_event.version > indexed_document.version:

    apply update

else:

    discard event

Kafka partitioning is also necessary for maintaining correct event ordering. Namely, events for the same shipment must be keyed by shipment ID. This approach allows engineers to keep updates in the same partition and helps preserve order inside the ingestion path. 

Search Index Refresh 

The third source of fresh-data delay is the search engine itself. For instance, a document can be indexed successfully yet be invisible in search results. In Elasticsearch systems, the refresh interval controls how quickly newly indexed data becomes searchable. 

Like replicas, these intervals are trade-offs. Long ones are beneficial for indexing efficiency. Shorter refresh intervals improve data freshness but also increase CPU pressure. In our case, for a platform that processes thousands of shipment updates per second, forcing a refresh after every write isn’t feasible from an architectural point. Hence, selective freshness is the best option. Critical shipment-state updates must be visible immediately, whereas less important enrichment fields can go through the normal refresh cycle. 

Cache Invalidation

Caching can make stale search results even more challenging to detect. For instance, a query may return in 80 ms because it comes from cache. If that cached response contains an outdated shipment status, the system returns the wrong answer more quickly. One solution we use to prevent this issue is to tie cache validation to events: when a shipment changes, the corresponding cache entries must be invalidated. 

The challenge is that search results are often cached only by query: 

status: delayed
carrier: DHL
region: Midwest
customer_id: 12345
sort: updated_at desc

One shipment update can affect many cached result sets. Perfect invalidation is expensive; the cache policy should be tiered. 

Background Repair 

Background repair is not the most obvious cause of stale search results, but it still needs to be considered. Consistent systems require maintenance of retries, reindexing, replica recovery, and reconciliation jobs. This is because events may still fail, and replicas may drift. These maintenance processes help the search operations work reliably. 

However, the maintenance procedures may diminish data freshness when competing with live traffic. To avoid this problem, we set limits on the batch size of repair jobs, worker count, rate limits, backoff during peak traffic, and circuit breakers when index lag becomes too high. That helped us save the system capacity needed for live shipment updates while still performing background work. 

Measure Data Freshness 

Finally, API latency alone is insufficient to address stale results. A search API can respond in 150 ms while returning data that is three seconds old. In such cases, the engineering team tracks end-to-end index lag: 

index_lag = search_visible_timestamp - source_commit_timestamp

The metric indicates how long it takes for a committed shipment update to become visible in search. It should also be broken down by stage: Kafka delay, consumer lag, indexer processing time, search refresh delay, cache invalidation delay, and repair queue depth. Without this breakdown, we would guess where the problem is. It is also important to highlight that, for sub-second search, freshness must be visible on the dashboard alongside latency. 

Conclusion 

In conclusion, eventual consistency is a normal aspect of creating distributed systems. The logistics visibility platform we’re developing monitors and manages it through Kafka, indexing, caching, and the repair process. For the engineering team at Project44, the key takeaway from addressing this challenge at scale is to treat data freshness as a vital product metric. Handling consistency boundaries gracefully may become a business differentiator between resilient distributed platforms and brittle ones.