Fast Service but Slow Client: When a Sidecar Makes Sense
Sidecars centralize client logic across services, improving consistency and performance at the cost of added operational complexity.
At a previous organization, we had an internal service that was called from several fairly important request paths. It wasn't doing anything glamorous. Given some request context, it returned tenant-related metadata that other services needed before they could continue processing.
The service was designed to be fast. Its latency was usually in the single-digit millisecond range, including at the higher percentiles, and it had very high availability. It was also called frequently enough that we paid attention to even relatively small changes in latency.
Then one of the consuming teams started complaining that it wasn't performant.
Their measurements were hard to reconcile with ours. The service team's dashboards looked healthy, but from the application team's point of view, the dependency was clearly adding much more latency than it should. There were also failures they associated with the service.
This is one of those situations where looking at another dashboard doesn't help much. We had to follow how that particular application was actually using the service.
Once we did that, the discrepancy made more sense. Some operations that could have been performed concurrently were happening sequentially. The client also wasn't following all the retry and failure-handling practices that had evolved around the service.
So the service could report a perfectly healthy P99 while the application still experienced poor end-to-end behavior.
Both measurements were correct.
That incident eventually pushed us toward a sidecar, but the interesting part wasn't really the sidecar itself. The more important question was why every consuming team was responsible for independently knowing the best way to communicate with this service.
Service Latency Is Not Client-Perceived Latency
People sometimes talk about service latency as if it were a single number. It isn't.
Imagine a service operation takes 4 or 5 milliseconds. That looks great on the server dashboard. Now imagine the application needs several independent pieces of information and happens to retrieve them one after another:
lookup A -> wait
lookup B -> wait
lookup C -> wait
continue
A simplified implementation might look like this:
Tenant a = client.lookupA(id);
Tenant b = client.lookupB(id);
Tenant c = client.lookupC(id);
Nothing about that code looks outrageous. In fact, this is the kind of implementation that can survive code review because each individual call is reasonable.
But if the calls are independent, the application is unnecessarily adding their latencies together. It might instead do something conceptually closer to:
Future<Tenant> a = asyncLookupA(id);
Future<Tenant> b = asyncLookupB(id);
Future<Tenant> c = asyncLookupC(id);
waitForAll(a, b, c);
Now the elapsed time is closer to the slowest operation instead of the sum of all three.
Our real implementation was obviously more complicated than those few lines, but this was one of the issues we found: the calling pattern mattered almost as much as the performance of the service itself.
Retries had the same problem. Connection management had the same problem. Unnecessary repeated calls had the same problem.
If an application makes five remote calls when it could have made three, there is nothing the server can do to make those other two disappear from the application's latency.
That sounds obvious when written down. In a large system, it is surprisingly easy to lose sight of it because the client and server are usually owned by different teams looking at different telemetry.
We Could Have Just Fixed That Client
Initially, that was the straightforward option: talk to the application team, point out which calls should be parallelized, review the retry behavior, fix the implementation, and move on.
There was nothing technically wrong with doing that. The problem was that it fixed exactly one consumer.
We had multiple internal applications using this service. Over time, each team had learned some version of the integration rules:
- Use these timeouts.
- Retry this category of failures.
- Don't retry that category.
- Make these requests in parallel.
- Reuse connections.
- Avoid this unnecessary lookup.
- Handle this response differently
Some teams followed nearly all of those rules. Some didn't. A new team could come along six months later and make a completely reasonable implementation that still missed two or three important details.
I have seen this pattern in enough internal platforms that I don't particularly like relying on documentation as the primary mechanism for enforcing runtime behavior.
Documentation is useful. It is not an execution environment.
If something materially affects reliability or load on a platform service, I would rather encode it somewhere than repeatedly tell teams to remember it.
A Client Library Was the Obvious Alternative
The normal way to encode that behavior is a client library. Instead of letting every application make raw calls, the service team publishes a client:
TenantClient client = TenantClient.create(config);
Tenant tenant = client.lookup(tenantId);
Inside the client, you can take care of details the application shouldn't have to think about: timeouts, retries, connection reuse, request shaping, parallelization, and error translation.
For many services, I would stop here. A good client library is considerably simpler operationally than another process running beside every application.
Our complication was that consumers weren't all written in the same language.
Suppose the canonical client is implemented in Go. That is useful for Go consumers, but the Java services still need something. Then another runtime comes along and needs its own implementation. Before long, you can end up with something like:
Service
|
+-- Go client
|
+-- Java client
|
+-- Rust client
|
+-- another client
Now you have several codebases that are all intended to express the same behavior.
Maybe the new retry policy gets implemented in one codebase before the others. Maybe an application stays on an old SDK version because upgrading it also changes fifteen unrelated dependencies. Maybe one implementation handles an edge case differently.
None of this makes client libraries bad. It just means that "we will maintain SDKs for every supported language and keep their behavior equivalent" is a real engineering commitment.
In our case, there was another option: stop putting the intelligent client inside the application process.
Moving the Integration Into a Sidecar
The basic arrangement was simple.
Instead of:
Application
|
| remote request
v
Internal Service
We moved toward:
+-------------------------------------------+
| Application Instance |
| |
| +----------------+ |
| | Application | |
| +-------+--------+ |
| | |
| | local HTTP |
| v |
| +----------------+ |
| | Sidecar | |
| +-------+--------+ |
+-----------|---------------------------------+
|
| remote request
v
+--------------+
| Core Service |
+--------------+
The application interacted with a loca
The application interacted with a local HTTP interface. The sidecar contained the behavior required to communicate properly with the remote service.
That distinction turned out to be much more useful than it might look from the diagram.
The calling application no longer needed to contain all of the service-specific integration logic. From its perspective, the interaction became thin. The sidecar could take care of things such as retries, parallelization, error handling, and other optimizations that the service team understood better.
It also gave us one implementation that could be used by applications written in different languages. A Java service could make the local call. So could a Go service. So could another runtime.
The sidecar didn't care what language the business application used, and the application didn't care what language the sidecar was implemented in.
That was the part we cared about.
The Useful Part Was Ownership
When people explain sidecars, they often start with deployment mechanics: two containers, same pod, localhost communication. All true. But that wasn't the reason I found this particular use case interesting.
The real change was ownership.
Previously, the boundary looked approximately like this:
Service team: owns server implementation
Application team: owns calling behavior, retries, concurrency choices, timeout choices, integration bugs
With the sidecar, more of that shifted toward:
Service team: owns server, owns sidecar, owns recommended client behavior
Application team: calls a small local API
This matters when a service is used widely. Say the service team discovers that a particular class of requests can be combined, or that a certain retry pattern behaves badly during partial degradation. With independently implemented clients, you may need multiple application teams to update multiple codebases.
With the sidecar, the team that owns the service has somewhere to implement that knowledge directly.
The useful property isn't "we have another container." It is "the people who understand the dependency best can control how the dependency is consumed."
That is a very different reason for introducing a sidecar than simply wanting a fashionable Kubernetes architecture.
Even a Well-Written Client Saw an Improvement
One thing I liked about the rollout was that the benefit wasn't limited to the application that had originally complained.
Another consumer was already doing a pretty good job. It followed the integration practices much more closely and saw latency of roughly 1 millisecond for the path being measured. After moving through the sidecar, that number was around 0.3 milliseconds — roughly a 70% reduction.
I wouldn't generalize that number into "sidecars make services 70% faster." That would be a ridiculous conclusion. The result depended on that workload and on what the sidecar was doing.
The interesting part was that even a reasonably optimized consumer still had room for improvement once the integration logic was owned and tuned in one place.
We saw another effect too: fewer unnecessary requests reached the backend. That matters more as the number of consumers grows. If one client performs an unnecessary call, it is noise. If dozens of clients independently reproduce the same inefficient pattern, it becomes platform traffic.
A shared implementation gives you leverage. Fix one behavior and every consumer using that path can benefit from it.
What Retry Logic Looks Like Once You Own It
Retries are a good example because they are easy to get mostly right and surprisingly easy to get badly wrong.
You don't want thousands of sidecars retrying immediately, over and over, the moment a request fails. When the backend is already unhealthy, that turns clients into traffic amplifiers.
Once retry behavior lives in a common component, you can define a policy around things such as maximum number of attempts, overall request deadline, retryable versus non-retryable failures, backoff, and jitter.
Simplified pseudocode:
deadline = now + request_timeout
for attempt in 1..max_attempts:
response = call_service()
if response.success:
return response
if response.is_not_retryable:
return response
if deadline_exceeded:
fail
sleep(backoff(attempt) + jitter())
I'm deliberately not presenting this as a production-ready retry algorithm. The point is that you now have one place where the policy exists.
Without that, Team A may retry twice, Team B five times, Team C not at all, and Team D may have copied an example somebody wrote two years ago.
That kind of variation is sometimes fine. For a heavily-used internal dependency, it may not be.
Sidecars Also Create New Problems
This is where I think discussions about sidecars sometimes become a little too enthusiastic.
You are adding another process. If your application has 500 instances, you may now have 500 sidecars. If it has 2,000 instances, you may have 2,000 sidecars. That has consequences.
Imagine, purely as an example, that a sidecar uses 50 MB of memory:
50 MB x 1,000 instances = ~50 GB
A 50 MB helper process sounds tiny when discussing one pod. It looks different after multiplying it across an entire fleet. The same applies to CPU.
For a containerized application, an illustrative deployment could look something like this:
apiVersion: v1
kind: Pod
metadata:
name: sample-service
spec:
containers:
- name: application
image: sample-service:1.0
env:
- name: TENANT_API
value: "http://localhost:8081"
- name: tenant-sidecar
image: tenant-sidecar:1.0
ports:
- containerPort: 8081
resources:
requests:
cpu: "50m"
memory: "64Mi"
limits:
cpu: "200m"
memory: "128Mi"
Those resource values are illustrative, not values from the system I'm describing. The important point is that I would treat the sidecar as real production software, with explicit resource accounting, rather than as a free helper that happens to live beside the application.
There Is Another Failure Boundary Now
Before introducing the sidecar:
Application -> Service
After introducing it:
Application -> Sidecar -> Service
The extra layer solves one class of problems and introduces another class.
What if the application is healthy, the backend service is healthy, but the sidecar isn't? From the application's point of view, the dependency is still broken.
Startup and shutdown also require thought. The application shouldn't start handling traffic if it depends on a sidecar that isn't ready yet. During shutdown, you don't want the sidecar disappearing while the main application is still finishing requests.
And then there is debugging. A request that previously had two obvious places to inspect now has three. If an application says a tenant lookup failed, somebody needs to determine whether the application reached the sidecar, whether the sidecar accepted the request, whether the sidecar called the backend, whether the backend timed out, whether the sidecar retried, whether the retry failed, or whether the response was rejected locally.
This isn't an argument against the pattern. It is just the operational bill that arrives with it.
If we are going to add another hop, even a local one, I want trace IDs or request IDs propagated through it and logs that let me reconstruct the path without spending an afternoon grepping unrelated timestamps.
That sounds mundane, but these are the details that determine whether an abstraction feels helpful at 2 PM and terrible at 2 AM.
Sidecar vs. Client Library
I don't think there is a universal rule here.
If you have three Java applications calling a straightforward service and the integration logic is tiny, I would probably build a Java client library and move on. Running another process per application instance just to avoid ten lines of client code would be architectural theater.
A sidecar starts becoming much more interesting when several of these are true:
- Many applications use the same dependency.
- Those applications use different languages.
- The integration logic is more complicated than constructing an HTTP request.
- Retries, concurrency, caching, batching, or connection behavior materially affect the backend.
- Client behavior changes as the service evolves.
- The service-owning team needs more control over how clients interact with it.
At that point, what you call the "client" may already be a small distributed-systems component. Giving it its own process and lifecycle starts to make more sense.
The trade is still real. With an SDK, you have dependency-version and language-maintenance problems. With a sidecar, you have deployment, resource, lifecycle, and observability problems.
You are choosing which complexity you would rather own.
This Pattern Shows Up in Other Places Too
The same reasoning is why sidecars have historically been useful for things such as observability.
An application can write logs or metrics locally, hand them to a collector, and let that collector forward them to a telemetry backend. The business application doesn't need all of the batching, buffering, formatting, and delivery behavior embedded in its own code.
Logging agents, metrics collectors, proxies, and other infrastructure helpers often fit naturally into this model for the same reason our internal client did: the capability needs to live close to the application, but it doesn't really belong to the application's business logic.
What I Took Away From It
The original complaint was basically, "this service is slow." It wasn't.
Then the problem looked like a badly implemented client. That was closer, but fixing one client still wasn't the most useful outcome.
What finally mattered was realizing that we were distributing service-specific expertise across application teams and then being surprised when those implementations behaved differently.
Once you look at the problem that way, a sidecar isn't primarily a deployment trick. It is one way of deciding where knowledge should live. In this case, knowledge about retries, concurrency, request patterns, and other integration details belonged closer to the team operating the service than to every individual consumer.
There is also a platform-engineering lesson here that I still find useful.
A lot of teams talk about providing a "paved road" and then publish a long document telling everyone how to build the road themselves. I don't think that is much of a paved road.
If there is one integration pattern that is safer, faster, and better understood, make that path easy to consume. Sometimes the answer is a good SDK. Sometimes it is an API gateway or proxy. And sometimes, when the client behavior is substantial enough, and the consumers are heterogeneous enough, a sidecar is the cleaner boundary.
For us, that was the useful part of the pattern. Not the extra container. The fact that we could implement the tricky behavior once, put it under the ownership of the team that understood it, and stop asking every application to rediscover the same lessons.
Opinions expressed by DZone contributors are their own.