Building Cross-Team SLO Contracts for Performance Accountability

If you have ever developed a popular website in a microservice architecture, then most likely you have come across this case when, at first, your latency seems to increase. Then you check your dashboard and notice that the P90 latency has increased by 300ms in the past two weeks. It is very likely you will want to analyze the individual spans and see that one of your upstream dependencies, which belongs to another team, is slower. You could write them a ticket or even page them to fix the problem. They agree to it, but they also say that all is well on their end. Still, your service-level objective (SLO) is being violated. 

This is exactly the accountability gap that latency alone cannot solve. The page owner is the one who is responsible for the end-to-end latency of the page and may also have some dependencies which they do not have complete control over. However, the dependency owner does not have a formal agreement to maintain a specific latency profile per service, and quite fairly so. Everyone is technically following their own objectives while the customer experience is going down the drain.

For a Tier-0 consumer surface at Doordash, I was in charge of running a latency maintenance, improvement, and optimization plan for multiple years. This page was reliant on more than twenty backend microservices, each one being owned by a different team. Besides all the optimizations, a good SLO contract between our services and other services is what helped maintain the performance.

What exactly is an SLO contract?

You can think of an SLO contract as a formal written contract specifying the terms and conditions between a service that is a consumer and one of its dependent services. It may identify a single or several endpoints that have a latency budget at a given percentile, how that budget is measured, and the actions following a failure to meet the budget. It is usually agreed upon by the teams' engineering managers and is possibly stored in a Google Doc, GitHub, or any other durable location. Essentially, it transforms a vague expectation into a definite commitment between teams. The consumer team is given a reliable figure for them to allocate their own e2e latency budget, whereas the dependency team receives a clear understanding of what they need to defend and also the liberty to optimize everything else. An SLO is the minimum performance level that one team guarantees to another.

Elements of a contract

Make it brief. A good contract should be contained on a single page. Nobody reads lengthy contracts, and contracts that are not read do not lead to a change in behavior.

The main components that need to be included in such a contract are (but not limited to): the endpoint or RPC method that is going to be measured, the latency target and percentile, the method used for measuring and the exact name of the metric, the traffic conditions under which the contract is applicable, the duration of the contract, the escalation path when the contract is violated. 

Here's an example: 

SLO Contract: Recommendations API -> Promotions Service

Endpoint:       POST /v1/promotions/lookup

Target:         p95 latency ≤ 800ms

Measurement:    server-side histogram, recorded at Promotions Service

                metric: promotions_lookup_duration_seconds, bucket 0.8

Conditions:     traffic up to 12,000 RPS, payload size ≤ 4KB

Effective:      2024-Q3 through 2025-Q2 (renewed quarterly)

Escalation:     If breached for two consecutive weeks at p95,

                Promotions on-call posts in #recs-promotions-slo with

                root cause within 5 business days. Sustained breach

                (4+ weeks) triggers a joint review with both EMs.

Signed:         [Recs EM]   [Promotions EM]

SLO Contract: Recommendations API -> Promotions Service

Endpoint:       POST /v1/promotions/lookup

Target:         p95 latency ≤ 800ms

Measurement:    server-side histogram, recorded at Promotions Service

                metric: promotions_lookup_duration_seconds, bucket 0.8

Conditions:     traffic up to 12,000 RPS, payload size ≤ 4KB

Effective:      2024-Q3 through 2025-Q2 (renewed quarterly)

Escalation:     If breached for two consecutive weeks at p95,

                Promotions on-call posts in #recs-promotions-slo with

                root cause within 5 business days. Sustained breach

                (4+ weeks) triggers a joint review with both EMs.

Signed:         [Recs EM]   [Promotions EM]

The hard part: How to negotiate a contract

Creating the contract is really not a challenge; you can even reference past data and also future plans. But the difficult part is to get all the teams to agree with the numbers. I have discussed and agreed on these quite a bit, and I want to share some points that really worked.

Begin with your e2e budget. If your web page has a 1.5-second p95 latency target, and the user request flow involves a call to four downstream services that run one after another, then those four services together can take no longer than about 1.2 seconds, allowing time for network, serialization, and your own processing. Do the math before the meeting. If you haven't figured out the breakdown of your own budget, then you're probably not ready to negotiate.

Have your data ready and explain the effect to everyone in a comprehensible way. Get the actual current latency distribution for the dependency endpoint for the last 30 days. Display p50 p95 p99. Then explain what the actual penalty is. For instance, if a 300ms delay on your page results in a loss of $20M in company annualized revenue, present it and provide evidence. This is a figure that both engineering teams and management can agree upon. Understanding each other's position is very important.

Operationalizing it

Signing a contract and then forgetting about it is not a situation that you want to end up with. There are obviously certain things that you must do, like having a dashboard with the right metrics, relevant metrics, and the histogram bucket boundaries for these metrics. I would suggest reviewing compliance regularly in a monthly performance review meeting. The team that depends on you for its review of all the outbound contracts is its own ops review.

You should consider treating any breach as a real signal. In fact, the escalation path should be triggered in case a contract is breached. A breach should be taken seriously, and a lightweight postmortem might be justified so that things could be done to root cause and fix what was broken. Actually, one of the things that most contracts have is a clause that states that they will expire. At renewal, the two teams will examine the compliance of the contract over the entire period, determine whether the assumptions that they made at the time of signing are still valid, and then decide if they want to renew the contract as it is, tighten the contract, loosen the contract, or terminate the contract. If the dependency is no longer the critical path that ultimately contributes to latency, then it might be okay to terminate the contract.

The compounding effect

Composing your initial contract is really the most challenging part. A brand new contract involves a lot of explaining, and the ground rules are set. But the second one is a piece of cake. At the platform where I was, the regimen was initiated with a lone contract between the homepage team and a single service downstream. 

Very quickly, this method proved to be the best fit for the entire organization whenever there was an inter-team latency dependency on a critical path. The contract-supported payload size tracking tool become a piece of infrastructure utilized by each and every team; and our team was not the only one using it. Even the contract system itself turned out to be the reference model for other departments of the company when they got to the point of formalizing their own performance accountability.

When should you not use this

I wouldn't recommend SLO contracts for every situation. They're overhead, and overhead has to be justified. Skip them for small organizations. If the consumer and dependency are owned by the same team or by two teams in the same group with the same manager, a contract adds process without changing incentives. A regular sync meeting is fine. Skip them for fast-changing dependencies. If the dependency service is in early development and its API surface is still in flux, wait until the dependency stabilizes. Skip them for low-criticality paths. The endpoint has to be on a critical path that matters to a real SLO.