My team has been working on improving the performance our API and identified a database call as the cause of some problems.

The team suggested three ways to tackle this problem:

1. Scale up the database until it can meet our requirements.
2. Introduce some lightweight caching in the application to reduce the load on the database.
3. Examine the query plan for this database call to find out whether the query can be optimized.

Which of these should we attempt first? There was some intense discussion about this, with arguments made in favor of each approach. What we needed was a simple framework for making decisions about how to improve our system.

This is where the Theory of Constraints (ToC) can help. Originally expounded as a paradigm for improving manufacturing systems, ToC is really useful in software engineering, both when managing projects and when improving the performance of the systems we create.

Theory of Constraints

The preliminary step in applying ToC is to identify the goal of your system. In the case of this API, the goal is to supply accurate data to consumers.

Now we understand the goal of the system, we can define the throughput of the system as the rate at which it can deliver units of that goal (in our case, API responses). We can also define the operating expenses of the system (the cost of servers) and its inventory (requests waiting for responses).

The next step is to identify the constraint of the system. This is the element in the system that dictates the system’s throughput. In a physical system, a useful heuristic is a build-up of inventory in front of this element. In our API, our monitoring helped us pinpoint the bottleneck.

The next three steps give us a sequence of approaches for tackling the constraint:

1. First, exploit the constraint by finding local changes you can make to improve its performance.
2. Second, subordinate the rest of the system to the constraint by finding ways to reduce pressure on it so it can perform more smoothly.
3. Third, elevate the constraint by increasing the resources available to it, committing to additional operating expenses if necessary.

Exploitation comes first because it’s quick, cheap and local. To Subordinate you need to consider the effects on the rest of the system, but there shouldn’t be significant costs involved. Elevating the Constraint may well cost a fair amount, so it comes last on the list.

Once you have applied these steps you will either find that the Constraint has moved elsewhere (you’ve ‘broken’ the original Constraint), or it has remained in place. In either case, you should repeat the steps as part of a culture of continuous improvement. Eventually, you want to see the constraint move outside your system and become a matter of consumer demand.

Applying ToC to Our Question

If we look at the team’s three suggestions, we can see that each corresponds to one of these techniques:

1. Scaling up the database is elevation; there’s a clear financial cost in using larger servers.
2. Introducing caching is subordination; we’re changing the rest of the system to reduce pressure on the constraint, and need to consider questions such as cache invalidation before we make this change.
3. Optimizing the query is exploitation; we’re making local changes to the constraint to improve its performance.

Applying ToC tells us which of these approaches to consider first, namely optimizing the query. We can look at caching if an optimized query is still not sufficient, and scaling should be a last resort.

In our case, query optimization was sufficient. We managed to meet our performance target without introducing additional complexity to the system or incurring further cost.

Further Reading

Goldratt, Eliyahu M.; Jeff Cox. The Goal: A Process of Ongoing Improvement. Great Barrington, MA.: North River Press.