The Fallback Trap
Fallbacks improve resilience, but can hide failures. Good observability makes their usage visible, measurable, and easier to trust.
When we think about resilience, one of the first things that comes to mind for someone who has worked for many years with Java is Resilience4j, more specifically, its main modules: circuit breakers, bulkheads, rate limiting, retries, timeouts, and caching. Less prominent is its support for fallbacks. The Vavr library also supports fallbacks, just to mention another library. But we also implement fallbacks with regular try/catches, and the libraries offer alternative ways to do so.
In a nutshell, fallbacks are an important ingredient for creating a resilient system, because they allow operations to still be performed even when there are problems in parts of those operations. In other words, they allow the system to degrade gracefully.
So the question now is: what does it mean to degrade gracefully? For sure, we want the main functionality to still be carried out, so we don't usually expect that all operations will have fallbacks. In terms of what they are, we have a broad range: from doing nothing and skipping that part of the functionality, to using a default value, making an approximate calculation, using simplified logic, or even performing a totally different flow.
The main objective of using fallbacks is to improve resilience, which implies keeping reasonable service levels, and to make sure this is the case, we need to answer the following questions:
- Will the fallbacks really work?
- Are they happening too frequently?
Here we will focus mainly on the second question. On the first question, and on whether they are even necessary, our recommendation is this article from Amazon AWS: Avoiding fallback in distributed systems.
So you reviewed your system and identified and implemented fallbacks where it makes sense to apply them in case of failure. This is a great starting point, but unfortunately it is not enough, because we still need to make sure fallbacks don't happen more often than expected; otherwise, we can’t say the system is degrading gracefully.
Obviously, if an operation has a fallback, it may not block the application’s main flow, but it may block secondary flows that could, for instance, make customers think you removed a feature they are very interested in. In this case, because no error message would be given, they would have no clue there is a problem in the system, so they may not report the issue. That is why we need to keep them under control.
This is the fallback trap. Fallbacks are meant to give the impression that there are no problems, but they also end up hiding them. And no, logs are not enough, as we will explain next. The good news is that the problem is not the fallback; the problem is the lack of proper observability.
Observing Fallbacks
The most basic strategy for dealing with fallbacks is to log them. One level above would be to also generate metrics. With metrics, we can more easily create charts and track them for longer time ranges.
Another important aspect of generating metrics and logs is that each fallback may have a unique occurrence pattern: some may be more frequent, and some may be rarer. The more straightforward approach of generating logs and metrics for each occurrence may not be appropriate in all cases.
If a fallback keeps happening during the day, but at a low rate, it may be acceptable in terms of functionality, but it will generate some noise in the logs. One strategy is to use a sliding window with calculations performed locally, so logs and metrics are generated only when the relative occurrence of the fallback in the current window exceeds the specified threshold.
If you prefer to handle the relative percentage of errors using metrics, you may also end up generating metrics when there are no errors, which may increase your costs.
Another pattern that may also happen quite regularly is when a given functionality fails on different business flows, in which case it is also a good idea to identify the context when generating the logs and metrics, because this could help isolate the root cause, for example, in case a given fallback only happens in one flow, but not in another.
One important case where logs and metrics won’t help much is automated tests, unless you have a long-running test environment with monitors configured; even then, unit tests wouldn’t be covered, given their ephemeral nature.
The Trap Is Even More Serious in Tests
The system you are working on is great; it has thousands of tests that assert that all those cases still work, so chances are low that a change will introduce a regression. Until the day you start executing a test that was supposed to cover the part you are working on, only to realize the code you just changed doesn’t even get called. When this happens, you will have to first find out why and then fix it before going back to what you were doing.
To be fair, this can always happen, even outside of the context of fallback handling, for example, in the case when the test is not in fact reproducing a real possible scenario. But with fallbacks, this problem reaches a whole new level, because parts of the system may execute the fallback flows without anyone noticing, so the tests keep passing, and no one wants Highlander tests in their system (Highlander, the 1986 movie where the protagonist is immortal).
The main issue here is that, when fallbacks are not under control, it is difficult to verify that each scenario is really doing what it is supposed to do. It is true that when you create a new test scenario, you could constantly check the logs to see whether everything is working as expected. The problem is that this approach covers only the test you are working on right now, but you still need to make sure your changes don't trigger new fallbacks in existing tests. On top of that, if the tests already cover fallback scenarios, you will need to be careful to distinguish expected from unexpected fallbacks.
So what can we do now? We propose a structured approach to fallbacks that covers the aspects mentioned earlier: logs, metrics, and tests.
The Fallback Habitat
Fallbacks can appear in the code in a number of different forms. The most basic one is a try/catch block where, inside the catch, we can handle the fallback. With Feign and Resilience4j, the fallback definition appears when creating the decorators applied to that client instance (https://resilience4j.readme.io/docs/feign), which can live in the Spring bean declaration, far from where it's used, so the developer may be unaware of the fallback.
In the case of Spring + Resilience4j, they can also appear in the attribute fallbackMethod of the annotation @CircuitBreaker and other similar annotations. They can, in fact, appear in multiple places, including any sort of in-house execution framework you happen to be using.
Another important aspect of how fallbacks appear in the code is the fact that it is also common that their handling will make the code less readable. For example:
try {
productClient.load(productId)
} catch (exception: IOException) {
logger.warn("Executing fallback for case={}", "product.load", exception)
Counter.builder("fallback.executions")
.description("Number of fallback executions")
.tag("fallback.case", "product.load")
.tag("exception", exception.javaClass.name)
.register(meterRegistry)
.increment()
productCache.load(productId)
}
Imagine this spread all across the code! Ideally, this try/catch block should be moved to a separate method to make sure the main part is easier to understand, but it still wouldn’t help streamline logs and metrics generation.
Structured Fallback Handling
Structured fallback handling provides a leap in quality, centralizing and standardizing it, thus allowing features that would be difficult to imagine without it. For example, it supports tracking fallbacks that happen during test execution and using them to verify that only the expected fallbacks actually happened.
The basic idea here is that we factor out all that boilerplate code for logs and metrics into a central place. In order to allow for greater flexibility, we propose the use of a registry where observers can be registered and will later receive notifications when fallback-related events happen. Then we can create the observers for the aspects we want to handle: logs, metrics, tests, etc.
In https://github.com/staminacode/fallback-observation-kotlin, we created an early-stage library, which we are using to explore the ideas presented here.
Here is what the registry creation looks like:
val observerRegistry = FallbackObserverRegistry(
listOf(
FallbackObserver.LOGGING,
MicrometerFallbackObserver(meterRegistry),
),
)
val fallbackFactory = FallbackFactory(observerRegistry)
In the case of logs and metrics, the observers are pretty straightforward.
Let’s look at what the previous example, given above for a try/catch, would look like. First, we create the fallback policy using the fallbackFactory:
val productLoadPolicy = fallbackFactory.fallbackCase("product.load") {
handle<IOException>()
}
In this case, we are telling the factory that we want to perform the fallback in case we have an IOException (like in the try/catch block above). We also give a name to the “case,” which is used when the logs and metrics are generated. The policy can and should be reused, so it would naturally appear as a class property.
Then we have the call site itself:
val product = productLoadPolicy.withFallback(
operation = { productClient.load(productId) },
fallback = { productCache.load(productId) },
)
This clearly reads much better than the try/catch block.
Keeping Tests With Fallbacks Under Control
One straightforward way to avoid unintended fallbacks in tests is to create an observer that throws an exception when it receives a fallback event. Unfortunately, this may not always work because that exception may end up being handled by the application code and not making the test fail. The objective here is to make the mechanism resilient, given that we want it to catch tests or application code that may have bugs. Yes, the tests themselves may have bugs.
To make sure we identify if a fallback happened during the execution of a test, even if the application is misbehaving, what we can do is create an observer that accumulates information about the fallbacks that were executed, so after the test ends we can check what happened.
This improves the solution, but it may cause some unintended side effects, for example, in case you are using it with Spring (or Spring Boot) and want to run parallel tests. The issue happens because Spring caches the application context, so tests sharing the same context will also share the same beans, including the registry. After the test finishes, there is no way to differentiate which fallbacks happened in which test cases. For this reason, the recommendation is to avoid parallel tests in case the registry is shared.
For those familiar with Go, a natural solution for passing information about which test is being executed would be to use the context.Context type.
Finally, to facilitate the adoption of this approach, a good idea is to create a JUnit extension (or an equivalent integration for your test framework), with the option of integrating it with Spring. The extension would automate adding the test observer and assert that there aren't any unexpected fallbacks. The definition of the expected fallbacks and their number can be in an annotation, for example:
@Test
@ExpectedFallback("product.load", times = 2)
fun executesExpectedFallbackTwice() {
operation()
operation()
}
Testing Fallbacks in Production
Having a central place to control fallbacks opens a few different options, as we saw in the previous part. What else can we do?
As discussed in the AWS article mentioned earlier, one of the risks associated with fallbacks is the fact that, in some cases, they can go unused for a long time, and exactly at the moment we need them, they may not work.
One option is to extend the fallback handler to always check a configurable condition that determines whether the regular flow should run or an individual fallback should be used directly, bypassing the main flow altogether. This way, from time to time, we could verify that the fallbacks are giving the expected results in a very controlled way.
Or we could do something more extreme in terms of fault injection, for example, introduce random errors or delays at the same point in the fallback handler we just mentioned.
Conclusion
As we saw in this article, fallbacks are the first step to improving resilience, and if you don’t take further steps to keep them observable and under control, the consequences will show up sooner or later.
We also saw some of the possibilities a structured approach enables in different areas: code readability, avoiding code duplication, tools to support tests that go beyond line coverage, and even periodically checking that fallbacks work in production by not leaving them dormant for too long.
Let me know if you find more use cases for a structured approach to fallback handling.
Opinions expressed by DZone contributors are their own.