The Circuit Breaker Module

We tend to think of our services in terms of their “happy path,” while also underestimating the not-so-happy paths. What if my API is trying to reach another API/system and constantly fails? Can I reach it less often in case of a timeout? I thought about that in the case of a Mule 4 application and I came up with Circuit Breaker pattern to address this issue. In this article, I will briefly introduce the pattern and show you how you can use it in your Mule application.

Circuit Breaker Pattern

Circuit breakers monitor API calls. When everything is working as expected, they are in a closed state. When the number of fails, like timeouts, reaches a specified threshold, circuit breakers will stop processing further requests. We call this the open state. As a result, API clients will receive instant information that something went wrong without waiting for the timeout.

The circuit is opened for a specified period of time. After timeout occurs, the circuit breaker goes into a half-opened state. Next, the API call will hit the external system/API. After that, the circuit will decide whether to close or open itself.

You can find all states and flows in the diagram below (from Martin Fowler’s site).

Custom Mule 4 Module

Mule does not have such functionality. As a result, we need to either create it within the flow or as a custom module. I have decided to write a separate module that I can reuse and share with you. In order to do this, I will use the new Mule SDK.

I have decided to use this functionality to introduce the Circuit Breaker pattern into my flows.

Design

My module is completely independent of other Mule components; therefore, I have prepared two possible operations:

• Filter — to stop processing in the case of opened circuits.
• Record Failure — to increase the counter of failures.

Apart from that, we have a global configuration that configures the circuit breaker behavior.

Global Configuration

In order to control when circuit breaker state changes we need to have defined:

• Threshold — how many failures mean that the circuit is opened.
• Timeout [ms] — how much time needs to pass before the circuit will be in a closed state.

We can introduce one circuit breaker for all HTTP requests or many, depending on the scenario. For example, when our process API calls two system APIs we may have two different circuits for them.

To differentiate the breaker we will use the breaker name property.

You may ask where the state will be kept. The module uses the default persistent Object Store instance. In the future, I plan to separate that logic and give the user the ability to assign separately defined Object Stores.

Filter

This operation blocks the execution of any further component placed next to it. In the case of an opened circuit breaker, it won’t allow for the execution of external calls by throwing the OpenedCircuitException error.

Filter passes the event to the next event processor only if the circuit breaker is closed. It evaluates whether the opened circuit should be closed and pass the traffic further.

Record Failure

So far we have only changed the status from open to closed. How can we detect if it should be made in the opposite direction?

We need to catch errors thrown by the external call, like an HTTP Requestor. Our error strategy should use the Record failure operation in order to increase the counter of failures.

When the threshold is met — i.e. when the counter of failures is equal to or greater than the threshold number from global configuration — the circuit state will be moved to an open state.

Module Installation

You may use the Circuit Breaker module prepared by me. It is available to download from GitHub.

After you download the project you need to install it in your local Maven repository using the mvn clean install command. Next, make a reference to the module by adding the following dependency:

<groupId>pl.profit-online.extensions</groupId>

<artifactId>circuit-breaker</artifactId>

<version>1.0.0</version>

<classifier>mule-plugin</classifier>

After a couple of seconds, you should see a new module available on the Mule Palette — like in the picture below.

Circuit Breaker Module Usage

I have prepared two simple applications depicting Circuit Breaker module usage. You can find them on GitHub here.

Although I have hardcoded the failure of POST /consents in the Consent-MongoDB-SAPI, you may imagine the successful path as shown below. The caller should receive a response in a reasonable amount of time – let’s say 200 milliseconds.

The application is designed to fail. In that case, our process API will call POST/consents, but will receive an error. Then that error will be propagated to the caller. As you may imagine, in the case of timeouts it may be 30 seconds or even more.

This could annoy our caller, especially if they want to try to make a call a couple of times. To correct this, we must embrace the Circuit Breaker Module. After the fifth error, the Circuit will be moved to the opened state and any further calls, made by the caller, will quickly be rejected without calling the failing system API.

We give the system API time to rest before handling new requests.

Below you can find sample filters and record failure uses for one of the operations. It would be an erroneous strategy to react to all errors and record failures. However, we may pick specific errors to which our app should react, like timeouts, etc.

Summary

When I was working with microservices using the Spring Boot framework I really liked the idea of Netflix's Hystrix Circuit Breaker library. It made it easy to use that pattern in my code.

My work experience tells me that in Mule, situations occur where you would like to use the same pattern in your code. Although it is not available out of the box you may implement such behavior.

Why use the Circuit Breaker pattern? In order to shorten the fail response time and give an endpoint service time to rest and get back into an operational state.

You may use the implementation that I prepared, make it better, or give me some comments so I can make it better. Let’s build it for the community!