Simplify Microservice Transactions With Oracle Database Sagas

The Saga Pattern

The saga pattern is used to provide data integrity across multiple distributed, isolated services involved in potentially long-lived activities (sometimes referred to as business activities or business transactions to disambiguate).  The classic example, and the one I use here and in the corresponding workshop, is a travel booking consisting of flight, hotel, and car reservations:

The original paper was published in 1987 and alludes to the advantages of an implementation that runs and stores data within a database and that is precisely what is being done in the Oracle database, making it a perfect fit for microservices and modern applications development for a number of reasons.  

Much is written about sagas and so I will focus on the Oracle database saga offering in this write-up, however, I will go into a few of the fundamental aspects first.  If you are familiar with these concepts you may skip to the "Oracle Database Saga Support and OSaga API"

Traditional Transaction Processing Models and Microservices

Traditional transaction processing systems use a two-phase commit (2pc) protocol and the XA standard to handle this data integrity and provide ACID guarantees across resource managers (databases, messaging systems, etc.).  This protocol was designed for short-running transactions, typically with blocking synchronous communication, in a controlled environment and uses distributed locks to achieve these ACID properties thus preventing transactions from seeing the effects of other inflight transactions and making it possible for the application to simply make a rollback call to the transaction manager in order to effectively set the state of the resource managers' data back to where it was before the transaction was begun.  

System failures are resolved by a central transaction manager/recovery manager. As the protocol involves the locking and availability of resources, it poses a potential problem in a microservices environment with disparate services, potentially long-running activities, increased use of asynchronous/event-driven communication, varying degrees of fault tolerance and availability, and a potentially increased number of participants due in particular to the database-per-service pattern. 

Oracle Converged Database

Note that the key aspect of the database-per-service pattern is the isolation and decoupling of data dependencies between microservices.  This is to facilitate agile development and deployment models, independent scaling, use of the best database for the particular use case/model, etc.  These may or may not necessarily require a literal database per service. The Oracle database can provide isolation and security at numerous levels including, eg, the schema. 

It also provides all of the various data models and open APIs such as traditional relational, JSON, XML, Spatial, etc., and even direct compatibility with other vendor APIs such as Kafka and MongoDB. Finally, it provides the strict database-per-service capability while also providing a single synchronized architecture for administration, HA and backups, knowledge base, etc. These are some of the reasons Oracle is referred to as the converged database and why it is a perfect fit for sagas and microservices in general.

Considerations of Saga Pattern

The saga pattern uses local resource transactions only (no distributed locks) and in doing so alleviates the issues mentioned as far traditional 2pc protocols, however, it has its own considerations and trade-offs. When using sagas, the developer must account for the isolation and eventual consistency needs of the business activity/transaction and provide their own compensation logic for the rollback case as there is no mechanism for managing the locks, data changes, etc.  

This, along with the debugging involved with it, can be a very involved and error-prone task which is why Oracle has put tremendous focus on exactly these issues, making them simple with guaranteed transactional behaviors as  I will discuss in more detail in this and future blogs.

The nature of most optimizations, in general, is that they require closer attention to the specific details and requirements of an actual use case and perhaps more fine-tuned handling.  For example, on the one hand, developers may not consider the importance of ACID properties and what can go wrong without one or more of them while on the other hand developers may have assumed there was no other way to architect a transactional solution than to have strong ACID properties. 

The posturing of the misconception that nothing but strict ACID properties is acceptable in any scenario is quite often actually the result of the convenience of clicking an "enable XA" checkbox next to a data source and sticking an @Transactional annotation on a method rather than an actual requirement of the use case. 

Just as an example, taking the case of many financial systems' transactions, the reality is that they are not a single 2pc XA transaction between two bank account tables but rather 2 separate transactions consisting of a withdrawal with a message send for tx1 and a message received with a deposit for tx2.  Many transactions also have ordering requirements that are not actually handled by the XA specification, etc.

There are cases that are best suited for 2pc and XA and it is not absolutely prohibited in a microservices environment per se, but it can pose a major drawback to some of the most important benefits that the microservices model brings such as data/locking isolation, deployment models, and limits on scaling and throughput. This is why it is suggested for controlled environments/architectures or self-contained within a given/single microservice.  

Saga Specifics and Variations

There are a few variations to the saga pattern.  

Choreography-based sagas involve participants/microservices resolving consensus between themselves. A basic version of a choreography-based saga can be found in the Simplifying Microservices with a converged Oracle Database Workshop involving an order and inventory service.  Orchestration-based sagas involve a devoted orchestrator/coordinator, analogous to a transactions manager, that is responsible for managing the lifecycle of sagas including making the necessary callbacks, etc. 

The saga support in the database is orchestration-based. Saga implementations based on the Eclipse Microprofile Long Running Actions (LRA) specification are also orchestration-based and include Narayana, Helidon, and an upcoming Transaction Manager for Microservices product which will also provide XA and Try-Confirm/Cancel transaction support.  The orchestrator/coordinator of these implementations runs in Kubernetes, whereas the Oracle database saga orchestrator/coordinator runs within the database itself.

There are also variations of saga communication and transfer of control such as aggregator vs chain flow.  The travel agency example, where all replies from travel participants are gathered by the travel agency before completion or compensation occurs, is an example of the aggregator pattern, while the order inventory example where compensation of inventory occurs before returning to order service, is an example of chain flow. While related, these and other communications patterns (Event Command Transformation Pattern, Event Sourcing, Transactional Outbox, etc.),  go beyond the scope of this blog and are not essential to the core concepts.

Saga Flow

The diagram below depicts the flow of a travel booking saga, differentiating the types of calls by color:

1. The travel agency service contacts the coordinator to begin a saga. 
2. The travel agency then contacts the hotel service with an application request to book a room.
3. The request includes a saga id as part of the context and so the library that resides with the hotel service contacts the coordinator and requests to join the saga, providing the saga with a callback to be used at the time of saga completion.
4. The Control returns to the travel agency and the same process is repeated for any other participants, such as the flight service.
5. Once the travel agency has received all replies, it determines whether to commit/complete or roll back/compensate the saga and issues this command against the coordinator.
6. The coordinator then calls the appropriate callbacks for the saga participants and returns control to the travel agency.

Oracle Database Saga Support and OSaga API

The following table compares the support provided when using  LRA vs the Oracle saga support in the database vs the case where the developer must implement the entire protocol.

As you can see, implementing the protocol correctly and robustly is a complex and daunting task that can also result in cluttered application code.  Oracle database saga support will initially be provided for both PL/SQL and Java and will then be extended to all other languages. Below is a Java example of the difference when developing from scratch vs using the Oracle database saga support and the OSaga API. The left is a conservative example of the extra logic, database logging, journaling, etc. required while the right is a very accurate depiction of how simple an alternative it is.

Escrow and Auto-Compensating Data Types

Escrow types provide a unique locking mechanism for high-concurrency hot spots that are tailored perfectly to microservice transaction models and will be the subject of a future blog as will a related feature, auto-compensating data types, however, I will touch on the latter now as it is directly intended for sagas.

In short, auto-compensating data types will provide the same implicit rollback support for commutative data operations as found in traditional two-phase commit XA transactions (thus removing the need for the development of callbacks and recovery) but for sagas (ie using only local transactions). By using this in tandem with the saga support in the database, developers can get all of the benefits of the saga pattern with all of the benefits and simplicity of traditional transaction patterns!

Again Oracle has put a tremendous focus on making microservices and saga support powerful and simple.

Try It Yourself Today

While the features mentioned have not yet been released, you can try a preview in the new workshop: Simplify Microservice Transactions With Oracle Database Sagas Workshop

Thanks for reading and as always please feel free to reach out and let me know of any questions, suggestions, etc. feedback you have.