Microservices and Distributed Transactions

An Example Using XTA: XA Transaction API

Abstract

The two-phase commit protocol was designed in the epoch of the “big iron” systems like mainframes and UNIX servers; the XA specification was defined in 1991 when the typical deployment model consisted of having all the software installed in a single server. Surprisingly enough, a consistent part of the specification can be re-used to support distributed transactions inside a microservices-based architecture. This post explains how LIXA and XTA enable the development of polyglot distributed transactional systems.

Introduction

In brief, the two-phase commit protocol [1] is a specialized consensus protocol that requires two phases. The first phase, the voting phase, is where all the concerned resources are asked to “prepare”: a positive reply implies that a “durable” state has been reached. The second phase, the commit phase, confirms the new state of all the concerned resources. In the event of an error, the protocol rolls back and all the resources reach the previous state.

The XA Specification

One of the most common implementations of the two-phase commit protocol is the XA specification [2]: it has been supported by many middlewares developed during the Nineties and it’s leveraged by the JTA specification [3].

Historical Acceptance

The usage of the two-phase commit protocol has been debated a lot since its inception. On one side, the enthusiasts tried to use it in every circumstance; on the other side, the detractors avoided it in all the situations.

A first note that must be reported is related to performance: with every consensus protocol, the two-phase commit increases the time spent by a transaction. This side effect can’t be avoided and it must be considered at design time.

It’s even common knowledge that some resource managers are affected by scalability limits when they manage XA transactions: this behavior depends more on the quality of the implementation than on the two-phase commit protocol itself.

The abuse of two-phase commit severely hurts the performance of a distributed system, but trying to avoid it when it's the obvious solution leads to baroque and over engineered systems that are difficult to maintain. More specifically, the integration of already existing services requires serious re-engineering when both the transactional behavior must be guaranteed and a consensus protocol like the two-phase commit is not used.

Two Phase Commit Protocol and Microservices

Many authors discourage the usage of both the XA standard and the two phase commit protocol in the realm of microservices; the reader can find some posts in the References section [4].

Anyway, this post provides some development tools that support distributed transactions; the reader can try them in few minutes and evaluate the opportunity to reuse them.

Architecture

LIXA is a transaction manager that implements the two-phase commit and supports the XA specification. It’s a free and open source software licensed under the terms of the GNU Public License and the Lesser GNU Public License; it can be freely downloaded from GitHub and from SourceForge [5].

XTA stands for XA Transaction API and it’s an interface designed to support contemporary programming needs: it reuses the core components developed by the LIXA project and it introduces a new programming model (see below). XTA is currently available for C, C++, Java, and Python; it can be compiled from source code and installed in a Linux system or it can be executed as a Docker container.

The above diagram shows the main picture about XTA architecture: independently from the programming language, the “Application Program” interacts directly with XTA to manage transactions and with one or more “Resource Managers” to manage persistent data. Typical resource managers are MySQL/MariaDB and PostgreSQL.

The above diagram shows the high-level architecture implemented by the example explained in this post:

• “rest-client” is a client program developed using Python 3; it persists data in a MySQL database.

• “rest-server” is a service implemented in Java and it’s called by mean of a REST API; it persists data in a PostgreSQL database.

• “lixad” is the LIXA state server, it persists the transactions state on behalf of the XTA processes.

All the components are executed as Docker containers [6] for the sake of easiness, but a traditional installation can be done as well.

Executing the Example

Before starting, you need an operating system with two fundamental tools: git and Docker. Both are available for free and are easy to set up in Linux, MacOS, and Windows.

Set Up

First of all, clone the git repository that contains all you need:

git clone https://github.com/tiian/lixa-docker.git

Then go to the example directory:

cd lixa-docker/examples/PythonJavaREST

Build the Docker images for the “rest-client” and for the “rest-server”:

docker build -f Dockerfile-client -t rest-client .
docker build -f Dockerfile-server -t rest-server .

Check the built images, you should see something like:

docker images | grep rest
rest-server          latest              81eda2af0fd4        25 hours ago        731MB
rest-client          latest              322a3a26e040        25 hours ago        390MB

Start MySQL, PostgreSQL, and the LIXA state server (lixad):

docker run --rm -e MYSQL_ROOT_PASSWORD=mysecretpw -p 3306:3306 -d lixa/mysql
docker run --rm -e POSTGRES_PASSWORD=lixa -p 5432:5432 -d lixa/postgres -c 'max_prepared_transactions=10'
docker run --rm -p 2345:2345 -d lixa/lixad

Check the started containers, you should see something like:

docker ps | grep lixa
16099992bd82        lixa/lixad          "/home/lixa/lixad-en…"   6 seconds ago       Up 3 seconds        0.0.0.0:2345->2345/tcp              sharp_yalow
15297ed6ebb1        lixa/postgres       "docker-entrypoint.s…"   13 seconds ago      Up 9 seconds        0.0.0.0:5432->5432/tcp              unruffled_brahmagupta
3275a2738237        lixa/mysql          "docker-entrypoint.s…"   21 seconds ago      Up 18 seconds       0.0.0.0:3306->3306/tcp, 33060/tcp   sharp_wilson

Start the Programs

Activate the Java service (replace the IP address “192.168.123.35” with the IP address of your Docker host):

docker run -ti --rm -e MAVEN_OPTS="-Djava.library.path=/opt/lixa/lib" -e LIXA_STATE_SERVERS="tcp://192.168.123.35:2345/default" -e PQSERVER="192.168.123.35" -p 18080:8080 rest-server

Wait for the service readiness and the check the following messages in the console:

Mar 24, 2019 9:21:45 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8080]
Mar 24, 2019 9:21:45 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
Jersey app started with WADL available at http://0.0.0.0:8080/xta/application.wadl
Hit enter to stop it...

Start the Python client from another terminal (replace the IP address “192.168.123.35” with the IP address of your Docker host):

docker run -ti --rm -e SERVER="192.168.123.35" -e LIXA_STATE_SERVERS="tcp://192.168.123.35:2345/default" rest-client

At this point, you should have some messages in the console of the Java service:

***** REST service called: xid='1279875137.c5b6d8bf7d584065b4ee29d62fe35ab5.ac3d62eb862b49fe6a50bfee46d142b9', oper='delete' *****
2019-03-24 21:23:04.047857 [1/139693866854144] INFO: LXC000I this process is starting a new LIXA transaction manager (lixa package version is 1.7.6)
Created a subordinate branch with XID '1279875137.c5b6d8bf7d584065b4ee29d62fe35ab5.ac3d62eb862b49fe9de52bd41657494a'
PostgreSQL: executing SQL statement >DELETE FROM authors WHERE id=1804<
Executing first phase of commit (prepare)
Returning 'PREPARED' to the client
Executing second phase of commit
***** REST service called: xid='1279875137.91f1712af1164dd38dcc0b14d58819d2.ac3d62eb862b49fe6a50bfee46d142b9', oper='insert' *****
Created a subordinate branch with XID '1279875137.91f1712af1164dd38dcc0b14d58819d2.ac3d62eb862b49fe20cca6a7fc7c4802'
PostgreSQL: executing SQL statement >INSERT INTO authors VALUES(1804, 'Hawthorne', 'Nathaniel')<
Executing first phase of commit (prepare)
Returning 'PREPARED' to the client
Executing second phase of commit

And some other messages in the console of the Python client:

2019-03-24 21:23:03.909808 [1/140397122036736] INFO: LXC000I this process is starting a new LIXA transaction manager (lixa package version is 1.7.6)
***** REST client *****
MySQL: executing SQL statement >DELETE FROM authors WHERE id=1840<
Calling REST service passing: xid='1279875137.c5b6d8bf7d584065b4ee29d62fe35ab5.ac3d62eb862b49fe6a50bfee46d142b9', oper='delete'
Server replied >PREPARED<
Executing transaction commit
***** REST client *****
MySQL: executing SQL statement >INSERT INTO authors VALUES(1840, 'Zola', 'Emile')<
Calling REST service passing: xid='1279875137.91f1712af1164dd38dcc0b14d58819d2.ac3d62eb862b49fe6a50bfee46d142b9', oper='insert'
Server replied >PREPARED<
Executing transaction commit

Execution Explained

Python client:

• Row 1: the client starts its transaction manager.

• Row 3: the client executes the SQL statement (“DELETE”) in MySQL.

• Row 4: the client calls the Java service passing the transaction identifier (xid) and the required operation (“delete”).

Java service:

• Row 1: the service is called, it receives the transaction identifier (xid) and the required operation (“delete”).

• Row 2: the service starts its transaction manager.

• Row 3: the service branches the global transaction and it creates a new transaction identifiers.

• Row 4: the service executes the SQL statement (“DELETE”) in PostgreSQL.

• Row 5: the service executes the first phase of the commit protocol, it “prepares” PostgreSQL.

• Row 6: the service returns the result “PREPARED” to the client.

• Row 7: the service performs the second phase of the commit protocol.

Python client:

• Row 5: the client receives the result “PREPARED” from the service.

• Row 6: the client executes the commit protocol.

The remaining steps repeats the same actions for the second SQL statement (“INSERT”).

XTA Programming Model

The client/server example described above implements one of the patterns supported by XTA: “multiple applications, concurrent branches/pseudo synchronous.”

The above diagram describes the interactions between the client and the server.

The transactional part of the sequence diagram is delimited by a red dashed box.

The magenta dashed box contains the REST call and the first phase of the commit protocol: tx.commit()  is called passing true as the value for “non-blocking” parameter.

The blue dashed box contains the second phase of the commit protocol.

The diagram does not show the interactions among “rest-client,” “rest-server,” and the LIXA state server:

• The magic happens when the background thread of the Java server calls tx.commit(false).

• The LIXA state server recognizes the multiple branches transaction and it blocks “rest-server” until “rest-client” completes its first phase of the commit.

• At that point, marked by a dashed green line in the picture, a global consensus has been reached.

• Finally, both the players can go on with the second phase of the commit protocol.

In the event of a crash either on the client side or on server side, the parties rollback or automatically recover a second time: the explanation of these scenarios is left for a future post.

The below pieces of code are provided to show how XTA objects and methods must be used.

The Python Client Code

Ignoring boilerplate and scaffolding, here is the interesting part of the Python client source code:

# initialize XTA environment
Xta_init()

# create a new MySQL connection
# Note: using MySQLdb functions
rm = MySQLdb.connect(host=hostname, user="lixa", password="passw0rd", db="lixa")

# create a new XTA Transaction Manager object
tm = TransactionManager()

# create an XA resource for MySQL
# second parameter "MySQL" is descriptive
# third parameter "localhost,0,lixa,,lixa" identifies the specific database
xar = MysqlXaResource(rm._get_native_connection(), "MySQL", 
      hostname + "/lixa")

# Create a new XA global transaction and retrieve a reference from
# the TransactionManager object
tx = tm.createTransaction()

# Enlist MySQL resource to transaction
tx.enlistResource(xar)

sys.stdout.write("***** REST client *****\n")
# Start a new XA global transaction with multiple branches
tx.start(True)

# Execute DELETE statement
sys.stdout.write("MySQL: executing SQL statement >" + delete_stmt + "<\n")
cur = rm.cursor()
cur.execute(delete_stmt)

# Retrieving xid
xid = tx.getXid().toString()

# Calling server passing xid
sys.stdout.write("Calling REST service passing: xid='" + xid + "', oper='delete'\n")
r = requests.post("http://" + hostname + ":18080/xta/myresource",
     data={'xid':xid, 'oper':'delete'})
sys.stdout.write("Server replied >" + r.text + "<\n")

# Commit the transaction
sys.stdout.write("Executing transaction commit\n")
tx.commit()

• Row 14: the XA resource (xar) is linked to a MySQL connection (rm).

• Row 22: the XA resource is enlisted to a transaction object.

• Row 26: the global transaction is started.

• Row 38: the client calls the REST service.

• Row 44: the transaction is committed.

The full source code is available here: https://github.com/tiian/lixa-docker/blob/master/examples/PythonJavaREST/client.py

The Java Server Code

Ignoring boilerplate and scaffolding, the example uses the Jersey framework, here is the interesting part of the Java server source code:

// 1. create an XA Data Source
xads = new PGXADataSource();
// 2. set connection parameters (one property at a time)
xads.setServerName(System.getenv("PQSERVER"));
xads.setDatabaseName("lixa");
xads.setUser("lixa");
xads.setPassword("passw0rd");
// 3. get an XA Connection from the XA Data Source
xac = xads.getXAConnection();
// 4. get an XA Resource from the XA Connection
xar = xac.getXAResource();
// 5. get an SQL Connection from the XA Connection
conn = xac.getConnection();
//
// XTA code
//
// Create a mew XTA Transaction Manager
tm = new TransactionManager();
// Create a new XA global transaction using the Transaction
// Manager as a factory
tx = tm.createTransaction();
// Enlist PostgreSQL resource to transaction
tx.enlistResource(xar, "PostgreSQL",
System.getenv("PQSERVER") + ";u=lixa;db=lixa");
// create a new branch in the same global transaction
tx.branch(xid);
System.out.println("Created a subordinate branch " +
"with XID '" + tx.getXid().toString() + "'");
//
// Create and Execute a JDBC statement for PostgreSQL
//
System.out.println("PostgreSQL: executing SQL statement >" +
sqlStatement + "<");
// create a Statement object
stmt = conn.createStatement();
// Execute the statement
stmt.executeUpdate(sqlStatement);
// close the statement
stmt.close();
// perform first phase of commit (PREPARE ONLY)
System.out.println("Executing first phase of commit (prepare)");
tx.commit(true);
// start a backgroud thread: control must be returned to the client
// but finalization must go on in parallel
new Thread(new Runnable() {
@Override
public void run() {
try {
    // perform second phase of commit
    System.out.println("Executing second phase of commit");
    tx.commit(false);
    // Close Statement, SQL Connection and XA
    // Connection for PostgreSQL
    stmt.close();
    conn.close();
    xac.close();
} catch  (XtaException e) {
    System.err.println("XtaException: LIXA ReturnCode=" +
e.getReturnCode() + " ('" +
e.getMessage() + "')");
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}).start();

System.out.println("Returning 'PREPARED' to the client");
return "PREPARED";

• Row 11: the XA resource (xar) is retrieved by the PostgreSQL connection (xac).

• Row 23: the XA resource is enlisted to a transaction object.

• Row 26: a new branch of the global transaction is created.

• Row 42: the first phase of the commit protocol (“prepare”) is executed.

• Row 51: the second phase of the commit protocol is executed by a background thread.

• Row 69: the service returns the result to the caller.

The full source code is available here: https://github.com/tiian/lixa-docker/blob/master/examples/PythonJavaREST/src/main/java/org/tiian/lixa/xta/examples/MyResource.java

LIXA and XTA Uniqueness

LIXA has been designed and developed with a clear and well defined architecture: all the transactional information is persisted by the state server (lixad), most of the transactional logic is managed by the client library (lixac and its derivatives). The strong decoupling between the “logic” and the “state” enables the embedding of the transaction management capabilities in the “Application Program” without requiring neither frameworks nor application servers.

XTA pushed the concept of “distributed transaction” a step further: client and server don’t have to be executed by any sort of “supervisor middleware” because they coordinate themselves autonomously interacting with the LIXA state server. The only information that the client must pass to the server is the ASCII string representation of the XID. Other approaches implemented in the past required configuration and coordination among application servers; most of the times, even the communication protocols had to be aware about the transactionality aspects.

Furthermore, XTA supports multiple client/servers in the same global transaction: the same XID can be branched by many called services, even hierarchically.

XTA supports synchronous protocols, like the RESTful one shown in this example, as well as asynchronous protocols by mean of a different pattern ("multiple applications, concurrent branches/pseudo asynchronous").

Conclusions

The XTA API in conjunction with the LIXA state server enables the development of systems that implement ACID [7] distributed transactions among 2 or more applications (services). XTA enables the development of polyglot transactional systems that use multiple resource managers and any communication protocol without requiring some sort of application manager.

References

[1] XA Transactions (2 Phase Commit): A Simple Guide: https://dzone.com/articles/xa-transactions-2-phase-commit

[2] Distributed Transaction Processing: The XA Specification, X/Open: https://publications.opengroup.org/c193

[3] Java Transaction API (JTA): https://download.oracle.com/otn-pub/jcp/jta-1_2-mrel2-eval-spec/JTA1.2Specification.pdf

[4] Two Phase Commit and Microservices:

• Distributed Transactions and Saga Patterns https://dzone.com/articles/distributed-transactions-and-saga-patterns

• Transaction Management in Microservices https://medium.com/@walkingtreetech/transaction-management-in-microservices-ab09b0cb803b

• Transactions across REST microservices? https://stackoverflow.com/questions/30213456/transactions-across-rest-microservices

• Pattern: Saga https://microservices.io/patterns/data/saga.html

• Distributed Transactions: The Icebergs of Microservices https://www.grahamlea.com/2016/08/distributed-transactions-microservices-icebergs/

• A Guide to Transactions Across Microservices https://www.baeldung.com/transactions-across-microservices

• Patterns for distributed transactions within a microservices architecture https://developers.redhat.com/blog/2018/10/01/patterns-for-distributed-transactions-within-a-microservices-architecture/

• Distributed Transactions in Microservices https://blog.aspiresys.com/software-product-engineering/producteering/distributed-transactions-in-microservices/  

[5] LIXA project: http://www.tiian.org/lixa/

[6] LIXA Docker images are available in DockerHub: https://hub.docker.com/search?q=lixa&type=image

[7] ACID: Atomicity, Consistency, Isolation, Durability: https://en.wikipedia.org/wiki/ACID_(computer_science)