This article is based on the 4th round of ESB Performance testing conducted by the author since June 2007. This performance framework has now become the de-facto ESB Performance test suite, with vendors such as WSO2, BEA and Mule using it to publish benchmark results. This round moves the test execution to the Amazon EC2 environment, so that end users can themselves run the tests and verify the results - as well as use the framework to compare any ESB of their choice against the options compared in this report.
In this round of performance testing, we have compared the UltraESB 1.0-beta-2 (b3) to one of the best open source ESB's available. We tested the performance of a Direct Proxy service that simply virtualizes a service, a Content Based Routing (CBR) Proxy service using an XPath expression over a SOAP payload, an XSLT transformation Proxy service that translated a message and its response, and a WS-Security Proxy service that verified and performed timestamp, signature and encryption.
The author has conducted performance tests comparing many ESB's in the past. Round 1 conducted in June 2007 compared the WSO2 ESB to the leading proprietary ESB at the time, and Round 2 conducted in July 2007 compared the WSO2 ESB to the open source alternatives Mule and ServiceMix. Round 3 conducted in June 2008 compared the WSO2 ESB to Mule CE, ServiceMix, the leading proprietary ESB at the time, and the proprietary version of an open source ESB.
It was nice to see the other vendors such as BEA and Mule conducting the same performance tests - as the performance test framework, and tools were shared freely with the public. However, certain vendors did not appreciate being included in the comparison, and thus we will not name and publish results of any other ESB's we compare against - but leave it upto real users to request the configuration required for these test scenarios from each vendor, so that they could run the tests themselves.
Improvements to the Performance test framework
With the advent of the Amazon EC2, it has been a pleasure to move the performance testing to the cloud. This allows all vendors to run their implementations on the exact same environment, and in future to fully automate the process and move the client, ESB and backend service to multiple nodes if desired! For simplicity, and to enable even end-users to test each ESB on their own, currently this test was run with the client, ESB and backend service simulation all on the same node. An EBS volume was created to hold the configurations and all necessary software to run the test, and the author is prepared to share this directly with any interested end-users so that the results could be independently verified.
This round of testing was performed on a single "m1.large" instance of the AMI "ami-eef61587" - which is a 64 bit Ubuntu 9.04 server. The instance had 7.5 G of RAM with 4 EC2 Compute Units of CPU. Both ESBs were compared on the exact same instance one after the other, using JDK 1.6.0_18 and a heap memory allocation of 2GB for each. A NIO based Echo Service was developed for this run, as the earlier Tomcat/Servlet based implementation and a Jetty/Servlet based implementation both failed to keep up with the load of 2560 concurrent users through the ESB layer using a reasonable amount of threads. This NIO based Echo service and the load test client, and the sample requests used all ships with the UltraESB distribution which can be downloaded from http://adroitlogic.org
We invite the developers and/or vendors of both proprietary and open source ESBs to share configurations for the scenarios tested by this framework, so that end-users could test and compare the performance of these on their own. This will also allow the vendors/developers to state the most optimal configurations - instead of someone new to those ESB developing a possibly sub optimal configuration and reporting results publicly.
This round includes all three scenarios from Round 3, and introduces a fourth scenario for WS-Security testing.
1. Virtualization, where the ESB hides the actual service and performs message routing
In this scenario we create a proxy service for the actual Echo service, which simply passes on the messages to the real service. This pattern allows users to hide real services behind proxies exposed over the ESB and thereby prevent direct application-to-application, application-to-service and/or service-to-service links from being created within an enterprise to bring order to a Service Oriented Architecture (SOA) deployment. Refer to the Round 3 for more information.
2. Content-based Routing (CBR), where the ESB routes on data within the message
The Content Based Routing (CBR) proxy service performs an evaluation of an XPath expression over the payload of the messages, before they are routed to the real service. For this example, we use an XML payload with a list of 'order' elements, and check if the first order element is for the symbol "IBM". If this condition is satisfied, we forward it to the actual service implementation, else return an error. Typically payloads are routed on transport/SOAP headers and/or user defined header elements, or payload body elements.
3. Transformation, where the ESB transforms the request and response messages using XSLT-based transformations
In this scenario, we use an XSLT to reverse the element names before forwarding to the Echo service. The reversed response received is translated again to the original format by another reversal. This is a typical use case when newer versions of a service are exposed, and a subset of its users now require backwards compatibility with the previous schemas etc. Refer to the Round 3 for more information.
4. WS-Security, where the ESB acts as a Security Gateway, receiving and validating secure messages and forwarding them to the actual service, and secures the response before sending it to the original client
In this newly introduced scenario, we receive a Timestamped, Encrypted and Signed SOAP request, and verify the message, and forward the decrypted message to the echo service. The returned response is Timestamped, Signed and Encrypted again, and returned to the client. This demonstrates the ESB acting as a WS-Security gateway. In real world scenarios, Username token authentication, and HTTP basic/digest authentication and SSL are used as well to secure messages.
The tabulated results are given for various payload sizes ranging from 512 bytes, 1K, 5K, 10K and 100K for concurrent users from 20, 40, 80, 160, 320, 640, 1280 and 2560 as applicable for the various scenarios. "UE" in the results denote the UltraESB TPS, while "O" denotes the TPS obtained by the open source ESB. Results highlighted with a Yellow background resulted in timeout errors at the client, where the timeout was set to 2 minutes.
Thus "UE-500 b" denotes the UltraESB TPS for a 512 byte payload, while "UE-10 K" denotes the TPS for a 10K payload. Similarly "O-100 K" denotes the TPS for the other ESB for a request of 100K etc.
1. Direct Proxy scenario
|Users||UE-500 b||O-500 b||UE-1 K||O-1 K||UE-5 K||O-5 K||UE-10 K||O-10 K||UE-100 K||O-100 K|
4. WS-Security Proxy
|Users||UE-500 b||O-500 b||UE-1 K||O-1 K||UE-5 K||O-5 K|
1. Direct Proxy - For tiny messages, and low loads, the Other ESB performed slightly better than the UltraESB. However, the UltraESB performed better overall on average, for each payload size; and with larger messages and more concurrent users, it emerged with a clear lead of about 5 times the TPS of the other ESB.
Note: In the EC2 and and other such virtualized environments, the Zero-Copy proxying ability of the UltraESB may not fbe ully utilized. Thus on real server grade hardware, the UltraESB is expected to perform better. (e.g. when deployed as an appliance)
2. Content Based Routing (CBR) Proxy - The Other ESB emerged a clear winner in this case, as the UltraESB performed XPath evaluation on a DOM parsed result of the payload, while the other ESB used Stax parsing. From a usability point of view however, the UltraESB can support XPath v2, while the Stax based implementation based on Jaxen is limited to XPath v1. The lead against the UltraESB was about 2.5 times for tiny payloads, and reduced drastically for larger payloads.
Note: The UltraESB team intends to soon support Stax based XPath evaluation as well, and these results are expected in the next round of performance testing
3. XSLT Proxy - For tiny messages, and low loads, the Other ESB performed slightly better than the UltraESB. However, the UltraESB performed much better than the other ESB with larger payloads - upto 3.5 times more TPS on average, and about 6 times more than the other ESB for 100K payloads at 640 concurrent users.
4. WS-Security Proxy - The UltraESB emerged a clear winner in this case, with its custom WS-Security library implementation. The Other ESB relied on WSS4J/Rampart which is already well known for performance drawbacks. On average the UltraESB performed over 3 times the TPS of the Other ESB in the scenarios where both could be tested within reasonable timebounds. The results obtained shows even better performance by the UltraESB with larger load or payload sizes.
Asankha Perera, AdroitLogic
You may contact the author directly to obtain more information on the performance test framework or procedure, and to request a comparison against another ESB of choice. If you are an ESB developer or vendor, you may submit suitable configurations for your ESB, and request that it be included into future rounds of testing. If you are an end user of an ESB, you may request for the above configuration for comparison from your vendor/developers and obtain free support from the author to conduct your own comparison test.
asankha AT adroitlogic DOT com
1. Results in XLS format for offline analysis
2. UltraESB configuration for the load tests of Round 4
3. The sample requests used for the test can be found in the samples/resources/requests directory of the UltraESB installation.
4. The NIO based Echo service could be started by passing the "-echo" option to the UltraESB ToolBox as shown below.
5. The Apache Bench style load test client is embedded to the UltraESB ToolBox, and can be executed with the provided scripts as shown below, or from the command line - from the lib/samples directory as "java -jar ToolBox.jar".
6. An article that explains the scenarios in detail, and how another ESB of your choice could be compared against the UltraESB can be found here
Steps to run the load test on Amazon EC2
1. Start an instance of the AMI "ami-eef61587" on EC2
2. SSH to the instance and execute the following as root
mount /dev/sda /home/asankha
echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range
echo "30" > /proc/sys/net/ipv4/tcp_fin_timeout
echo "2097152" > /proc/sys/fs/file-max
echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
Note: this assumes you have access to the contents of the EBS used for performance testing, and attached it as /dev/sda to the instance. If not, do the following:
- Download and install the JDK 1.6.0_18 64-bit version to /home/asankha/jdk1.6.0_18, and install the unlimited strength JCE policy files
- Download the UltraESB 1.0-beta-2 build 3 from http://downloads.adroitlogic.com/adroitlogic-ultraesb-1.0-beta-2.zip and extract it to the directory /home/asankha/PerformanceTesting
3. Open 3 more SSH sessions, and execute the following on each session
ulimit -n 8192
su - asankha
4. Start the Echo service as follows on one of the new SSH sessions
nohup ./toolbox.sh -echo &
5. Download and extract the contents of the file http://downloads.adroitlogic.com/performance/round4/ue.tar to /home/asankha/PerformanceTesting/ultraesb-1.0-beta-2/conf
6. Edit the /home/asankha/PerformanceTesting/ultraesb-1.0-beta-2/bin/wrapper.conf and set the Heap size to 2G as follows:
Also, disable the auto JVM restart of the Java service wrapper by setting the following:
7. Start the UltraESB as a daemon process by executing the following on another SSH session
~/PerformanceTesting/ultraesb-1.0-beta-2/bin$ ./ultraesb-daemon.sh start
8. Download the scripts from http://downloads.adroitlogic.com/performance/round4/scripts.tar and extract to the directory /home/asankha/PerformanceTesting/LoadGenerator. Execute the load test script from this directory as follows from a new SSH session started in Step 4
./r3.sh http://localhost:8280/service > ue-r3.txt
9. Download the log file when the test completes to the desktop, and convert the resulting log file into a CSV file by executing the ab-to-csv utility, and analyze as desired
java -jar ab-to-csv.jar ue-r3.txt > ue-r3.csv