New JPA Performance Benchmark

this article presents a new open source database performance benchmark for jpa that covers hibernate, eclipselink, datanucleus, openjpa and objectdb.

my name is ilan kirsh and i am the founder of objectdb software, a provider of objectdb , a high performance commercial java object database that natively supports both jpa and jdo.

i have created a comprehensive new benchmark for measuring jpa provider performance. as part of this, i collected results for many different combinations of jpa implementations ( hibernate , eclipselink , datanucleus , openjpa ) and leading rdbms products ( mysql , postgresql , derby , h2 , hsqldb , sqlite ), as well as objectdb .

you can see the results at jpab.org

try out the site – you can compare products and rank them according to the results.

why a benchmark focused on jpa?

the java persistence api (jpa) is the new standard for working with databases in java. unlike older persistence solutions (jdbc,ejb cmp), jpa supports direct persistence and retrieval of pojo (plain old java objects), which increases development productivity.

there are many open source benchmark programs for testing database performance. however, most of the benchmark programs for java are based on jdbc and cannot be used for comparison of java persistence api (jpa) providers, such as hibernate, eclipselink and objectdb.

the few jpa benchmarks that have been published so far for jpa were very limited in scope. they were restricted to a single rdbms and to a few very specific database operations.

a good benchmark for jpa must support comparing effectively many combinations of jpa providers and databases, because a jpa provider might work well with one dbms but may become very slow with another dbms (as indeed shown in the benchmark results).

why do i care? i wanted a transparent, repeatable set of tests to compare objectdb with the other products out there. i also wanted something that people could directly critique and try for themselves.

benchmark results

in addition to objectdb, 4 orm implementations have been tested (hibernate, eclipselink, openjpa, datanucleus) with 6 different rdbms (mysql, postgresql, derby, h2, hsqldb, sqlite), some of them in both client-server and embedded modes.

the results can be used in different ways. first, you can browse the results and use filters to focus on a specific test.

for example, the following chart presents results of a specific test - persisting simple entity objects in batches of 5000 per transaction:

second, you can view result summary for:

• every tested jpa provider, e.g. hibernate .
• every tested database, e.g. mysql .
• every tested combination of jpa provider and database, e.g. eclipselink and h2 .

for example, the following chart presents the results of sqlite in the benchmark (with hibernate and eclipselink):

finally, you can produce a head to head comparison of any two tested combinations, e.g. comparison of hibernate with derby in server mode vs embedded mode:

the benchmark program

let me explain a little about the main criteria i used in creating the benchmark. first, it should work with any jpa 2 implementation. to avoid influence of specific code for specific implementations on the results - exactly the same code (standard jpa) must be used for all the tested implementations.

second, the benchmark should work with any database with jpa support and it should be able to generate separate results for every combination of tested jpa/dbms, e.g. hibernate/mysql, hibernate/postgresql, eclipselink/mysql, etc.

third, the tests should be easy to run and easy to configure and it also has to be easy to add new tests.

in addition, the benchmark has to be open source. on the jpab.org site, you can download the full sources for all the tests, try them yourself, view and critique the code.

more details are provided in the test description and faq pages.

objectdb performance

one of the clear outcomes of this testing is that objectdb outperforms the other jpa/dbms combinations often by an order of magnitude. these results initially startled me.

"haha", i hear you mutter. "he would say that". it’s a reasonable response -- after all, i created the tests.

the key point is that the tests are open, transparent and the source is available. please run the tests yourself and verify the results. there’s no magic there – i want to get as much feedback and make it as fair as possible.

when i started selling objectdb in 2003, i didn't attach too much importance to performance. the original purpose of developing objectdb was merely to provide an easy to use database that can store objects (and graphs of objects) with no need for object relational mapping.

however, over the years feedback from customers told me that performance is actually the main source of attraction of objectdb. many objectdb users have conducted their own benchmarks. many of them have sent me the results. it is weird but actually i learned about a performance gap between objectdb and other products from users. so, why is objectdb faster? i put it down to the fact that there are a lot less layers involved. with a jpa orm and an rdbms, the provider must convert the objects, process the metadata etc, and then write out the sql. with objectdb, the process is far more direct. what i didn’t realize is that this directness seems to improve performance by 10x.

so, frankly, one of my main motivations for developing a transparent jpa benchmark was to present the performance capabilities of objectdb. my aim is to be completely transparent, and in that spirit i am making all the results and test code available. the faq explains the limitations of the benchmark and in which situations the results are relevant and in which they are not. in addition, performance is not critical in every application, so in many cases using a slower jpa/dbms combination is not an issue.

notice that the benchmark should be useful also for anyone that is only interested in rdbms and orm based jpa implementations and not in an object database. in that case it is even very objective since i don't have a preference for any particular orm or rdbms product.

call for help in tuning

all the results that are currently presented on the benchmark website reflect using default configuration for all the tested products (except one exception that is explained in the faq). because the tests are relatively simple, the default configuration is expected to work well. it also seems fair to use the default configuration for all the products.

however, i would like to run a second phase, in which all the products are run with optimized tuning for this benchmark. we have had some preliminary success for openjpa - by adjusting parameters we were able to get results much closer to the high end of the orm range and much higher test pass ratio (initial results are presented on http://temp.jpab.org). this next phase will require expert help from the community. if you can help in tuning one or more of the tested jpa/dbms combinations please send an email to feedback at jpab dot org.

about objectdb

objectdb is a high performance commercial object database for java, offering native support for jpa and jdo. one of its major selling points is its phenomenal performance. all license prices (per server, per site and oem) are available online . we also offer free licenses to open source projects, individual developers and small startup companies. you can also download . a free edition of objectdb.