Remote Access to Your Services: EJB vs Spring/Hessian

The other day, I got to come up with a design solution for a small website. Small enough to fit on a single physical server. Nothing fancy, I had a pretty good idea on the frameworks to use and it would be a classic front-end plus back-end server deployment. With a twist, the back-end server would most likely be JBoss. And this is where things got interesting.

I was planning on using Spring on the back-end. But then I thought, how much sense does it make to fire-up a full-blown JBoss, only to use Spring on top of it? Maybe EJB would be a better pick in this instance? Specifically, I was thinking about the communication between the two servers: has RMI-IIOP evolved over time enough to best Hessian? A good software engineer would know the specs for both and have the answer already. So I had to come up with something else.

I quickly wrote two services, one for Spring, one for EJB. And a client for both. It would do a number of calls in a loop and time them - for both services. The services would return a char[] of various sizes, so that I could see the effect of various payloads (0, 1024, 2048, up to 16384). True, it's not a perfect real-world scenario if I don't hit the DB and perform some business logic in the process, but I'm trying to benchmark the protocol implementations here.

Still nothing fancy. So what's this article about, then? Well... the results!

This is what I got for the first runs:

                      Run 1 - Time [s]        Run 2 - Time [s]
Calls   Payload         EJB      Spring         EJB      Spring
===============================================================
  1,000      0        3.652        .836       3.628        .754
             1k       2.473        .694       2.679        .633
             2k       2.143        .653       2.291        .634
             4k      40.547        .611      40.242        .766
             8k      40.311       1.102      40.335       1.251
            16k      25.658       1.725      27.965       1.672
---------------------------------------------------------------
 10,000      0       17.218       3.218      16.751       3.413
             1k      11.919       2.236      12.017       2.279
             2k      11.205       2.454      11.280       2.717
             4k     401.615       4.618     401.225       4.655
             8k     401.252       7.999     401.645       8.248
            16k      91.225      12.545      78.676      12.762
---------------------------------------------------------------
100,000      0      112.856      13.395     111.962      14.113
             1k     111.384      19.956     111.236      22.019
             2k     109.806      26.476     107.961      27.201
             4k    4019.744      51.204    4019.709      38.711
             8k    4025.306      70.093    4013.178      68.229
            16k    1094.349     123.929    1044.610     124.129

Run 1 and 2: Kubuntu 9.10 beta, kernel 2.6.31 x86_64, SunJDK 1.6.0

So there's my answer: Spring/Hessian is faster than remote EJB. Even CPU usage favours the Spring/Hessian combo. For Spring/Hessian, typical client:server load ratio is about 3:1, while for EJB it was between 1:1 and 1:2 most of the time. More specific, Spring puts about 30% CPU load on the client and 10% on the server; at the same time EJB loads about 25% - 25% (client load can dip below, server load can peak over 30%).

But while Spring exhibits lower response time and a nice, almost linear scaling, what about those whacky EJB numbers? I specifically ran this tests twice to confirm them.
For payloads up to 2k, the graphs are almost flat for both Spring and EJB, but there some nasty spikes for EJB at 4k and 8k payloads. Also strange, during 4k and 8k EJB calls, the CPU load was virtually 0 for both client and server.

Looking for bugs in JDK, I switched to OpenJDK next.

                      Run 3 - Time [s]
Calls   Payload         EJB      Spring
=======================================
  1,000      0        3.599        .870
             1k       2.522        .664
             2k       2.469        .590
             4k      40.191        .680
             8k      40.220       1.108
            16k      26.832       1.671
---------------------------------------
 10,000      0       16.076       3.029
             1k      11.712       2.229
             2k      10.673       2.876
             4k     401.004       4.669
             8k     401.069       7.815
            16k      93.677      12.297
---------------------------------------
100,000      0      111.863      13.325
             1k     107.690      19.045    
             2k     106.736      25.726
             4k    4016.907      40.947
             8k    4023.713      72.376
            16k     902.372     125.865

Run 3: Kubuntu 9.10 beta, kernel 2.6.31 x86_64, OpenJDK 1.6.0

OpenJDK results are virtually identical. Must be either JBoss or the OS causing this.

Next, some Windows numbers, let's see if it's the OS.

                      Run 4 - Time [s]        Run 5 - Time [s]
Calls   Payload         EJB      Spring         EJB      Spring
===============================================================
  1,000      0        6.246       1.350       3.531        .781
             1k       5.623       1.174       2.703        .704
             2k       4.594       1.433       2.516        .718
             4k       6.354       1.710       2.375        .891
             8k       5.852       3.237       2.562       1.281
            16k       6.729       3.996       2.563       1.953
---------------------------------------------------------------
  10,000     0       41.604       6.824      23.797       4.266
             1k      23.960       6.534      22.250       3.906
             2k      32.041       9.198      22.406       4.781
             4k      37.564       9.312      22.391       6.657
             8k      23.752      10.246      22.421      10.547
            16k      27.492      24.511         N/A      18.109
---------------------------------------------------------------
100,000      0      371.700      42.819         N/A      30.266
             1k     267.341      59.343         N/A      39.640
             2k     353.062      83.713         N/A      48.610
             4k     335.661      90.985         N/A      67.265
             8k     294.875     141.625         N/A     105.719
            16k     306.000     270.254         N/A     182.906

Run 4: Win7RC/Virtualbox x86_64, SunJDK 1.6.0
Run 5: WinXP x86, SunJDK 1.6.0

Ok, no more spikes in EJB response time, though the times still look suspiciously flat. Plus, on WinXP, past 8k/1,000 calls, I just get an exception (Exception in thread "main" org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://127.0.0.1:3873/]). Still, I am using the same JBoss revision (5.1.0GA-jdk6, btw), so it seems JBoss is not causing those spikes. This leaves me with the OS.

Back to Linux.

                      Run 6 - Time [s]        Run 7 - Time [s]
Calls   Payload         EJB      Spring         EJB      Spring
===============================================================
  1,000      0        2.996        .779       3.314       1.012
             1k       2.553        .617       2.462        .658
             2k       2.568        .843       2.301        .900
             4k      42.797       1.015      43.080       1.002
             8k      42.122       1.639      43.046       1.369
            16k       5.507       2.489      22.103       2.401
---------------------------------------------------------------
  10,000     0       21.182       3.502      22.448       4.074
             1k      19.669       4.604      22.458       6.292
             2k      19.714       5.975      22.794       8.794
             4k     423.331       8.382    2590.024      10.251
             8k     421.513      14.162    2597.836      14.667
            16k      29.179      26.055     132.132      21.854
---------------------------------------------------------------
100,000      0      188.130      26.938     221.967      53.123
             1k     189.032      42.154     224.095      67.212
             2k     189.972      55.225     226.334      86.580
             4k    4184.557      78.404   21679.748     103.160
             8k    4169.555     136.656         N/A         N/A
            16k     306.762     236.289         N/A         N/A

Run 5: Kubuntu 9.10 beta/Virtualbox, kernel 2.6.31 x86, SunJDK 1.6.0
Run 6: CentOS 5.3/Virtualbox, kernel 2.6.18 x86, SunJDK 1.6.0

Switching to 32bit, the spikes are live and kicking - this is not an architecture problem either. Is it a kernel bug? This project is supposed to be hosted on CentOS 5.3, so I tested that next (and last). Spikes are back with a vengeance (~5x higher now). I stopped testing after reaching ridiculous values (~6h for 100,000 calls).

At this point, I can only conclude that this is either a long-standing bug in the Linux kernel or a misuse of kernel API by the JVM implementations - I know I used two different JVMs, but they are very similar nonetheless. Testing with JDK 1.5.0 would be probably better, but I did not have the time to do that, too.

What can we learn from all these? A lesson we already know: never assume, always test. At least in this case, I believe the findings were pretty unexpected. Should I have just gone ahead with EJB and done some profiling on the final application, I would have only seen an application that does not scale well past a certain point (or unexpected connection failured, had I been targeting WinXP). And I would have been lucky to be able to fix that after a lot of working hours, if at all.


PS: Attached, you will find an archive of the test code, should you wish to confirm my findings or test additional combos.
PPS: I know you can always write another article about how sloppy the code is, please don't bring that to my attention. It's good enough to serve its purpose ;-)