Danger lurks in virtualization
I'm pretty sure some of you are working for organisations that make extensive use of virtualization solutions. If you do, and by a chance you happen to develop or maintain Java applications, you may find this post interesting.
Every now and then I am asked to investigate Java related problems such as crashes that end with core dumps. For someone who haven't had a chance of getting deeply into this kind of issues, t One of the first facts you have to establish is if the environment you are investigating is running on a virtualized platform. This is crucial, because there is a chance that the crashes the system is experiencing are due to guest system memory configuration.
When a VM administrator creates a new guest system instance, he very often does not know exactly for what purposes the machine being created will be used. Usually when there is a requirement for a VM to have 8 GB of RAM, the administrator allocates 4 GB of host system's physical memory and adds the remaining 4 from the host's swap. This of course constitutes the amount of memory needed to satisfy the requirement and saves some money at the same time, but may also lead to serious problems if the VM is to run a Java application.
Let's assume our VM with 8 GB of memory (configured as described above) is running an application on 64-bit JVM with maximum heap size of 6 GB (-Xmx6g). Despite the fact that 2 out of 6 GB reserved for our JVM virtually resides in physical machine's swap space, nothing wrong will happen as long as JVM will keep allocating space that resides in host's physical memory. The nighmare begins when the allocations start to occur in host system's swap. This almost instantly leads to a JVM crash that may begin like this one:
===== BEGIN DUMP =============================================================
JRockit dump produced after 0 days, 06:14:50 on Mon Jan 24 18:01:22 2011
Additional information is available in:
/usr/xxx/core or core.779 (max size 1650065408kb)
If you see this dump, please open a support case with BEA and
supply as much information as you can on your system setup and
the program you were running. You can also search for solutions
to your problem at http://forums.bea.com in
the forum jrockit.developer.interest.general.
Error Message: Illegal memory access. 
Signal info : si_signo=11, si_code=1 si_addr=0x4df4004
Version : BEA JRockit(R) R27.2.0-131-78843-1.5.0_10-20070320-1511-linux-ia32
GC Mode : Garbage collection optimized for throughput
GC Strategy : Generational Parallel Mark & Sweep
: Current OC phase is: marking. YC is not running.
: GC strategy for GC 49 was genparpar
: GC strategy for GC 50 was genparpar
: GC strategy for GC 51 was genparpar
: GC strategy for GC 52 was genparpar
: GC strategy for GC 53 was genparpar
: mmHeap->data = 0x8300000, mmHeap->top = 0x18300000
: mmStartCompaction = 0x15300000, mmEndCompaction = 0x16300000
: References are 32-bit.
CPU : Intel Pentium 4 (HT) SSE SSE2 SSE3 NetBurst EM64T
Number CPUs : 2
This of course comes from a JRockit crash, but
The reason for this type of problems to