The Curse of Resource Utilization

i got a request from a customer to look at the resource utilization of ravendb under load. in particular, the customer loaded the entire freedb data set into ravendb, then set up sql replication and a set of pretty expensive indexes (map reduce, full text searches, etc).

this was a great scenario to look at, and we did find interesting things that we can improve on. but during the testing, we had the following recording taken.

in fact, all of them were taken within a minute of each other:

the interesting thing about them is that they don’t indicate a problem. they indicate a server that is making as much utilization of resources as is available to it to handle the work load that it has.

while i am writing this, there is about 1gb of free ram that we’ve left for the rest of the system, but basically, if you have a big server, and you give it a lot of work to do, it is pretty much given that you’ll want to get your money’s worth from all of that hardware. the problem in the customer’s case was that while it was very busy, it didn’t get things done properly, but that is a side note.

what i want to talk about is the assumption that the server is using a lot of resources, that is bad. in fact, that isn’t true, assuming that it is using them properly. for example, if you have a 32gb ram, there is little point in trying to conserve memory utilization. and there is all the reason in the world to try to prepare ahead of time answers to upcoming queries. so we might be using cpu even on idle moments. the key here is how you detect when you are over using the system resources.

if you are under memory pressure, is is best to let go of that cache. and if you are busy handling requests, it is probably better to reduce the load of background tasks. we’ve been doing this sort of auto tuning in ravendb for a while. a lot of the changes between 1.0 and 2.0 were done around that area.