A Case Study in Database Security

We take security very seriously. With the move to X509 certificates only for authentication (on all RavenDB editions) I feel that we have a really good story around securing RavenDB and controlling access to it.

Almost. One of the more annoying things about security is that you also need to consider the hard cases, such as the administrators messing up badly. As in, losing the credentials that allow you to administrator RavenDB. This can happen because the database has just run without issue for so long that no one can remember where the keys are. That isn’t supposed to happen, but RavenDB has been in production usage for close to a decade now, which means that we have seen our fair share of mess ups (both our own and by customers).

In some cases, we have had to help a customer manage a third system handover between different hosting providers, which very much felt like half forensics and half hacking. In short, when we design a system now, we also consider the fact that as secure as we want the system to be, there must be a way for an authorized person to get in.

If this made you cringe, you are in good company. I both love and hate this feature. I love it because it is going to be very useful, I hate it because it was a headache to figure it out. But I’m jumping ahead of myself. What is this back door that I’m talking about?

Properly configured, RavenDB will require a client certificate (that was registered in the cluster) to access the server. However, in addition to listening over HTTPS, RavenDB will also listen for commands on standard input. An admin can use the standard input/output as a way to talk with RavenDB without requiring any authentication. Basically, we expose a mini shell that you can use to enter commands and inspect and change our state.

Here is how it looks like when running in console mode:

From a security point of view, if a user is able to access my standard input, that usually means that they are the one that has run this process or are able to so. RavenDB obviously won’t have any setuid bits turned on, so no need to worry about a user tricking us to do something that the user doesn't have permission to do.

So using the console is a really nice way for us to offer the administrator an escape hatch to start messing with the internals of RavenDB in an interesting way. However, that only works if you are running RavenDB in interactive mode. What about when you're running it as a service or daemon? They don’t have a standard input that is available to the admin. In fact, in most production deployments, you won’t have an easy time at all trying to connect to the console.

So that option is out, sadly. Or is it?

The nice thing about operating systems is that we can lean on them. In this case, we expose the exact same console that we have for stdin/stdout using Named Pipes (actually, Unix Sockets in Linux/Mac, but pretty much the same idea). The idea is that those are both methods for inter-process communication that are local to the machine and can be secured by the operating system directly. In this case, we make sure that the pipe is only accessible to the RavenDB user (and to root/Administrator, obviously). That means that an admin can log into the box, run a single command, and land in the RavenDB admin shell where he can manage the server. For example, by registering a new certificate in the server.

Because only the user running the RavenDB process or an administrator/root can access the pipe (ensured by setting the proper ACL on the pipe during creation) we know that there isn’t any security risk here. An admin can already override any security in the box, and the permissions are always on the user level, not the process level, so if you are running as the same user as the RavenDB process you can already do anything that RavenDB can do.

After we ensured that our security isn’t harmed by this option, we can relax knowing that we have an easy (and safe) way for the administrator to manage the server in an emergency.

In fact, the most obvious use of this feature is during the initial cluster setup, when you don’t have anything yet. This allows you to enter the system as a trusted party and do the initial configuration.