With all the focus on writing to disk and ensuring the high speed of the database, I almost missed a very important aspect of databases: How do we actually communicate with the outside world?
In many cases, there is no need for that. Many databases just run either embedded only (LevelDB, LMDB, etc.) or just assume that this is someone else's problem, pretty much. But as it turns out, if you aren't working with an embedded database, there is quite a bit that you need to get right when you are working on the network code. For the rest of this post, I'm going to assume that we are talking about a database that clients connect to through the network. It can be either locally (shared memory/pipes/UNIX sockets/TCP) or remotely (pretty much only TCP at this point).
The actual protocol format doesn't really matter. I'm going to assume that there is some sort of a streaming connection (think TCP socket) and leave it at that. But the details of how the database communicates are quite important.
In particular, what kind of communication do we have between client and server? Is each request independent of one another, are there conversations, what kind of guarantees do we give for the lifetime of the connection, etc?
Relational databases use a session protocol, in which the client will have a conversation with the server, composed of multiple separate request/replies. Sometimes you can tell the server "just process the requests as fast as you can, and send the replies as well, but don't stop for sending replies," which helps, but this is probably the worst possible way to handle this. The problem is that you need a connection per client, and those conversations can be pretty long, so you hold up to the connection for a fairly long duration, which means that you have to wait for the next part of the conversation to know what to do next.
A better approach is to dictate that all operations are strictly one way, so once the server has finished processing a request, it can send the reply to the client and move on to process the next request (from the same client or the other one) immediately. In this case, we have several options. We can pipeline multiple requests on the same channel (allowing out of order replies for requests/replies), we can use the serial nature of TCP connections and have a pool of connections that we can share among clients.
An even better method is if you can combine multiple requests from the server to a single reply from the server. This can be useful if you know that the shape of all the commands are going to be same (for example, if you are asked in insert a million records, you can process them as soon as possible, and only confirm that you accepted every 10,000 or so). It may sound silly, but those kinds of things really add up because the client can just pour the data in and the server will process it as soon as it can.
Other considerations that you have to take into account in the communication protocol are security and authentication. If you are allowing remote clients, then they need some way to authenticate. And that can be expensive (typically, authentication required multiple round trips between server and client to verify the credentials, and potentially and out of server check in something like LDAP directory), so you want to authenticate once and then keep on to the same connection for as long as you can.
A more problematic issue comes when talking about the content of your conversation with the database, you typically don't want to send everything over plain text, so you need to encrypt the channel, which has its own costs. For fun, this means dealing with things like certificates, SSL, and pretty complex stuff that very few properly understand. If you can get away with letting something else do that for you, see if you can. Setting up a secured connection that cannot be easily bypassed (man in the middle, fake certs, just weak encryption, etc.) is not trivial and should be left for experts in their field.
Personally, I like to just go with HTTPS for pretty much all of that because it takes a lot of the complexity away, and it means that all the right choices have probably already been made for me. HTTPS, of course, is a request/reply, but things like connection pooling, keep-alive and pipelining can do quite a lot to mitigate that on the protocol level. And something like Web Sockets allows you to have most of the benefits of TCP connections with a lot of the complexity handled for you.