Introducing Bolt, Neo4j's Upcoming Binary Protocol – Part 1

I recently had the opportunity to talk with the "Bolt" project’s co-founder, Nigel Small, about the development of the upcoming Bolt protocol and how it fits in with Neo Technology’s overall strategy for the also-upcoming Neo4j 3.0.  Read on to see what Nigel has to say!

(This is the first part of a two-part series—read Part 2 here.)

Neo4j is arguably the most well-known and most popular option available today in the graph database space.  With its excellent performance and its expressive querying language, Cypher, Neo Technology’s answer to what I call “Join Hell” and to traversing relationship-dense data has progressed leaps and bounds since being released as a JVM-based platform in 2010.  The database has evolved from being available only as an embedded process in a JVM-based application to being available as a traditional, detached RDBMS setup would be.

Neo4j 2.0, released in 2013, made this clustering possible.

In order to facilitate communication with the database in what is affectionately known as “server mode,” a series of RESTful APIs were implemented, thereby removing the need for those applications using Neo4j to be of the JVM variety.  Awesome!

Now, as any developer who has ever implemented an application requiring a database with a standard architecture will tell you, the bane of his/her existence is database performance while under load.

And, as any developer who has had to consume services over the wire will tell you, consuming services over HTTP(S) isn’t always the fastest.  (One needs only think about the difference between communicating directly with a database versus communicating with an application stack that includes a database all while incurring everyone’s favorite overhead cost: Communicating via HTTP(S).)

So, as you can imagine, not everyone in the world was entirely happy with Neo4j’s query performance over its REST services.

“Well, duh,” you’re all saying collectively; I’m also positive numerous faces are being palmed.

The good news is that the good folks over at Neo Technology also knew this and have been hard at work addressing this concern.  As part of Neo4j’s upcoming 3.0 release (stay tuned for upcoming articles about this!), a team of Neo engineers has been working on what every killer database needs: A binary protocol.

Called “Bolt”, this implementation promises to make Neo4j even more accessible to various tech stacks along with bringing the performance that every project needs.

Being a Neo4j-head myself (I really should get a job making up words), I just had to find out more about Bolt.

Duncan Brown: Nigel, thanks for taking the time to speak with me today!  First off, when was the genesis of “Bolt”?

Nigel Small: The project itself has been going since the start of 2015 but the desire to do this - throughout much of the company and community - has been around for some time longer. Neo Technology veteran Jake Hansson and I began the "New Remoting" project back in January with the aim to equalize the experience across languages and improve performance of client-server interactions.

DB: So was Bolt planned as a REST services follow-on?  Or did the need for a binary protocol come from a more feedback-based point of view?

NS: The REST API itself was vital to move Neo4j from an embedded product to a client-server one. Bolt is the next step in that journey and will take us firmly outside our comfort zone, the Javasphere. We have tried to take on-board feedback from the community at large and my own personal experiences with building the Python driver, py2neo, have played a big part too.

DB: Very cool.  Now, given what you said about the evolution of Neo4j, is Bolt part of an overarching plan to eliminate Neo4j embedded?  Will Neo4j 3.x still support the embedded mode?

NS: There are no plans to deprecate the core API: It will still exist but we want to give people a good reason to not have to use it. Right now, embedded mode is necessarily bound to Java and that puts some people off using it. We want to make the new surface so compelling that, for most people, it becomes a no-brainer to choose it.

DB: Bringing performance further in-line with Neo4j running in embedded mode would certainly go a very long way in creating some more converts. 

Continuing along the same lines of the product’s evolution, what does the future hold for Neo4j’s REST services?  Will it co-exist with Bolt for the foreseeable future?

NS: HTTP/REST will also remain in the product for the time being. It will coexist along with Bolt but Bolt is intended to be a complete (and improved) replacement for HTTP. There are still some good use cases for an HTTP API as well so, while we haven't made a definitive decision of what will happen to it, we're more likely to bring HTTP in-line with Bolt at some point in the future rather than remove it completely.

DB: How’s the performance of Bolt looking so far (even though it’s basically alpha or pre-alpha)?  Do you have any numbers you can share?

NS: We've been measuring performance as we've been going along and, while we're not ready to share any numbers yet, we've seen many cases where it's significantly faster than HTTP.

DB: That’s definitely going to turn some heads.  Now, do you have any goals or “ideals” you’re looking to hit?  What’s the expected performance of Bolt once released?

NS: Ultimately, we're aiming for the same order of magnitude performance as embedded. Although with a network stack in play too, that'll obviously be affected by the configuration of your infrastructure.

We're also working closely with our customers and community to ensure we optimize for the right use cases and cater for all the features that people need.

DB: On the question of drivers that will utilize this protocol, will any “official” drivers be released alongside of the official Bolt release?

NS: We'll be releasing four drivers along with the 3.0 server release; these will be for Java, Python, JavaScript and .NET and have been built in conjunction with specialists from each of those ecosystems. The Neo Technology drivers themselves will provide a uniform, minimal API and will be perfectly complemented by community drivers, which can add enormous value. We'll certainly be working with the community, not competing with them and plan to introduce a driver compliance program at some point during the 3.x series specifically for this purpose.

In addition, the community can add support for languages that we don't yet provide support for ourselves. We want to add more languages in future and will grow our own driver suite along with user need.

DB: Any news/plans on a JDBC implementation to replace the one that has been used in the Spring Data Neo4j project?

NS: There will be a Bolt-compatible offering built on top of the Java driver.

(Stay tuned for the conclusion of this interview!)