DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Send Email Using Spring Boot (SMTP Integration)
  • What to Pay Attention to as Automation Upends the Developer Experience
  • Using OpenAI Embeddings Search With SingleStoreDB
  • 4 Expert Tips for High Availability and Disaster Recovery of Your Cloud Deployment

Trending

  • Send Email Using Spring Boot (SMTP Integration)
  • What to Pay Attention to as Automation Upends the Developer Experience
  • Using OpenAI Embeddings Search With SingleStoreDB
  • 4 Expert Tips for High Availability and Disaster Recovery of Your Cloud Deployment
  1. DZone
  2. Data Engineering
  3. Databases
  4. Neo4j at Ludicrous Speed

Neo4j at Ludicrous Speed

Max De Marzi user avatar by
Max De Marzi
·
Mar. 06, 14 · Interview
Like (0)
Save
Tweet
Share
4.71K Views

Join the DZone community and get the full member experience.

Join For Free

spaceballs_ludicrous_speed

in the last blog post we saw how we could get about 1,250 requests per second (with a 10ms latency) using an unmanaged extension running inside the neo4j server… but what if we wanted to go faster?

the easy answer is to scale up . however, trying to add more cores to my apple laptop doesn’t sound like a good time. another answer is running a neo4j cluster and (almost) linearly scaling our read requests as we add more servers. so a 3 server cluster would give us between 3,500 and 3,750 requests per second.

but can we go faster on a single server without new hardware? well… yes .

neo4j started its life as an embeddable java library, and running embedded is still a valid way to deploy neo4j. however it’s mostly reserved to customers who use java as their primary language. for us non-java folks, lets take a peek at one way to do this.

i was looking around for a really simple and performant java web server and ran in to undertow .

undertow_banner

undertow looks really easy to use and scores high marks in the techempower web framework benchmarks

we will re-use the graph database we created in the previous blog post, and just point to it. we’ll use the graphdatabasefactory to accomplish this:

static graphdatabaseservice graphdb = new graphdatabasefactory()
        .newembeddeddatabasebuilder( storedir )
        .loadpropertiesfromfile( pathtoconfig + "neo4j.properties" )
        .newgraphdatabase();

the first thing we’ll do is register a shutdown hook for our neo4j database so it shuts down correctly when you stop the server. next we’ll build our undertow server, having it listen on the same port as neo4j. we’ll have two paths here. the root path will just say “hello world” just to make sure everything is wired up and we can use it to test the baseline performance of undertow on our system. the second path will match our unmanaged extension so we can reuse our performance test.

public static void main(final string[] args) {
    registershutdownhook(graphdb);
    undertow server = undertow.builder()
            .addlistener(7474, "localhost")
            .sethandler(new pathhandler()
                    .addpath("/", new helloworldhandler())
                    .addpath("/example/service/crossreference", new crossreferencehandler(objectmapper, graphdb))
            ).build();
 
    server.start();
}

let’s take a look at that helloworldhandler first. it handles the request by simply responding with the plain text hello world.

public class helloworldhandler implements httphandler {
    @override
    public void handlerequest(final httpserverexchange exchange) throws exception {
        exchange.getresponseheaders().put(headers.content_type, "text/plain");
        exchange.getresponsesender().send("hello world");
    }
}

we can test the performance of this request with a little gatling :

val base = scenario("get hello world")
   .during(30) {
     exec(
       http("get base request")
         .get("/")
         .check(status.is(200))
     )
     .pause(0 milliseconds, 1 milliseconds)
 }
 
setup(
   base.users(16).protocolconfig(httpconf)
 )

…and our results are just under 20k requests per second:

screen shot 2014-02-27 at 11.25.39 am

now let’s take a look at our crossreferencehandler . you’ll notice we are passing in the graphdb we created earlier and the objectmapper. it’s not needed since we only have one end point, but if you had many you don’t want to recreate those objects every time. in fact, neo4j won’t let you since it needs exclusive access to the graph.db directory.

public class crossreferencehandler implements httphandler {
    private static final relationshiptype related = dynamicrelationshiptype.withname("related");
    objectmapper objectmapper;
    graphdatabaseservice graphdb;
 
    public crossreferencehandler(objectmapper objectmapper, graphdatabaseservice graphdb){
        this.objectmapper = objectmapper;
        this.graphdb = graphdb;
    }

when we handle the request, we first check to see if the request is a post method, then we will get the inputstream of the request (a json blob) and convert it to a hashmap. the funny startblocking() call is required to read the inputstream.

public void handlerequest(final httpserverexchange exchange) throws exception {
     try {
         if (exchange.getrequestmethod().equals(methods.post)) {
             exchange.startblocking();
             final inputstream inputstream = exchange.getinputstream();
             final string body = new string(bytestreams.tobytearray(inputstream), charsets.utf_8);
             hashmap input = objectmapper.readvalue(body, hashmap.class);
             ...   

from here our code follows exactly what our unmanaged extension did, and responds with the answer in json format:

exchange.getresponseheaders().put(headers.content_type, "application/json; charset=utf-8");
exchange.getresponsesender().send(bytebuffer.wrap(objectmapper.writevalueasbytes(results))); 

when we test this using the existing performance test we get:

screen shot 2014-02-27 at 11.29.50 am

over 8000 requests per second with a latency of 1ms. thats about 6.5x the number of requests we we able to do before and our mean latency dropped to just 1ms…on my laptop. now that’s fast! the complete source code is available on github as always, try it out with your own neo4j projects.

i’ll leave you with other options including running neo4j embedded with ratpack on stefan armbruster’s blog…. and even more fun ideas from nigel small :

have been experimenting with a different way to wrap #neo4j as a server, using zeromq - https://t.co/2ek8wgs9ey

— nigel small (@technige) february 22, 2014

if you have a problem that is graphy in nature and your relational database just isn’t cutting it, you gotta try neo4j.

Neo4j

Published at DZone with permission of Max De Marzi, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Send Email Using Spring Boot (SMTP Integration)
  • What to Pay Attention to as Automation Upends the Developer Experience
  • Using OpenAI Embeddings Search With SingleStoreDB
  • 4 Expert Tips for High Availability and Disaster Recovery of Your Cloud Deployment

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: