Mark Needham

I wrote about cypher’s MERGE function a couple of weeks ago, and over the last few days, I’ve been exploring how it works with schema indexes and unique constraints. An Exciting Time to Be a Developer There is so much that could be said about the merging of Neo4j and Cypher right now, but it is certainly reasonable to point out that this merger will likely result in many exciting developments in the programming world. Programmers virtually always appreciate it when they are given the products and tools they require to get their job done properly, and now is the time for steps like this to be taken. The fact that Neo4J and Cypher have decided to merge means that the upsides of both will soon be apparent. You deserve to use all of the best tools to make informed decisions about your next software project, and a great way to make it happen is to use what has been given to you regarding product functionality. This is to say that you can use both the upsides of Neo4J and Cypher to come up with the exact tools you need to make a difference in your sphere of influence. Could Other Products Soon Merge? There has been some strong demand for other software development products to consider merging. Coders and programmers want to use their favorite projects in exactly how they were meant to be used, and this means getting them to merge in ways that are useful to the programmers. They just want to be able to squeeze as much use out of each program as they possibly can. You want to make sure that you can see what is going on with your codes as you are directly applying them to whichever problem you are working on at this time. To be sure, it is not an easy task, but no one ever said it would be easy. The important thing is that you get the work done so that you can start to become more productive in the coding you are doing now. A common use case with Neo4j is to model users and events where an event could be a tweet, Facebook post, or Pinterest pin. The model might look like this: We’d like to ensure that we don’t get duplicate users or events, and MERGE provides the semantics to do this: MERGE (u:User {id: {userId}) MERGE (e:Event {id: {eventId}) MERGE (u)-[:CREATED_EVENT]->(m) RETURN u, e We’d like to ensure that we don’t get duplicate users or events and MERGE provides the semantics to do this: MERGE ensures that a pattern exists in the graph. Either the pattern already exists, or it needs to be created. import org.neo4j.cypher.javacompat.ExecutionEngine; import org.neo4j.cypher.javacompat.ExecutionResult; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.graphdb.factory.GraphDatabaseFactory; import org.neo4j.helpers.collection.MapUtil; import org.neo4j.kernel.impl.util.FileUtils; ... public class MergeTime { public static void main(String[] args) throws Exception { String pathToDb = "/tmp/foo"; FileUtils.deleteRecursively(new File(pathToDb)); GraphDatabaseService db = new GraphDatabaseFactory().newEmbeddedDatabase( pathToDb ); final ExecutionEngine engine = new ExecutionEngine( db ); ExecutorService executor = Executors.newFixedThreadPool( 50 ); final Random random = new Random(); final int numberOfUsers = 10; final int numberOfEvents = 50; int iterations = 100; final List userIds = generateIds( numberOfUsers ); final List eventIds = generateIds( numberOfEvents ); List merges = new ArrayList<>( ); for ( int i = 0; i < iterations; i++ ) { Integer userId = userIds.get(random.nextInt(numberOfUsers)); Integer eventId = eventIds.get(random.nextInt(numberOfEvents)); merges.add(executor.submit(mergeAway( engine, userId, eventId) )); } for ( Future merge : merges ) { merge.get(); } executor.shutdown(); ExecutionResult userResult = engine.execute("MATCH (u:User) RETURN u.id as userId, COUNT(u) AS count ORDER BY userId"); System.out.println(userResult.dumpToString()); } private static Runnable mergeAway(final ExecutionEngine engine, final Integer userId, final Integer eventId) { return new Runnable() { @Override public void run() { try { ExecutionResult result = engine.execute( "MERGE (u:User {id: {userId})\n" + "MERGE (e:Event {id: {eventId})\n" + "MERGE (u)-[:CREATED_EVENT]->(m)\n" + "RETURN u, e", MapUtil.map( "userId", userId, "eventId", eventId) ); // throw away for ( Map row : result ) { } } catch ( Exception e ) { e.printStackTrace(); } } }; } private static List generateIds( int amount ) { List ids = new ArrayList<>(); for ( int i = 1; i <= amount; i++ ) { ids.add( i ); } return ids; } } We create a maximum of 10 users and 50 events and then do 100 iterations of random (user, event) pairs with 50 concurrent threads. Afterward, we execute a query that checks how many users of each id have been created and get the following output: +----------------+ | userId | count | +----------------+ | 1 | 6 | | 2 | 3 | | 3 | 4 | | 4 | 8 | | 5 | 9 | | 6 | 7 | | 7 | 5 | | 8 | 3 | | 9 | 3 | | 10 | 2 | +----------------+ 10 rows Next, I added a schema index on users and events to see if that would make any difference, something Javad Karabi recently asked on the user group. CREATE INDEX ON :User(id) CREATE INDEX ON :Event(id) We wouldn’t expect this to make a difference as schema indexes don’t ensure uniqueness, but I ran it anyway t and got the following output: +----------------+ | userId | count | +----------------+ | 1 | 2 | | 2 | 9 | | 3 | 7 | | 4 | 2 | | 5 | 3 | | 6 | 7 | | 7 | 7 | | 8 | 6 | | 9 | 5 | | 10 | 3 | +----------------+ 10 rows If we want to ensure the uniqueness of users and events, we need to add a unique constraint on the id of both of these labels: CREATE CONSTRAINT ON (user:User) ASSERT user.id IS UNIQUE CREATE CONSTRAINT ON (event:Event) ASSERT event.id IS UNIQUE Now if we run the test, we’ll only end up with one of each user: +----------------+ | userId | count | +----------------+ | 1 | 1 | | 2 | 1 | | 3 | 1 | | 4 | 1 | | 5 | 1 | | 6 | 1 | | 7 | 1 | | 8 | 1 | | 9 | 1 | | 10 | 1 | +----------------+ 10 rows We’d see the same result if we ran a similar query checking for the uniqueness of events. As far as I can tell, this duplication of nodes that we merge on only happens if you try to create the same node twice concurrently. Once the node has been created, we can use MERGE with a non-unique index, and a duplicate node won’t get created. All the code from this post is available as a gist if you want to play around with it.

We take a look at using Leaflet.js to help visualize a route on a map with the ability to ensure that the map is zoomed to show all of the points on a given route.

It's easier than ever to run Neo4j backup jobs against Kubernetes clusters. Check out how to use a Cronjob to execute a backup now that Kubernetes 1.8 has been released.

I came across some issues when I tried do analyze data about Russian Twitter trolls, but ran into some issues. Here's how I solved them.

Learn how to solve an error in Python that says "TypeError: unsupported format string passed to numpy.ndarray.__format__."

Explore the wonderful world of persistent storage as we learn how to copy datasets to a Kubernetes PersistentVolume using Neo4j as our sample DB.

When we first created the Kubernetes templates, I wrote a blog post about it. In the comments, someone suggested that we should create a Helm package for Neo4j. 11 months later… we have it!

I accidentally ended up with a bunch of duplicate nodes on a graph that I'd failed to put unique constraints on. Oops! Here's how I fixed it.

I had the code to create sparklines inside a Pandas DataFrame, but I had to tweak it a bit to get it to play nicely with Python 3.6. Here's what I did.

If you're running into 502 errors when making HTTP calls with Lambda, make sure you read the manual and are returning maps.

If you're a Python fan who enjoys using the Serverless library and virtualenv, you might be running into a dependency error. Here's one solution to the problem.

I did some experimenting with a dataset I've been playing around with to find any columns/fields that have null values in them. Learn how I did it!

In my time using get_dummies in panda to generate dummy columns for categorical variables to use with scikit-learn, I realized it didn't always work. Here's why.

In this quick tutorial, we go over how to use Shell scripting in a pinch to create working strings of code that will help us make basic calculations.

A query that I initially wrote didn't work since jsonb_array_elements returns a set of boolean values. Instead, we can use a LATERAL subquery to achieve our goal.

AWS Lambda is a solid serverless option, but setting up automatically scheduled events might not be intuitive. Let's see how CloudWatch can solve the problem.

Read on for a quick and easy tutorial on how to use Python's flask library to generate an HTML file and create a static page from there.

In this post we take a look at how to quickly create a Python function using AWS Lambda, including their configurations and uploading them to the service.

Importing CSVs is a great timesaver, but how do you get around the null values lurking within? Fortunately, we can work our way toward a query that handles them in Neo4j.

If you're unable to find your Docker machines and get an error with some confused certificates, try re-running the $ docker-machine env command.fix

See how the APOC library can be used to automatically create relationships when loading data.

In this post, we find out how to add dynamic properties to nodes in Neo4j via Cypher. Read on for more details!

Here we are, the finale! Today, we take a look at LOAD JSON, a piece of the import puzzle that converts JSON to a CSV.

Running Hadoop and having problems with your DataNode? Read on to find out one possible solution.

This quick tutorial shows you how the simple and versatile the find command can help you sort through your data on Neo4j.

In the world of regular expressions, matching characters outside of the usual Latin character set can be a challenge. Read on to find out how author Mark Needham tackled this issue in Python.

How I parse a JSON HTTP chunking stream in Python using meetup.com's API to filter RSVPs for events I'm interested in.

How to use jq (which is like sed for JSON data) to extract some information from a JSON file, and what to do when you encounter the error in the title.

Learn how to convert an RDD to DataFrame in Databricks Spark CSV library.

It’s a sad but true fact: Most data scientists spend 50-80% of their time cleaning and munging data and only a fraction of their time actually building predictive models. This is most often true in a traditional stack, where most of this data munging consists of writing lines upon lines of some flavor of SQL, leaving little time for model-building code in statistical programming languages such as R. These long, cryptic SQL queries not only slow development time but also prevent useful collaboration on analytics projects, as contributors struggle to understand each others’ SQL code. For example, in graduate school, I was on a project team where we used Oracle to store Twitter data. The kinds of queries my classmates and I were writing were unmaintainable and impossible to understand unless the author was sitting next to you. No one worked on the same queries together because they were so unwieldy. This not only hindered our collaboration efforts but also slowed our progress on the project. If we had been using an appropriate data store (like a graph database) we would have spent significantly less time pulling our hair out over the queries. Why Today’s Data Is Different This data-munging problem has persisted in the data science field because data is becoming increasingly social and highly-connected. Forcing this kind of interconnected data into an inherently tabular SQL database, where relationships are only abstract, leads to complicated schemas and overly complex queries. Yet, several NoSQL solutions – specifically in the graph database space – exist to store today’s highly-connected data. That is, data where relationships matter. A lot of data analysis today is performed in the context of better understanding people’s behavior or needs, such as: How likely is this visitor to click on advertisement X? Which products should I recommend to this user? How are User A and User B connected? Written by Nicole White People, as we know, are inherently social, so most of these questions can be answered by understanding the connections between people: User A is similar to User B, and we already know that User B likes this product, so let’s recommend this product to User A. The Good News: Data-Munging No More Data science doesn’t have to be 80% data munging. With the appropriate technology stack, a data scientist’s development process is seamless and short. It’s time to spend less time writing queries and more time building models by combining the flexibility of an open-source, NoSQL graph database with the maturity and breadth of R – an open-source statistical programming language. The combination of Neo4j’s ability to store highly-connected, possibly-unstructured data and R’s functional, ad-hoc nature creates the ideal data analysis environment. You don’t have to spend an hour writing CREATE TABLE statements. You don’t have to spend all day on StackOverflow figuring out how to traverse a tree in SQL. Just Cypher and go. Learn More at OSCON 2015 At my upcoming OSCON session we will walk through a project in which we analyze #OSCON Twitter data in a reproducible, low-effort workflow without writing a single line of SQL. For this highly-connected dataset we will use Neo4j, an open-source graph database, to store and query the data while highlighting the advantages of storing such data in a graph versus a relational schema. Finally, we will cover how to connect to Neo4j from an R environment for the purposes of performing common data science tasks, such as analysis, prediction and visualization.