Visualizing a Set of Hiveplots with Neo4j

what should a graph look like and how can i tell two graphs apart?

these are questions martin krzywinski (genome sciences center, vancouver, bc) has been asking. take a look at the picture below:

it’s the same graph, the same data, visualized 8 different ways. which is the right way? what advantage does one layout give over the other? can you tell it’s the same network? i can’t.

eight layouts might be too much, so let’s just look at one on the next picture:

martin took the spring embedded visualization and tweaked it around. can you tell it’s the same graph, the same data underneath? i can’t.

to tackle this problem, martin invented the hive plot , a perceptually uniform and scalable layout visualization for network visual analytics.

if you want to learn more about hive plots, take a look at his website and this presentation (it is quite large at 20 mb). i cannot do it justice in this short blog post, and in all honestly haven’t had the time to study it properly.

today i just want to give you a little taste of hiveplots. i am going to visualize the github graphs of nine languages you might not have heard of: boo, dylan, factor, gosu, mirah, nemerle, nu, parrot, self. i’m not going to show you how to create the graph this time, because this is real data we are using. you can take a look at it on the data folder in github.

the graph is basically: (language)–(repository)–(user). there are two relationships between repository and user, wrote and forked.

i’ll show you how to get the data out and into our visualization.

def wroterepos(language)
  neo = neography::rest.new
  neo.execute_script("m = [:]
                      g.v.filter{it.type == 'language' && it.name == '#{language}'}
                       .in.transform{m[it.name] = it.in('wrote').gather{it.name}.next()}
                       .iterate()
                      m")
end

we do the same thing but for forked. this may seem a bit strange to you, but what i am doing is kind of like the sql equivalent of a left outer join with gremlin.

def forkedrepos(language)
  neo = neography::rest.new
  neo.execute_script("m = [:]
                      g.v.filter{it.type == 'language' && it.name == '#{language}'}
                       .in.transform{m[it.name] = it.in('forked').gather{it.name}.next()}
                       .iterate()
                      m")
end

now we do some ruby magic to put our data into the json format the visualization wants.

get '/hive/:name' do
  repos        = []
  writers      = [] 
  forkers      = []
  temp_forkers = []
  temp_writers = []

  wroterepos(params[:name]).each_pair do |key, value|
    repos << {"name" => key, "imports" => value, "node_type" => "repo"}
    temp_writers << { "name" => value[0] }
  end

  i = 0
  forkedrepos(params[:name]).each_pair do |key, value|
    repos[i]["imports"] =  repos[i]["imports"] + value
    temp_writers[i]["imports"] = value
    temp_forkers << value
    i += 1
  end

  temp_writers.group_by {|i| i["name"]}.each do |w, f|
    writers << {"name" => w, 
                "imports" => f.collect{|y| y["imports"]}.flatten.uniq, 
                "node_type" => "writer"}
  end

  temp_forkers.flatten.uniq.delete_if{|x| writers.collect{|y| y["name"]}.include?(x)}.each do |f|
    forkers << {"name" => f, 
                "imports" => [], 
                "node_type" => "forker"}
  end

  (repos + writers + forkers).to_json
end

the blue color nodes are our repositories, the yellow nodes are our writers, and the green nodes are our forkers. the 12 o’clock axis (the top) shows nodes with only outgoing relationships. the bottom-left axis shows nodes with only incoming relationships. these are the writers without any forks, and the forkers who never started their own public projects. the remaining nodes in the bottom-right have both incoming and outgoing relationships. these are the repository writers who created projects other people found worth forking.

the graphs are ordered across for each row in the following manner:

• boo, dylan, factor
• gosu, mirah, nemerle
• nu, parrot, self

can you see the similarities between boo, factor and numerle? see how different they are from gosu and self? what does the hive plot tell you about these language github repositories?

you can try a live version at hiveplot.herokuapp.com/index.html and as always the code is available on github .

our visualization was done by rich morin and mike bostock with d3.js . is is a hot off the press work in progress. you can follow the action on this d3.js google group thread .