Neo4j on Heroku - Part 2

we are picking up where we left off on neo4j on heroku –part one so make sure you’ve read it or you’ll be a little lost. so far, we have cloned the neoflix project , set up our heroku application and added the neo4j add-on to our application. we are now ready to populate our graph.

bring up two browser windows. on one you’ll go to your neo4j instance running on heroku,

$ heroku config
neo4j_url      => http://xxxxxxxx:yyyyyyyy@70825a524.hosted.neo4j.org:7014

and on the other you’ll go to the create_graph route of your app. so if you named your app neoflix, you’d go to neoflix dot herokuapp dot com/create_graph.

this will run the create_graph method and you’ll see nodes and relationships being created on the neo4j dashboard. it’s just over a million relationships, so it will take a few minutes. there are faster ways to load data into neo4j (wait for part three of this series), but this will work in our case.

the fine folks at themoviedb.org provide an api for any developers that want to integrate movie and cast data along with posters or movie fan art. you can request an api key and they’ll respond very quickly. so let’s add this to our heroku configs.

heroku config:add tmdb_key=xxxxxxx
adding config vars and restarting app... done, vxx
  tmdb    => xxxxxxx

if you want to test locally you can do so by:

export tmdb_key=xxxxxxx

we can now use this environment variable on our application along with the ruby-tmdb gem by aaron gough :

require 'ruby-tmdb'

tmdb.api_key = env['tmdb_key']
tmdb.default_language = "en"

  def get_poster(data)
    movie = tmdbmovie.find(:title => cgi::escape(data["title"] || ""), :limit => 1)
    if movie.empty?
     "no movie poster found"
    else
      "<a href="#{movie.url}" target='_blank'>
       <img src="#{movie.posters.first.url}">
       <h3>#{movie.tagline}</h3>
       <p>rating: #{movie.rating} <br />
          rated: #{movie.certification}</p><p>#{movie.overview}</p>"
    end
  end

we will visualize the graph like i showed you earlier using neovigator, but instead of retrieving the properties of our node (since they’re pretty bland), we’ll request a movie poster.

we will not visualize the explicit relationships we created. instead we will visualize the implicit movie recommendations graph. let’s take a look at that method now:

def get_recommendations(neo, node_id)
  rec = neo.execute_script("m = [:];
                            x = [] as set;
                            v = g.v(node_id);

                            v.
                            out('hasgenera').
                            aggregate(x).
                            back(2).
                            ine('rated').
                            filter{it.getproperty('stars') > 3}.
                            outv.
                            oute('rated').
                            filter{it.getproperty('stars') > 3}.
                            inv.
                            filter{it != v}.
                            filter{it.out('hasgenera').toset().equals(x)}.
                            groupcount(m){\"${it.id}:${it.title.replaceall(',',' ')}\"}.iterate();

                            m.sort{a,b -> b.value <=> a.value}[0..24];",
                            {:node_id => node_id.to_i})

  return [{"id" => node_id,
           "name" => "no recommendations",
           "values" => [{"id" => "#{node_id}",
                         "name" => "no recommendations"}]
          }] if rec == "{}"

  values = rec[1..rec.size-1].split(',').collect{ |v| {:id => v.split(':')[0].strip, 
                                                       :name => v.split(':')[1] } }

  [{"id" => node_id ,"name" => "recommendations","values" => values }]
end

let’s go through the code. in groovy [:] is a map (equivalent to a ruby hash) and ultimately what we want to return, so we’ll create an empty one and fill it later. then we’ll create a set “x” (which is an unordered collection see groovy list for ordered collections). we also get our starting vertex and assign it to “v”.

m = [:];
x = [] as set;
v = g.v(node_id);

we will fill the empty set we created with the generas of our movie and we’ll compare the generas of other movies against it later on.

v.
out('hasgenera').
aggregate(x).

we then go back 2 steps, which puts us at our starting movie and go to the users that have rated the movie with more than 3 stars.

back(2).
ine('rated').
filter{it.getproperty('stars') > 3}.

from these users, we step out to find all the movies they have also rated with more than 3 stars.

outv.
oute('rated').
filter{it.getproperty('stars') > 3}.

which are not our starting movie (remember we set it to the variable “v”).

inv.
filter{it != v}.

…and we check that these movies have the same generas as our starting movie (remember we filled the set “x”).

filter{it.out('hasgenera').toset().equals(x)}.

groupcount does what it sounds like and stores the value in the map “m” we created earlier. however, we want to get the id, title and count, so we do a little string wrangling to get both id and title (minus commas… i’ll tell you why in a minute) and iterate() . the gremlin shell iterates automatically for you, but since we’re sending this gremlin script over the rest api, it doesn’t. one day you’ll be pulling out your hair trying to figure out what’s wrong and you’ll curse “iterate” once you figure it out…

groupcount(m){\"${it.id}:${it.title.replaceall(',',' ')}\"}.iterate();

here we sort our map (b has the count) and get the top 25 entries.

m.sort{a,b -> b.value <=> a.value}[0..24];",

since neo4j will be executing this code many times over, you want to parametize it, so it parses it only once.

{:node_id => node_id.to_i})

if we get an empty hash back, we’ll return an unfortunate “no recommendations” message,

return [{"id" => node_id,
         "name" => "no recommendations",
         "values" => [{"id" => "#{node_id}",
                       "name" => "no recommendations"}]
        }] if rec == "{}"

finally we structure our groovy map into an array of hashes which we use in our visualization like i showed you with neovigator . notice i’m splitting the record by commas (hence why we substituted them earlier). this piece won’t be necessary very soon as the final version of neo4j 1.6 will have json support for groovy maps.

values = rec[1..rec.size-1].split(',').collect{ |v| {:id => v.split(':')[0].strip, 
                                                     :name => v.split(':')[1] } }
[{"id" => node_id ,"name" => "recommendations","values" => values }]

we save the results of getting a movie poster and its recommendations for 30 days by taking advantage of the varnish cache provided to us by heroku. we then get our starting node either by id or by title.

get '/resources/show' do
  response.headers['cache-control'] = 'public, max-age=2592000'
  content_type :json

  if params[:id].is_numeric?
    node = neo.get_node(params[:id])
  else
    node = neo.execute_script("g.idx(tokens.t.v)[[title:'#{cgi::unescape(params[:id])}']].next();")
  end

  id = node_id(node)

  {:details_html => "<h2>#{get_name(node["data"])}</h2>" + get_poster(node["data"]),
   :data => {:attributes => get_recommendations(neo, id),
             :name => get_name(node["data"]),
             :id => id}
   }.to_json
end

by title? yes, we are adding jquery ui autocomplete to our application. which will pass the name of the movie and look it up in the automatic index we created.

node = neo.execute_script("g.idx(tokens.t.v)[[title:'#{cgi::unescape(params[:id])}']].next();")

… and there you have it. your very own movie recommendation website on heroku. see the complete code at github.com/maxdemarzi/neoflix .

update: looks like a few of you dear readers tried to run create_graph multiple times and it made a mess. i will try to fix it and get it back up soon. note to future self: remove create_graph route on heroku before publishing post.

source: http://maxdemarzi.com/2012/01/16/neo4j-on-heroku-part-two/