Real-Time Twitter Heat Map with MongoDB

over the last few weeks i got in touch with the fascinating field of data visualisation which offers great ways to play around with the perception of information.

in a more formal approach data visualisation denotes “ the representation and presentation of data that exploits our visual perception abilities in order to amplify cognition “

nowadays there is a huge flood of information that hit’s us everyday. enormous amounts of data collected from various sources are freely available on the internet. one of these data gargoyles is twitter producing around 400 million (400 000 000!) tweets per day!

tweets basically offer two “layers” of information. the obvious direct information within the text of the tweet itself and also a second layer that is not directly perceived which is the tweets’ metadata . in this case twitter offers a large number of additional information like user data, retweet count, hashtags, etc. this metadata can be leveraged to experience data from twitter in a lot of exciting new ways!

so as a little weekend project i have decided to build a small piece of software that generates real-time heat maps of certain keywords from twitter data.

this is a first static peek on what it’s gonna look like (apparently the friendly floatees use twitter, too):

see a screencast here: screencast.com

to get this working i have used lots of shiny things:

• twitter streaming api
• mongodb’s capped collections
• flask
• redis
• tweepy
• jquery eventsource
• google maps api
• heatmap-gmaps.js

the app is written in python and consists of mainly three components:

tstream.py:
a small service based on tweepy that implements a streamlistener which inserts incoming data in a mongodb capped collection. here you can also set filter terms. this example uses mostly terms related to “big data”.

tweet_service.py:
a flask based web app which get’s new data from mongodb and makes use of the publish-subscribe pattern . being “ tailable ” mongodb’s capped collections come in handy. there is no need to remember which messages a client has already received, the cursor itself yields new documents on arrival. also, capped collections are of a fixed size which is appropriate for this use case but your mileage may vary. incoming tweets are published to a redis channel for which there is also a listener that returns a “text/event-stream” “content-type” header for connecting clients.

map.html:
a few lines of html and javascript which bring up a google maps canvas and a listener for server-sent events. when new data, basically consisting of lat/lon tuples, arrives the new point is added to a heat map overlay based on heatmap-gmaps.js in real-time.

rough overview:

conclusion

with a relatively small amount of code it is possible to turn text data in astonishing visualizations. with only a little more effort different hash tags could be illustrated by different colors. or one could count tweets on a topic in certain regions and then compare activity based on number of citizens. there are lots of interpretations possibel through the underlying data set.

you can find the code on github , feel free to fork and play around!