DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Real-Time Twitter Heat Map with MongoDB

Real-Time Twitter Heat Map with MongoDB

Comsysto Gmbh user avatar by
Comsysto Gmbh
·
Dec. 05, 12 · Interview
Like (0)
Save
Tweet
Share
10.36K Views

Join the DZone community and get the full member experience.

Join For Free

over the last few weeks i got in touch with the fascinating field of data visualisation which offers great ways to play around with the perception of information.

in a more formal approach data visualisation denotes “ the representation and presentation of data that exploits our visual perception abilities in order to amplify cognition “

nowadays there is a huge flood of information that hit’s us everyday. enormous amounts of data collected from various sources are freely available on the internet. one of these data gargoyles is twitter producing around 400 million (400 000 000!) tweets per day!

tweets basically offer two “layers” of information. the obvious direct information within the text of the tweet itself and also a second layer that is not directly perceived which is the tweets’ metadata . in this case twitter offers a large number of additional information like user data, retweet count, hashtags, etc. this metadata can be leveraged to experience data from twitter in a lot of exciting new ways!

so as a little weekend project i have decided to build a small piece of software that generates real-time heat maps of certain keywords from twitter data.

this is a first static peek on what it’s gonna look like (apparently the friendly floatees use twitter, too):

see a screencast here: screencast.com

to get this working i have used lots of shiny things:

  • twitter streaming api
  • mongodb’s capped collections
  • flask
  • redis
  • tweepy
  • jquery eventsource
  • google maps api
  • heatmap-gmaps.js

the app is written in python and consists of mainly three components:

tstream.py:
a small service based on tweepy that implements a streamlistener which inserts incoming data in a mongodb capped collection. here you can also set filter terms. this example uses mostly terms related to “big data”.

tweet_service.py:
a flask based web app which get’s new data from mongodb and makes use of the publish-subscribe pattern . being “ tailable ” mongodb’s capped collections come in handy. there is no need to remember which messages a client has already received, the cursor itself yields new documents on arrival. also, capped collections are of a fixed size which is appropriate for this use case but your mileage may vary. incoming tweets are published to a redis channel for which there is also a listener that returns a “text/event-stream” “content-type” header for connecting clients.

map.html:
a few lines of html and javascript which bring up a google maps canvas and a listener for server-sent events. when new data, basically consisting of lat/lon tuples, arrives the new point is added to a heat map overlay based on heatmap-gmaps.js in real-time.

rough overview:

conclusion

with a relatively small amount of code it is possible to turn text data in astonishing visualizations. with only a little more effort different hash tags could be illustrated by different colors. or one could count tweets on a topic in certain regions and then compare activity based on number of citizens. there are lots of interpretations possibel through the underlying data set.

you can find the code on github , feel free to fork and play around!



twitter Heat map MongoDB HEAT (software) Data (computing)

Published at DZone with permission of Comsysto Gmbh, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Visual Network Mapping Your K8s Clusters To Assess Performance
  • PostgreSQL: Bulk Loading Data With Node.js and Sequelize
  • Key Considerations When Implementing Virtual Kubernetes Clusters
  • Why Does DevOps Recommend Shift-Left Testing Principles?

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: