Over a million developers have joined DZone.

Importing a Twitter Archive into ElasticSearch

DZone's Guide to

Importing a Twitter Archive into ElasticSearch

· Performance Zone
Free Resource

Learn how real real-time monitoring is critical for DevOps. Because you can't build what you can't see.

I recently downloaded my full history of tweets from Twitter (via the "Request your archive" button on the Settings page), which gives the data in CSV and JSON formats. The download comes with a simple web UI so you can browse locally, all your historical tweets, as well as perform basic searches. When I tried performing a wildcard search, it resulted in an unresponsive script warning and nobbled the browser tab.

Cue ElasticSearch

I've kept meaning to have another hack around with ElasticSearch for a while now, as I started looking into it in 2011 and was very impressed with it. So I decided to knock up a little .NET app to take the .js files provided by Twitter in the export, extract the tweet data and push into ElasticSearch. It seems like a good little starter project to learn more about ElasticSearch and actually could give me some useful insights into all the tweets I've made. I'm sure I'm going to find some really good links I tweeted and then couldn't find again!

At the moment there's a very basic GUI (and I mean basic!) exposing the functionality to import the tweet data into ElasticSearch. I plan to stick a basic search GUI together too and then...well, I'm not quite sure yet, but I'm sure as time goes on I'll see more ways to extract interesting information from my tweet history.

The ElasticTweets project as I've called it, can be found on GitHub here - feel free to use and abuse it, and let me know how you get on!

Get real-time alerts and visualizations across your cloud infrastructure for real real-time cloud monitoring. Try it FREE now


Published at DZone with permission of Adrian Hills, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}