If you've read some other articles on this blog you might know that I like creating visualizations of various datasets. I've just started a small project where I want to visualize some data from Twitter. For this I want to retrieve information about followers and profile information directly from twitter. I actually started looking for a set of all twitter accounts, but could only find one that was two years old. So, only option left, directly access the twitter API and get the data myself.

There are a couple of open source libraries out that we can use directly from Scala (or java) but as far as I could see they use the old v1 API and not the v1.1 API. The old API has a very strict data rate limit, which is a bit lighter in the new API. And besides that I'm more interested in the raw data and parsing the returning JSON isn't that hard with Scala (or Java for that matter).

Register an application at twitter

First thing to do, and easiest way to get started is registing a new application for your twitter account. Go to https://dev.twitter.com/apps/new and create a new application. Don't worry about the urls, since we won't be using the OAuth callback mechanism:

Depending on what you want to do with the API you need to give additional permissions to this app. Default is "read-only", if you want to allow your new application to post or access your direct messages you need to update the permissions. This is done from the settings page of your application:

Once you've created the application and setup the correct permissions you can generate an access token. Doing this will avoid having to go through the complete OAuth dance. You do this by going to your new app details and at the bottom select the "create my access token" option.

Now you'll have a set of tokens (see the details part of your applications):

We'll use these tokens to authenticate the requests we'll make to twitter.

Using a OAuth library

The OAuth protocol is a pretty good documented protocol, but implementing it yourself is a lot of work and very error-prone. Luckily there are many OAuth libraries that can help you. I've tried a couple and the one that's most easy to use (at least for me) was signpost. The examples below show how to do this from Scala, but you can follow the same approach for Java.

First the dependencies. I've used sbt and from signpost I use the client with support for HTTP commons. In sbt add the following:

..
libraryDependencies ++= Seq(
	"oauth.signpost" % "signpost-core" % "1.2",
	"oauth.signpost" % "signpost-commonshttp4" % "1.2", 
	"org.apache.httpcomponents" % "httpclient" % "4.2",
         ...
)

For maven you can use the same libraries. Next we can write a simple test to see if everything is working. In java it looks like this:

import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
 
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
 
 
public class Tw {
 
	  static String AccessToken = "access token for your app";
	  static String AccessSecret = "access secret for your app";
	  static String ConsumerKey = "consumer key for your app";
	  static String ConsumerSecret = "consumer secret for your app";
 
	/**
	 * @param args
	 */
	public static void main(String[] args) throws Exception {
		OAuthConsumer consumer = new CommonsHttpOAuthConsumer(
                ConsumerKey,
                ConsumerSecret);
 
        consumer.setTokenWithSecret(AccessToken, AccessSecret);
        HttpGet request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
        consumer.sign(request);
 
        HttpClient client = new DefaultHttpClient();
        HttpResponse response = client.execute(request);
 
        int statusCode = response.getStatusLine().getStatusCode();
        System.out.println(statusCode + ":" + response.getStatusLine().getReasonPhrase());
        System.out.println(IOUtils.toString(response.getEntity().getContent()));
	}
}

And in Scala it looks pretty much the same:

import org.apache.http.client.HttpClient
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpGet
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer
import org.apache.commons.io.IOUtils
 
 
object TwitterPull {
 
	  val AccessToken = "access token for your app";
	  val AccessSecret = "access secret for your app";
	  val ConsumerKey = "consumer key for your app";
	  val ConsumerSecret = "consumer secret for your app";
 
 
  def main(args: Array[String]) {
 
	 val consumer = new CommonsHttpOAuthConsumer(ConsumerKey,ConsumerSecret);
	 consumer.setTokenWithSecret(AccessToken, AccessSecret);
 
     val request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
     consumer.sign(request);
     val client = new DefaultHttpClient();
     val response = client.execute(request);
 
     println(response.getStatusLine().getStatusCode());
     println(IOUtils.toString(response.getEntity().getContent()));
  }
}

When you run this the output will look something like this:

200
{"previous_cursor_str":"0","next_cursor":0,"ids":[48342167,21011010,824959303,97242821,16953163,218083367,20869799,5234221,13604142,804783128,271050984,405121284,26470609,50201837,1723451,374494377,120867838,14311946,253114713,39554511,7375412,42507395,112806109,92787154,218238023,110443797,76922155,198798790,294104985,305625416,217698029,21803482,14927822,15453445,15715866,15657036,186956616,36028164,70380613,326158542,573546312,14401332,521488579,9108612,576970378,293236313,16398366,16220300,15234937,32000283,439444353,14300622,67204409,155850135,14198255,32264673,15852981,313248158,20123099,608942046,234930032,36896958,18466675,45496942,330899833,18980755,88253383,461023805,31175627,11044952,142780445,63175189,107991607,94830953,600993241,6195002,115391430,550080945,381418927,168603682,142388604,8258462,218411138,30450578,77728346,2521381,182867524,494119147,29426983,572417260,94344849,325413275,389354525,501438275,164346498,22730282,8293302,21085554,341645357,56978853,180507788,10074002,22536424,14247654,581293627,15259428,483317230,462826270,4774641,15366832,96850673,278486993,22273826,17716679,14566626,158473088,20461042,161242434,43756629,40163100,141165981,5325152,7620782,266749648,524476136,557713614,39602637,18843154,1623,565954426,39639621,166672305,18683074,233118689,44876099,235258223,219310062,10699922,12660502,218030046,91552210,19361980,206645598,35346200,58440021,470388557,26495649,59066453,40292255,543375441,33242290,6015852,317150447,22935775,232300346,476045917,90913482,249088920,67658976,614873,522722520,186766721,285517705,71683175,131444964,166501605,477920664,38154550,18738205,8861832,15594932,18536741,7595202,465378842,11838952,14848133,431696576,14358671,414520167,222578501,67058139,28976735,95601387,426582611,24874129,418762594,128157235,106030956,31352215,18733178,260196778,153179029,91842580,229494512,83414433,285579699,19957600,54295155,14929418,51516573,200076011,18758733,17776895,59397841,216802709,149834999,327507356,8200322,174345369,108636400,27504001,326877592,139919716,49949338,215035403,118421144,49410665,149550914,18446431,25662335,261725134,267634174,57737391,146506056,126964949,71055234,20870640,210196418,222806923,13290742,72247756,180410163,14784480,36684216,25611502,95614691,54629161,112967594,181656257,17994312,72918901,140082918,149087212,137272324,99534020,121755576,93964779,35848342,43059008,34704029,87672717,113137792,17863333,90407665,90591814,54297023,57924897,87551006,28300354,48990752,26188013],"previous_cursor":0,"next_cursor_str":"0"}