DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Integration
  4. Access the Twitter REST API (v1.1) from Scala and Java using signpost

Access the Twitter REST API (v1.1) from Scala and Java using signpost

Jos Dirksen user avatar by
Jos Dirksen
·
Sep. 23, 12 · Interview
Like (1)
Save
Tweet
Share
8.96K Views

Join the DZone community and get the full member experience.

Join For Free

If you've read some other articles on this blog you might know that I like creating visualizations of various datasets. I've just started a small project where I want to visualize some data from Twitter. For this I want to retrieve information about followers and profile information directly from twitter. I actually started looking for a set of all twitter accounts, but could only find one that was two years old. So, only option left, directly access the twitter API and get the data myself.

There are a couple of open source libraries out that we can use directly from Scala (or java) but as far as I could see they use the old v1 API and not the v1.1 API. The old API has a very strict data rate limit, which is a bit lighter in the new API. And besides that I'm more interested in the raw data and parsing the returning JSON isn't that hard with Scala (or Java for that matter).

Register an application at twitter

First thing to do, and easiest way to get started is registing a new application for your twitter account. Go to https://dev.twitter.com/apps/new and create a new application. Don't worry about the urls, since we won't be using the OAuth callback mechanism:

Create 1 | Twitter Developers.png

Depending on what you want to do with the API you need to give additional permissions to this app. Default is "read-only", if you want to allow your new application to post or access your direct messages you need to update the permissions. This is done from the settings page of your application:

SmartjavaTest | Twitter Developers.png

Once you've created the application and setup the correct permissions you can generate an access token. Doing this will avoid having to go through the complete OAuth dance. You do this by going to your new app details and at the bottom select the "create my access token" option.

Create 2 | Twitter Developers.png

Now you'll have a set of tokens (see the details part of your applications):

SmartjavaTest | Twitter Developers-1.png

We'll use these tokens to authenticate the requests we'll make to twitter.

Using a OAuth library

The OAuth protocol is a pretty good documented protocol, but implementing it yourself is a lot of work and very error-prone. Luckily there are many OAuth libraries that can help you. I've tried a couple and the one that's most easy to use (at least for me) was signpost. The examples below show how to do this from Scala, but you can follow the same approach for Java.

First the dependencies. I've used sbt and from signpost I use the client with support for HTTP commons. In sbt add the following:

..
libraryDependencies ++= Seq(
	"oauth.signpost" % "signpost-core" % "1.2",
	"oauth.signpost" % "signpost-commonshttp4" % "1.2", 
	"org.apache.httpcomponents" % "httpclient" % "4.2",
         ...
)

For maven you can use the same libraries. Next we can write a simple test to see if everything is working. In java it looks like this:

import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
 
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
 
 
public class Tw {
 
	  static String AccessToken = "access token for your app";
	  static String AccessSecret = "access secret for your app";
	  static String ConsumerKey = "consumer key for your app";
	  static String ConsumerSecret = "consumer secret for your app";
 
	/**
	 * @param args
	 */
	public static void main(String[] args) throws Exception {
		OAuthConsumer consumer = new CommonsHttpOAuthConsumer(
                ConsumerKey,
                ConsumerSecret);
 
        consumer.setTokenWithSecret(AccessToken, AccessSecret);
        HttpGet request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
        consumer.sign(request);
 
        HttpClient client = new DefaultHttpClient();
        HttpResponse response = client.execute(request);
 
        int statusCode = response.getStatusLine().getStatusCode();
        System.out.println(statusCode + ":" + response.getStatusLine().getReasonPhrase());
        System.out.println(IOUtils.toString(response.getEntity().getContent()));
	}
}

And in Scala it looks pretty much the same:

import org.apache.http.client.HttpClient
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpGet
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer
import org.apache.commons.io.IOUtils
 
 
object TwitterPull {
 
	  val AccessToken = "access token for your app";
	  val AccessSecret = "access secret for your app";
	  val ConsumerKey = "consumer key for your app";
	  val ConsumerSecret = "consumer secret for your app";
 
 
  def main(args: Array[String]) {
 
	 val consumer = new CommonsHttpOAuthConsumer(ConsumerKey,ConsumerSecret);
	 consumer.setTokenWithSecret(AccessToken, AccessSecret);
 
     val request = new HttpGet("http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen");
     consumer.sign(request);
     val client = new DefaultHttpClient();
     val response = client.execute(request);
 
     println(response.getStatusLine().getStatusCode());
     println(IOUtils.toString(response.getEntity().getContent()));
  }
}

When you run this the output will look something like this:

200
{"previous_cursor_str":"0","next_cursor":0,"ids":[48342167,21011010,824959303,97242821,16953163,218083367,20869799,5234221,13604142,804783128,271050984,405121284,26470609,50201837,1723451,374494377,120867838,14311946,253114713,39554511,7375412,42507395,112806109,92787154,218238023,110443797,76922155,198798790,294104985,305625416,217698029,21803482,14927822,15453445,15715866,15657036,186956616,36028164,70380613,326158542,573546312,14401332,521488579,9108612,576970378,293236313,16398366,16220300,15234937,32000283,439444353,14300622,67204409,155850135,14198255,32264673,15852981,313248158,20123099,608942046,234930032,36896958,18466675,45496942,330899833,18980755,88253383,461023805,31175627,11044952,142780445,63175189,107991607,94830953,600993241,6195002,115391430,550080945,381418927,168603682,142388604,8258462,218411138,30450578,77728346,2521381,182867524,494119147,29426983,572417260,94344849,325413275,389354525,501438275,164346498,22730282,8293302,21085554,341645357,56978853,180507788,10074002,22536424,14247654,581293627,15259428,483317230,462826270,4774641,15366832,96850673,278486993,22273826,17716679,14566626,158473088,20461042,161242434,43756629,40163100,141165981,5325152,7620782,266749648,524476136,557713614,39602637,18843154,1623,565954426,39639621,166672305,18683074,233118689,44876099,235258223,219310062,10699922,12660502,218030046,91552210,19361980,206645598,35346200,58440021,470388557,26495649,59066453,40292255,543375441,33242290,6015852,317150447,22935775,232300346,476045917,90913482,249088920,67658976,614873,522722520,186766721,285517705,71683175,131444964,166501605,477920664,38154550,18738205,8861832,15594932,18536741,7595202,465378842,11838952,14848133,431696576,14358671,414520167,222578501,67058139,28976735,95601387,426582611,24874129,418762594,128157235,106030956,31352215,18733178,260196778,153179029,91842580,229494512,83414433,285579699,19957600,54295155,14929418,51516573,200076011,18758733,17776895,59397841,216802709,149834999,327507356,8200322,174345369,108636400,27504001,326877592,139919716,49949338,215035403,118421144,49410665,149550914,18446431,25662335,261725134,267634174,57737391,146506056,126964949,71055234,20870640,210196418,222806923,13290742,72247756,180410163,14784480,36684216,25611502,95614691,54629161,112967594,181656257,17994312,72918901,140082918,149087212,137272324,99534020,121755576,93964779,35848342,43059008,34704029,87672717,113137792,17863333,90407665,90591814,54297023,57924897,87551006,28300354,48990752,26188013],"previous_cursor":0,"next_cursor_str":"0"}

If you get a 403 check whether the tokens match.
API twitter REST Web Protocols Scala (programming language) Java (programming language) application

Published at DZone with permission of Jos Dirksen, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Unleashing the Power of JavaScript Modules: A Beginner’s Guide
  • Core Machine Learning Metrics
  • A Beginner's Guide to Back-End Development
  • Promises, Thenables, and Lazy-Evaluation: What, Why, How

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: