This article gives an introduction to working with XML / HTTP APIs from Groovy in the context of a real world scenario using the GoogleCode API.

This article first appeared in the March 2013 issue of GroovyMag. Since the script was originally written Google deprecated the Issue Tracker API and scheduled it for closure on the 14th June 2013. So whilst the script is now for interest only, the principles are still valid for other purposes.

A short while back I had a requirement to rename a Google Code project – it was due to a typo in the project name rather than a ‘cease & desist’ notice. Once my colleague had assigned me owner-level permissions I discovered that even the advanced administration options don’t permit you to change the project name. However, being Google, the issues have an API and being a dab hand with Groovy it was short work to migrate the issues to a new project (sans-typo).

This was early on in the project lifecycle, so the main challenge wasn’t migrating the code (there are several blog posts with instructions on that depending on your flavour of source code control) – rather it was the multitude of issues and comments from a requirements gathering workshop.

The final script is available from GitHub as a gist:https://gist.github.com/rbramley/5073413

API & toolset

The Issue Tracker API is documented athttp://code.google.com/p/support/wiki/IssueTrackerAPI

It is a RESTful API using Atom feeds/entries – therefore the tools we’ll use are:

• Apache HttpComponents HttpClient 4.x (replaced commons-httpclient 3.x)
• XmlSlurper – for parsing Atom Feeds that we GET
• MarkupBuilder – for creating Atom entries to POST

DEPENDENCIES

HttpClient and its dependencies can be obtained using the Groovy GRAPE @Grabannotations shown in Listing 1. GRAPE (GRoovy Advanced Packaging Engine –http://groovy.codehaus.org/Grape) uses Apache Ivy for dependency resolution and was introduced in Groovy 1.6.

@Grab(group='commons-logging', module='commons-logging', version='1.1.1')
@Grab(group='commons-codec', module='commons-codec', version='1.4')
@Grab(group='org.apache.httpcomponents', module='httpclient', version='4.1.2')

Listing 1: Grab annotations

How does the script work?

To better understand the context of the examples it is useful to quickly recap what the script does – the process is as follows:

1. Form POST credentials
2. If not 403 forbidden, extract auth token (for use in Authorization header on all subsequent requests)
3. GET the issues list from the source project
4. Parse the response body Atom XML (declaring the issues namespace ofhttp://schemas.google.com/projecthosting/issues/2009)
5. For each entry in the feed:
   1. Extract the issue ID
   2. Convert the entry to the required form
   3. POST the Atom entry to the target project issue creation URL
   4. If not 400 bad request, GET the list of comments by issue ID from the source project
   5. Parse the response body Atom XML (declaring the issues namespace as before)
   6. For each entry in the feed:
      1. Convert the entry to the required form
      2. POST the Atom entry to the target project comment creation URL for the current issue

HTTP interaction basics

To work with the API requires the use of two verbs, GET and POST. The examples rely on the definition of HttpClient httpclient = new DefaultHttpClient()

GET

get = new HttpGet(issuesCommentsListUrl)
get.setHeader('Authorization', "GoogleLogin auth=${authToken}")
response = httpclient.execute(get)

println "${response.getStatusLine().getStatusCode()} - ${response.getStatusLine().getReasonPhrase()}"
commentsAtom = EntityUtils.toString(response.entity)

Listing 2: HttpClient GET method

The commentary for Listing 2 falls into two parts: making the request and handling the response. 
For the request we have to construct an HttpGet object with the target URL, add the authorization token to the header and then instruct the HttpClient to execute our request.
In terms of handling the response, we require it as a String for XML parsing, so the entity has to be obtained and converted to a String.

Whilst this is fairly straightforward, it doesn’t beat the simplicity of the Groovy augmented getText method on the java.net.URL class (e.g. def response = new URL(url).getText()) – in this case HttpClient is used for consistency and for the ability to set headers.

LOGIN FORM POST

The first POST operation requires a form post to login, and as shown in Listing 3 the form parameters are constructed using NameValuePair and added to a list. This list is used to construct the URL encoded form entity set on the HttpPostobject.

// set up login parameters
NameValuePair accountType = new BasicNameValuePair('accountType', 'GOOGLE')
NameValuePair email = new BasicNameValuePair('Email', emailAddress)
NameValuePair passwd = new BasicNameValuePair('Passwd', password)
NameValuePair service = new BasicNameValuePair('service', 'code')
NameValuePair source = new BasicNameValuePair('source', sourceScript)

List params = new ArrayList(5)
params.addAll()

HttpPost post = new HttpPost(googleLoginUrl)
post.setEntity(new UrlEncodedFormEntity(params))

HttpResponse response = httpclient.execute(post)

Listing 3: Form POST

If the response status code isn’t a 403 Forbidden, the resulting authorization token is extracted from the body of the response.

POSTING XML

As can be seen in Listing 4, XML payload POST operations are simpler as we are sending the XML using a StringEntity. In this case we also have to set theContent-type header to application/atom+xml.

// post the issue
post = new HttpPost(issuePostUrl)
post.setHeader('Content-type', 'application/atom+xml')
post.setHeader('Authorization', "GoogleLogin auth=${authToken}")
post.setEntity(new StringEntity(issueCreationXml))
response = httpclient.execute(post)

Listing 4: String body POST

Groovy XML processing

Groovy has some very useful helper classes for working with XML – the following sections will cover XmlSlurper and MarkupBuilder as used by the script. GPath, a powerful set of functions for querying nested data structures such as an XML document, is beyond the scope of this article but is worth further investigation if you need to perform more advanced XML handling than is required for the mapping exercise in this script.

CONSUMPTION WITH XMLSLURPER

The Groovy XML Slurper ([a href="http://groovy.codehaus.org/Reading+XML+using+Groovy%E2%80%99s+XmlSlurper" rel="nofollow" style="font-style: inherit; color: rgb(51, 51, 51);"]http://groovy.codehaus.org/Reading+XML+using+Groovy’s+XmlSlurper) can parse XML to an object tree and is namespace aware. Listing 5 shows an overview of the important fields in the Atom feed that are used in Listings 6 and 7. As this script was designed for issue migration, we parse the XML into nested objects (Listing 6), which are used to generate XML for recreating the issue in the target project (Listing 7).

<feed xmlns='http://www.w3.org/2005/Atom' ...>
<entry>
 <id>
 <title>
 <content type='html'>
 <author>
 <name>
 </author>
 <issues:id>
 <issues:label>Type-Defect</issues:label>
 <issues:label>Priority-Medium</issues:label>
 <issues:owner>
 <issues:username>
 </issues:owner>
 <issues:state>
 <issues:status>
</entry>

Listing 5: Atom entry fields

// Parse and process the atom feed
feed = new XmlSlurper().parseText(atom).declareNamespace([issues:issuesXmlns])
feed.entry.each { entry ->
 issueId = entry.'issues:id'
 println "Issue ${issueId} - ${entry.title}"

 issueCreationXml = buildIssue(entry, issuesXmlns, atomXmlns)

Listing 6: XmlSlurper

PRODUCTION WITH MARKUPBUILDER

MarkupBuilder is a helper class for creating XML or HTML markup using the closure-based builder syntax. Unlike XmlSlurper groovy.xml.MarkupBuilderisn’t on the standard Groovy classpath so will need to be imported. Listing 7 shows the creation of an Atom entry with the Google Data issues API extensions.

For fine-grained control over specific elements you can use theMarkupBuilderHelper class (http://groovy.codehaus.org/api/groovy/xml/MarkupBuilderHelper.html) that is accessed as mkp and enables direct insertion of data escaped or unescaped e.g. b { mkp.yield( '3 < 5' ) } which results in the output 3 < 5.

/** Build an atom feed for an issue */
def buildIssue(entry, issuesXmlns, atomXmlns) {
 def writer = new StringWriter()
 def xml = new MarkupBuilder(writer)
 
 xml.'atom:entry'('xmlns:atom':atomXmlns,'xmlns:issues':issuesXmlns) {
 'atom:title'(entry.title)
 'atom:content'(type:'html', entry.content)
 'atom:author' {
 'atom:name'(entry.author.name)
 }
 entry.'issues:label'.each {
 'issues:label'(it)
 }
 'issues:owner' {
 'issues:username'(entry.'issues:owner'.'issues:username')
 }
 'issues:state'(entry.'issues:state')
 'issues:status'(entry.'issues:status')
 }
 
 return writer.toString()
}

Listing 7: MarkupBuilder

GROOVY USER GUIDE REFERENCES

http://groovy.codehaus.org/GPath

http://groovy.codehaus.org/Processing+XML

http://groovy.codehaus.org/Reading+XML+using+Groovy%27s+XmlSlurper

http://groovy.codehaus.org/Creating+XML+using+Groovy%27s+MarkupBuilder

Executing the script

If you have the need to migrate your issues then the necessary steps are:

1. Create your new target project
2. Add the members from the old project (I did this manually) – this is critical to prevent creation errors
3. Grab the IssueMigrator.groovy script from GitHub gist
4. Change the project names, credentials and script source (lines 32-36)
5. If you have more than 50 issues you’ll need to increase the max-results parameter on line 50
6. Understand that the issues/comments will be created as the account running the script (and it has no warranty) – the original owners will be retained
7. Run the script (either cross your fingers or run against a guinea pig project first!)
8. Sit back and enjoy your new project

You may see some 400 ‘bad request’ responses being output from issue comments where it thinks there have been no updates. This can be ignored as the issue was created in the end state, rather than being created in the start state and then rolled forwards.