Paging through an ATOM feed using Node.js, CoffeeScript, MongoDB and Atom Hopper

I work daily with Atom Hopper which is a Java ATOMPub server. As we add new features we need to test those features and a lot of times we end up with small scripts just like the one I’ll show you today. Atom Hopper recently was updated to allow for paging through and ATOM feed. Basically, Atom Hopper will produce Next and Previous links in the feed head that can be followed easily to get to either the tail (oldest entries) of the feed or the head (newest entries).

See my previous articles or the wiki on how to get Atom Hopper setup as well as my article on configuring it to use the MongoDB data adapter. You can also use the Hibernate data adapter if you wish.

The feed head that Atom Hopper produces looks similar to this:

<feed xmlns="http://www.w3.org/2005/Atom">
  <link href="http://localhost:8080/namespace/feed/" rel="current" />
  <link href="http://localhost:8080/namespace/feed/" rel="self" />
  <id>urn:uuid:0b5061ec-94b1-4ffc-a383-803221793e2c</id>
  <title type="text">namespace/feed</title>
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:34ee3a9a-31dd-46e4-b39e-05bd3596966d&limit=25&search=&direction=forward" rel="previous" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&limit=25&search=&direction=backward" rel="next" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&limit=25&search=&direction=backward" rel="last" />
  <updated>2012-06-09T03:43:01.743Z</updated>
 
... Entries would be seen here ....
 
</feed>

You can see that there are several links. Since I’m at the head of the feed where the newest entries are, I can then follow the next link to move towards the tail of the feed. As I do this I could be parsing the ATOM entries contained within the returned data. As a sample I loaded 100 ATOM entries into MongoDB and I set my limit in the script to only load 25 entries per call (25 is the default and the maximum is 1000).

Using CoffeeScript I can easily follow the next links until I get to the tail. Once I reach the tail there are no more entries and therefor no next link will be produced at the tail of the feed.

You will need to install a couple NPM packages first:

$ npm install elementtree
$ npm install coffee-script

Here is the CoffeeScript:

fs = require("fs")
et = require("elementtree")
http = require("http")
url = require("url")

element = et.Element
 
opts =
    host: "localhost"
    port: 8080
    method: "GET"
    path: "/namespace/feed/?limit=25"
 
loadAtomFeed = ->
    http.get(opts, (response) ->
        returnedData = ""
        response.on "data", (chunk) ->
            returnedData += chunk
 
      response.on "end", ->
          etree = et.parse(returnedData)
          nextLink = etree.find("./link[@rel=\"next\"]")
          return if not nextLink?
 
          parsedUrl = url.parse(nextLink.attrib["href"])
          console.log "Following Next Link: " + parsedUrl.href
 
          opts.host = parsedUrl.hostname
          opts.path = parsedUrl.path
          loadAtomFeed()
    ).on "error", (e) ->
        console.log "Error: " + e.message
 
loadAtomFeed()

Note: I’m pretty sure this can be made much more efficient. As this is just a testing script it works “good enough”.

I’m using ElementTree to parse the XML. Although I’m not actually parsing out anything other than the next link you could definitely pull out anything you want from the feed.

I saved the script as app.coffee. Assuming you have CoffeeScript and Node.js installed you can run the script directly like this:

$ coffee app.coffee

My results looked like this:

Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:1ed87690-632b-4bc9-a9f1-c7c607d771bb&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:82e12cc7-06d3-4b89-bedc-4f6bd4a3f4b6&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&limit=25&search=&direction=backward