DZone
Database Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Database Zone > Paging through an ATOM feed using Node.js, CoffeeScript, MongoDB and Atom Hopper

Paging through an ATOM feed using Node.js, CoffeeScript, MongoDB and Atom Hopper

Chad Lung user avatar by
Chad Lung
·
Jun. 14, 12 · Database Zone · Interview
Like (0)
Save
Tweet
4.58K Views

Join the DZone community and get the full member experience.

Join For Free

I work daily with Atom Hopper which is a Java ATOMPub server. As we add new features we need to test those features and a lot of times we end up with small scripts just like the one I’ll show you today. Atom Hopper recently was updated to allow for paging through and ATOM feed. Basically, Atom Hopper will produce Next and Previous links in the feed head that can be followed easily to get to either the tail (oldest entries) of the feed or the head (newest entries).

 

See my previous articles or the wiki on how to get Atom Hopper setup as well as my article on configuring it to use the MongoDB data adapter. You can also use the Hibernate data adapter if you wish.

The feed head that Atom Hopper produces looks similar to this:

<feed xmlns="http://www.w3.org/2005/Atom">
  <link href="http://localhost:8080/namespace/feed/" rel="current" />
  <link href="http://localhost:8080/namespace/feed/" rel="self" />
  <id>urn:uuid:0b5061ec-94b1-4ffc-a383-803221793e2c</id>
  <title type="text">namespace/feed</title>
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:34ee3a9a-31dd-46e4-b39e-05bd3596966d&limit=25&search=&direction=forward" rel="previous" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&limit=25&search=&direction=backward" rel="next" />
  <link href="http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&limit=25&search=&direction=backward" rel="last" />
  <updated>2012-06-09T03:43:01.743Z</updated>
 
... Entries would be seen here ....
 
</feed>

You can see that there are several links. Since I’m at the head of the feed where the newest entries are, I can then follow the next link to move towards the tail of the feed. As I do this I could be parsing the ATOM entries contained within the returned data. As a sample I loaded 100 ATOM entries into MongoDB and I set my limit in the script to only load 25 entries per call (25 is the default and the maximum is 1000).

Using CoffeeScript I can easily follow the next links until I get to the tail. Once I reach the tail there are no more entries and therefor no next link will be produced at the tail of the feed.

You will need to install a couple NPM packages first:

$ npm install elementtree
$ npm install coffee-script

Here is the CoffeeScript:

fs = require("fs")
et = require("elementtree")
http = require("http")
url = require("url")
element = et.Element
 
opts =
    host: "localhost"
    port: 8080
    method: "GET"
    path: "/namespace/feed/?limit=25"
 
loadAtomFeed = ->
    http.get(opts, (response) ->
        returnedData = ""
        response.on "data", (chunk) ->
            returnedData += chunk
 
      response.on "end", ->
          etree = et.parse(returnedData)
          nextLink = etree.find("./link[@rel=\"next\"]")
          return if not nextLink?
 
          parsedUrl = url.parse(nextLink.attrib["href"])
          console.log "Following Next Link: " + parsedUrl.href
 
          opts.host = parsedUrl.hostname
          opts.path = parsedUrl.path
          loadAtomFeed()
    ).on "error", (e) ->
        console.log "Error: " + e.message
 
loadAtomFeed()

Note: I’m pretty sure this can be made much more efficient. As this is just a testing script it works “good enough”.

I’m using ElementTree to parse the XML. Although I’m not actually parsing out anything other than the next link you could definitely pull out anything you want from the feed.

I saved the script as app.coffee. Assuming you have CoffeeScript and Node.js installed you can run the script directly like this:

$ coffee app.coffee

My results looked like this:

Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:9b2e6989-0771-4622-b0eb-49d33f4f5487&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:1ed87690-632b-4bc9-a9f1-c7c607d771bb&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:82e12cc7-06d3-4b89-bedc-4f6bd4a3f4b6&limit=25&search=&direction=backward
Following Next Link: http://localhost:8080/namespace/feed/?marker=urn:uuid:69128671-daf3-4bac-adbb-64e459b6701e&limit=25&search=&direction=backward

 

Coffeescript Node.js MongoDB

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • What Is URL Rewriting? | Java Servlets
  • Role of Development Team in an Agile Environment
  • 10 Steps to Become an Outstanding Java Developer
  • Choosing Between REST and GraphQL

Comments

Database Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo