Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Scraping Images and Files Using Casper.JS

DZone's Guide to

Scraping Images and Files Using Casper.JS

A few weeks ago I showed you how you could use CasperJS to scrape content from a website — let's bring it to the next level with image and file scraping

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

CasperJS allows you to do much more than just scrape text content from a website. Hopefully you've already read through part one of this series, where we do some simple scraping. With just a few lines of code you can go ahead and get image or other binary files downloaded to your machine as you parse through a site.  

Downloading

Everything revolves around the download() API call. At it's simplest, you can download a copy of any HTML page: 

casper.open("http://techmeme.com/");

casper.then(function() {

 this.download("http://techmeme.com", 'localcopy.html');

});

The function accepts a URL and a local target, and also allows you to specify the HTTP method and request parameters.

For images it's no different:

 this.download("http://techmeme.com/img/techmeme135.png", 'techmeme_logo.png');

Note that if you are having unexpected issues with your downloads, you may need to switch off web security. 

A Note on the FileSystem API

It's easy to assume that you've got access to all the NodeJS APIs when running a CasperJS script, but you don't. For the filesystem, you just get PhantomJS's FileSystem API when you use require('fs'). It's sufficient for most operations, but you won't be able to do some of the more advanced stuff that NodeJS provide.

If this limitation provides too much of a roadblock, you can always save results from your CasperJS run in CSV/JSON file, and then load those results up in a separate NodeJS script, where you can use any module you want! 

I had a task where I needed to download a number of files, and needed to determine their URL. To do this, I used CasperJS to determine the URL, and then used a separate NodeJS script that utilized the request module to download the files in sequence (along with some promises written in Q to avoid too many files be downloaded at the same time).

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

Topics:
casperjs ,javascript

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}