Website Scraping Using Selenium, Docker, and Chrome With Extensions

for a specific project, we needed a quick way to get the content of a specific url and check whether a word was present in the text there. if all we had to scrape were static websites, this wouldn't be that difficult. we would just get the sources, parse them with jsoup, and extract the readable content. however, a large percentage of the target sites are single page apps or angular applications, which only show the content after some javascript processing. so, we started looking at an alternative way to do this.

first try, phantomjs

first, we looked at whether we could simply use phantomjs to get the website and return the content. with phantomjs, this is really simple, and the whole code looks something like this:

var args = system.args;

 var webpage = require('webpage');
 var page = webpage.create();

 page.open(args[1], function (status) {
 var content = page.plaintext;
 console.log('content: ' + content);
 phantom.exit();
 });

and, we can use this minimal script like this:

$ phantomjs test.js https://news.ycombinator.com/
content:
hacker news
new | comments | show | ask | jobs | submit
login
1.
outlook 2016’s new pop3 bug deletes your emails (josh.com)
40 points by luu 2 hours ago | 3 comments
2.
52-hertz whale (wikipedia.org)
148 points by iso-8859-1 5 hours ago | 46 comments
3.
what my phd was like (jxyzabc.blogspot.com)
30 points by ingve 2 hours ago | 6 comments
...

well, that was a couple of minutes work, and with a quick minimal api surrounding it, it perfectly matched our requirements and would be easily embeddable. we can just use phantomjs to render the page, handle the javascript and no more worries. but, unfortunately, after testing a couple of sites, we ran into an issue with a number of european sites. the problem was this: the eu cookie law ( http://ec.europa.eu/ipg/basics/legal/cookies/index_en.htm ). this guideline pretty much states this:

what is the cookie law.

"the eprivacy directive – more specifically article 5(3) – requires prior informed consent for storage of or access to information stored on a user's terminal equipment. in other words, you must ask users if they agree to most cookies and similar technologies (e.g. web beacons, flash cookies, etc.) before the site starts to use them."

or, in other words, before you can use the site, you have to consent that cookies may be used. most sites implement this through the use of a modal pop-up, or an overlay at either the top or the bottom. depending on the country you're in, this law is more or less enforced (see https://cookiepedia.co.uk/cookie-laws-across-europe for country specific details). in the netherlands, where i live and work—and most of our customers are—it's been pretty much mandated by the government for websites to strictly follow this rule.

so, we ran into the problem that instead of getting the real content of the site, we'd sometimes get only the cookie message, or our content contained information that wasn't relevant. for example, see the following site, which provides such a pop-up:

while this was probably created with the best intentions in daily use, it is very annoying, and when trying to automate and scrape stuff, it makes stuff unnecessarily complex.

luckily though, there is a way we can circumvent most of these popups. by installing the "i don't care about cookies ( http://www.kiboke-studio.hr/i-dont-care-about-cookies/ )" extension in your browser, most of these popups and consent forms will just be ignored. so, what we'd want to accomplish is running this extension so that we can scrape websites without having to worry about invalid content based on this cookie law. running chrome extensions in phantomjs, however, isn't possible, so we need an alternative approach.

selenium and chrome.

basically, what we want is, instead of scripting phantomjs, we want to script chrome with this extension installed. actually doing this is surprisingly easy. the first thing we need is to be able to run and control chrome. for this, we can use chromedriver ( https://sites.google.com/a/chromium.org/chromedriver/ ) which is an implementation of the webdriver specification ( https://w3c.github.io/webdriver/webdriver-spec.html ) which allows remote control of a specific chrome instance. when we've got chrome running with webdriver enabled, we can connect using one of the webdriver client libraries ( http://www.seleniumhq.org/download/ ) and start scripting our chrome browser. to start a chrome enabled webdriver (especially on a headless server, which is where we'd be running this) is a bit cumbersome, so we'll just use a ready to use docker image instead.

$ docker run -d -v /dev/shm:/dev/shm -p 4444:4444 selenium/standalone-chrome
cbca5a2daf290c72f5771bbc1655e61a7700053ff1e2b95ece5301286692885e
➜ tmp docker ps
container id image command created status ports names
cbca5a2daf29 selenium/standalone-chrome "/opt/bin/entry_point" 4 seconds ago up 1 seconds 0.0.0.0:4444->4444/tcp high_ritchie

this will start up a single node with selenium and chrome configured. we can see this is working by opening up a browser and pointing it to the hub url on the exposed port. note the /dev/shm:/dev/shm part, this is needed to circumvent a bug when running chrome in docker ( https://github.com/elgalu/docker-selenium/issues/20#issuecomment-133011186 ).

the hub url on my local machine is: http://192.168.99.100:4444/wd/hub:

not that exciting right? let's run a simple script to see if everything is working. you can use selenium with a number of different tools and languages, for this article i'll use scala, but libraries are available for a lot of different languages. let's look at the code you can use to get a page, and output all it's text content to the console:

import java.net.url

import org.openqa.selenium.webdriver
import org.openqa.selenium.chrome.{chromeoptions}
import org.openqa.selenium.remote.{desiredcapabilities, remotewebdriver}
import org.scalatest.selenium.webbrowser

object seleniumscraper extends app with webbrowser {

 val capability = desiredcapabilities.chrome()
 implicit val webdriver: webdriver = new remotewebdriver(new url("http://192.168.99.100:4444/wd/hub"), capability)

 try {
 go to ("http://bbc.co.uk/")
 find(tagname("body")) match {
 case some(el) => println(el.text)
 case none => println("no body element found")
 }
 } finally {
 close()
 quit()
 }
}

when we run this, we use the selenium chrome instance running in our docker container, to fetch the page. the first lines of the output look something like this:

cookies on the bbc website
the bbc has updated its cookie policy. we use cookies to ensure that we give you the best experience on our website. this includes cookies from third party social media websites if you visit a page which contains embedded content from social media. such third party cookies may track your use of the bbc website. we and our partners also use cookies to ensure we show you advertising that is relevant to you. if you continue without changing your settings, we'll assume that you are happy to receive all cookies on the bbc website. however, you can change your cookie settings at any time.
continue
change settings
find out more
accessibility links
accessibility help
sign in
bbc navigation
news
sport
weather
shop
earth
travel
more
search the bbc
saturday, 27 february
hillary clinton secures a big win over bernie sanders in the south carolina primary, the latest battleground for the democratic presidential nomination.
...

while this is already pretty good, we can see the cookie law in action here. while on the bbc site it doesn't hide any content or prevent you from continuing, it still shows us a pop-up and could give us false positives when we want to search for words occuring on a specific page.

adding extensions to chrome using selenium

i've mentioned the i-don't-care-about-cookies plugins earlier. if we could install this plugin in our automated browser, we could use the plugin's filtering to remove and ignore these cookie messages. to install a plugin automatically in chrome we could follow the procedure from here: https://developer.chrome.com/extensions/external_extensions#preferences . but that would mean creating a new docker image with the correct settings. an easier approach, however, is offered by selenium. first, though, we need the plugin as a crx archive. for this, you can just use http://chrome-extension-downloader.com/ to download the plugin as a file from the filestore. once you have this file, just add the following to your selenium script and the plugin will be automatically loaded whenever your script starts.

val capability = desiredcapabilities.chrome()
val options = new chromeoptions();
options.addextensions(new file("/users/jos/downloads/idontcareaboutcookies-2.5.3.crx"))
capability.setcapability(chromeoptions.capability, options)

now, when we run this same simple script again, we see the following:

accessibility links
accessibility help
sign in
bbc navigation
news
sport
weather
shop
earth
travel
more
search the bbc
saturday, 27 february
huge win for clinton in south carolina
hillary clinton secures a big win over bernie sanders in the south carolina primary, the latest battleground for the democratic presidential nomination.

as you can see, this time, the specific cookie message is ignored.

so with just a couple of simple steps, we can scrape websites, and make sure we only get the core content by using an extension. we can of course also use this to run other plugins such as adblockers.