Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Common Crawl now on AWS

DZone's Guide to

Common Crawl now on AWS

· Cloud Zone
Free Resource

What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDB. Sign up and you'll receive instructions for how to get started!

The Common Crawl Corpus, a repository of valuable web crawl metadata composed of 5 billion web pages, is now available on AWS Public Data sets and is available for free on the Amazon Simple Storage Service (S3) in the ARC (.arc) file format.

This is great news because it means that the Common Crawl data corpus is now much more readily accessible and visible to the public. The greater accessibility and visibility is a significant help in our mission of enabling a new wave of innovation, education, and research.

-- Common Crawl

Sematext Founder, Otis Gospodnetić, is certainly excited about the announcement:

@otisg 5 Billion(!) Web pages from the CommonCrawl dataset are now on AWS! http://ow.ly/8DXa7 #dataset #webcrawl


The ARC file format, developed by System Enhancement Associates (SEA), allows for the data to be compressed and uncompressed without losing any data.

Common Crawl provides the glue code required to launch Hadoop jobs on Amazon Elastic MapReduce that can run against the crawl corpus residing here in the Amazon Public Data Sets. By utilizing Amazon Elastic MapReduce to access the S3 resident data, end users can bypass costly network transfer costs.

-- AWS

Common Crawl provides its Hadoop classes in its GitHub repository and a tutorial on how to analyze Common Crawl's dataset using Amazon Elastic MapReduce is available on the Common Crawl blog.

What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDBSign up and you'll receive instructions for how to get started!

Topics:

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}