The Story of SpiderDuck, Twitter's Real-Time URL Fetcher

The process that Twitter's application engineers went through to solve the problem they were having with managing all of their incoming links is an interesting test case in building a tool with available open source technologies to meet a unique need.  The end product was SpiderDuck:

Prior to SpiderDuck, Twitter had a service that resolved all URLs shared in Tweets by issuing HEAD requests and following redirects. While this service was simple and met the needs of the company at the time, it had a few limitations:
    •    It resolved the URLs but did not actually download the content. The resolution information was stored in an in-memory cache but not persisted durably to disk.  This    meant that if the in-memory cache instance was restarted, data would be lost.
    •    It did not implement politeness rules typical of modern bots, for example, rate limiting and following robots.txt directives.

Twitter isn't the only large internet company that uses URL fetchers and URL crawlers, but the fact that Twitter's services needed to work in real-time presented the unique challenge that SpiderDuck was meant to address.  The SpiderDuck architecture of six main components:

Kestrel: This is message queuing system widely used at Twitter for queuing incoming Tweets.

Schedulers: These jobs determine whether to fetch a URL, schedule the fetch, follow redirect hops if any. After the fetch, they parse the downloaded content, extract metadata, and write the metadata to the Metadata Store and the raw content to the Content Store. Each scheduler performs its work independently of the others; that is, any number of schedulers can be added to horizontally scale the system as Tweet and URL volume grows.

Fetchers: These are Thrift servers that maintain short-term fetch queues of URLs, issue the actual HTTP fetch requests and implement rate limiting and robots.txt processing. Like the Schedulers, Fetchers scale horizontally with fetch rate.

Memcached: This is a distributed cache used by the fetchers to temporarily store robots.txt files.

Metadata Store: This is a Cassandra-based distributed hash table that stores page metadata and resolution information keyed by URL, as well as fetch status for every URL recently encountered by the system. This store serves clients across Twitter that need real-time access to URL metadata.

Content Store: This is an HDFS cluster for archiving downloaded content and all fetch information.

To learn more about how these components of the SpiderDuck architecture work (with nice diagrams!) and how Twitter uses SpiderDuck, check out the full story at the Twitter Engineering blog.