What is Streamdrill Good For?

a few weeks ago, we released the beta (oh, sorry, the ”β”) of stream drill . one question we heard quite often was, “well, what is it good for?” so i’ll try to elaborate a bit more on what it does.

first of all, streamdrill is not yet another general purpose big data framework. in its current version it doesn’t even have support for clustering (although that’s something we plan to add in the next months).

in a nutshell: streamdrill is useful for counting activities of event streams over different time windows and finding the most active ones. for the sake of simplicity, let’s call the above problem the top-k problem .

so let’s break this down a bit.

event streams are actually common place. some examples:

• really any kind of server logs
• user interactions in a social network
• logs of web accesses, page impressions, etc.
• monitoring applications of any kind

in general, such logs may contain all kinds of data and have varying level of structures, but often it is possible to identify groups of events that have similar structure. for example:

• for server logs: performance reports, errors, exceptions thrown, etc.
• for user interactions: user posting something, user sending messages, etc.
• for web logs: request of page consisting of path, referrer, ip address of the client, user-agent

such event streams usually contain an enormous amount of data, much too much for a single human being to grasp. as a first step, you’re often interested in doing some kind of aggregation, extracting some basic statistics. for example:

• for server logs: what are the average performances over the last hour on average for the whole cluster? which exceptions are thrown most often?
• for user interactions: which are the most active users? which media are trending (i.e. viewed/reshared most often)?
• for web logs: which web pages are viewed most often, from which locations? who are the most active referrers?

this is exactly what streamdrill does. for a given type of event, it counts activities and aggregates them over time windows. furthermore, it let’s you filter results so that you can drill down into your trends.

for streamdrill, an event consists of a fixed number of fields (which are called entities ). currently, they are all just plain strings, but this might change in the future.

for every event, you send an update command to streamdrill which then automatically updates the counts for the timescales configured. these timescales aggregate the counters using a technique called exponential decay counters, which is very memory effective. in addition, streamdrill keeps an ordered index of all entries, such that you can quickly query the most active entries, potentially filtering for some of the entities (for example, only show entires for a given path, user-agent, ip address, etc.)

in the next post, we’ll discuss why you would want to use streamdrill for that purpose instead of cooking something up of your own.