Performance Monitoring Will Need to Join the Crowd

[This article was written by Laura Strassman] 

A number of applications or apps have been developed that rely on crowd-sourcing to work. It’s a change in architecture that is a natural progression of the current state of the art of software development and delivery; by component architecture on distributed infrastructure serving huge numbers of users. In any case, these apps must be high performing and provide a great user experience in order to grow and retain their users, otherwise their crowd sourced information will be just a few people hanging out, not a crowd. I use one of these applications almost daily, Waze, and I rely on it to tell me how to get around an ever changing commute.

I assume that the providers of these applications want to make money, possibly with advertising like Waze has started to do… so the performance of the applications from that perspective is equally important. They may need to provide SLA’s to advertisers or sell services to other software providers.

Here’s the rub–how do they monitor the performance of this application that is largely out of their control?

You can monitor applications today in several ways: with synthetic tests which replicate what a user would do on a website or application and see if that path is responsive. It works by driving a browser from a network location and duplicating actions a user would do, search, type, navigate. You can use a server agent to collect data on real users if my activities go through a server- you passively gather all the measurements when users travel the application delivery chain from the browser to the application server. But how would I manage the peer to peer/ crowd-sourcing performance? Crowd-sourced applications come from the endpoints of the internet – mostly phones- travelling over a variety of networks, cell, Wi-Fi, carriers and the patterns change all the time.

If you want to accurately monitor performance of web applications you need to follow the same delivery chain from the actual browsers that people use, if you want to monitor crowd-sourcing, I’m thinking you have to monitor from the crowd.

I came up with the question, but not being an engineer I didn’t know the answer, so I went hunting. What I found is that there is a small move to introduce crowd-sourced monitoring. There are some academic papers around putting monitoring on smartphones and then collating the data to tell you what’s going on. There is some product movement around monitoring PC’s and using the aggregate data to understand how SaaS business applications are working. Also some theories and tests and papers published around crowd-sourced monitoring being used to actually monitor crowds, or the internet.

Essentially, crowd-sourced monitoring operates on the exact same notion as obtaining the data in the first place–you have to put the monitoring in the same place as the users and then aggregate the data around the performance the same way you are currently aggregating the data about what the crowd tells you. So you monitor actual users, but the key information you want is about the performance of the data feed/sharing and network. It seems to be the inverse of server agent monitoring.

Pretty clearly you would use crowd-sourcing to monitor crowd-sourcing. The question is when does it become a business imperative to do that? Once that is established then commercial solutions will be available for now the fastest way to know what’s happening is probably twitter.

In my travels I read the following Papers and visited the following Websites:

“CrowdWatch: Enabling In-Network Crowd-sourcing Robin” Kravets, Hilfi Alkaff, Andrew Campbell, Karrie Karahalios, Klara Nahrstedt University of Illinois and Dartmouth College

“Crowdsourcing Service-Level Network Event Monitoring” David R. Choffnes, Fabian E. Bustamante, Sihui Ge, Northwestern University, At&T Labs – Research

Exoprise.com – a company that does performance monitoring for SaaS apps using crowd-sourced data.