Big Data is Everything and Nothing (Depending on Who You Ask)
It seems big data means something different to everyone. In the great debate/hype about big data, there’s no lack of opinion on the topic and it seems to mostly depend on an individual’s product, skill set and business challenges. This ambiguity shares a great deal of the blame for why the term is often polarizing and why there’s a fair amount of cynicism in corners of the marketplace. Just for fun, let’s take a look at some of the points of contention.
- Big data isn’t anything new – This is a very legitimate argument for why big data doesn’t deserve quite so much hype. You’ll hear this argument mostly from the companies that have been solving problems and earning a living with vast amounts of data for decades. There are exceptional examples of this like Nielsen, the company that started off rating the advertising value of media and morphed into consumer preference and pattern juggernaut.
- Big data is really about small data – Also a legitimate argument against some of the hype. Companies that crunch data sets, small or large, often find that the pattern exists in just one variable, like the way preferences for wine often come down to our tolerance for acidity. Some of what’s called big data isn’t big when the results come in, but it often takes large data sets to prove that a small amount of data matters…a big data paradox.
- Big data is about the right algorithm, not more data - Like the other two points, this is also mostly true. The shows up in the crowdsourced contest Netflix used to improve on the company’s Cinematch predictive powers, which became about tiny tweeks to algorithms to raise results by .01%. There was no human X factor that solved the problem. This argument pits the traditional quantitative analysts against the new breed of data scientists. You could say it is also the fight between math and science and causation and correlation. This is a fascinating debate and I suspect both sides are right in differing circumstances.
- The 3 V’s (volume, velocity and variety) aren’t enough – Coming up with a new V for the description of big data is now the object of derision. “How many V’s do you have?” comes up often as an easy way to understand someone’s perspective on the topic but has also reached the point of silliness. Gartner’s Doug Laney came up with the 3 V’s back in 2001 and the debate has raged ever since around value,
- Big data is creepy – This one really depends on the definition of creepy. People with Rain Man-like capabilities have always been able to mentally process exceptional amounts of data and that ability could be used to cheat, manipulate and get ahead. Just because we’re able to see more complex patterns in ever more data doesn’t make big data itself creepy. Its use, just like before computerization, is what can be creepy.
At the end of the day, big data is going to continue to be a topic of intense debate because so much of what we do is affected by someone’s ability to gather, analyze and then predict who we are from our patterns, even the non-transactional ones like social media use. Enterprises can’t afford to ignore the technology that their competitors are using to better understand customers and be more efficient in their operations.