New Data, Not Just Big Data

The Insight 2015 conference highlighted some impressive applications of big data: predicting the path of hurricanes more accurately (as we saw with hurricane Patricia), improving the performance of athletes, making cars safer, etc.

These applications involve large amounts of data. But more importantly they involve new data, not simply greater quantities of data we’ve had before. Cheap sensors make it possible to measure things more directly and in higher resolution than before. We have sources of data, such as social media, that are qualitatively different from what we’ve had in the past.

Simply saying we have more data than before obscures what’s happening. For example, we don’t know more about consumer behavior than a generation ago because we do more phone surveys and have more customer satisfaction post cards to fill out. We know more because we can observe things we couldn’t observe before.

Clever analysis deserves some credit for the successes of big data, but more credit goes to new sources of data and the technologies that make these sources possible.