Is Big Data in the Trough of Disillusionment?

big data is showing the first signs of heading for gartner’s trough of disillusionment as a spate of voices, some very well known, like nassim taleb , author of antifragile and other prescient books, begin to question the hype. i’m going to show you why big data is in the trough in some ways and beyond that and on the slope of enlightenment in others.

the hype cycle

in case you’re not aware, gartner publishes a hype cycle for emerging technologies meant to give us an idea of when expect technology to be commercially viable. with all of the noise in the marketplace fed by both vendors and media, it would be easy to think anything less than the latest product offering is already antiquated and won’t work. gartner’s hype cycle helps to bust that myth.

because the truth is much different. most companies can’t afford to chase their technology tail and most it departments can’t suffer the lost credibility of technology that isn’t ready for prime time.  all technology goes through a hype cycle that follows the pattern in this image, even if the peak may not be as high or the trough as deep…or it could be worse.

big data’s tracking along the hype cycle

getting back to big data, we’re seeing a technology pattern that is certainly evolving. gartner has five ‘key phases’ that determine where technology falls, and it would be worthwhile to break down big data’s circumstances using gartner’s familiar language:

technology trigger: the technology breakthrough that brought big data to technology stage was the development of hadoop. while it is a usable technology, it wasn’t developed for public consumption and is challenging for organizations that don’t have strong it support. hadoop came about to satisfy the demands of web scale problems like search and social media.

peak of inflated expectations: there were certainly early success stories from the companies that developed and first used hadoop like google, facebook and yahoo. their success launched a series of hadoop-based solutions that attracted vc investment, including companies like cloudera, datameer, platfora and sisense . likewise, there has been fast growth for powerful analytics solutions that allow business users to  manipulate and visualize enormous data sets. the marketplace doesn’t lack for solutions.

trough of disillusionment: in some industries and use cases, we’re at the point where the rubber meets the road…the point where big data delivers or doesn’t. what happens now determines if we’re extending the peak of expectations or starting the slide into the trough. patience with experimentation will wear thin over the next year or so and there need to be more ‘everyday’ companies taking advantage of big data and talking about their successes. big data is headed for the trough as long as there are more people trying than succeeding, and that’s where we are right now.

slope of enlightenment: curiously, in some ways big data is on the slope already in just some areas. there are use cases around pharmaceutical clinical trials , customer preference , transport and logistics, healthcare and others. in some cases, second and third generation products are here, especially in the analytics space. there are many home-built solutions. where hadoop-y vendors need to be careful is simple: big data was here before the hype.

many of the organizations with demonstrated success on the books were managing big data before hadoop was developed. for them, hadoop isn’t a breakthrough and is just a choice among many. this is where you’ll see big data hype get eye rolls and groans.

the irony of big data is that the places where success is quite clear are often the places where the term simply isn’t used.

plateau of productivity: this is a far-off land at the moment for all but those just mentioned. the mainstream technology buyer is struggling with the plethora of messages and products, suspicious of the arguments that make big data sound too good to be true.

nearly on year ago, gartner had big data nearing the peak of inflated expectations (see below). it will be interesting to see where they place big data for the report due out this summer.

july 2012 gartner hype cycle