Over a million developers have joined DZone.

Big Data Ends the Era of Hunches

DZone's Guide to

Big Data Ends the Era of Hunches

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

I made great money in college recruiting people for focus groups. Those were tough days for students in a state (New York) where the minimum wage was only $3.35 an hour. Instead, I was making $25 an hour working in a phone bank that called and convinced specific demographic groups to spend a few hours at a marketing company as part of a focus group.

I could recruit 30-year-old stay at home moms or 40-year-olds making between 55 and 65K per year…you name it, I found it.


This job so paid well because it was critically important and cost intensive to test products and services through focus groups. It took just the right mix to figure out how to sell to the general population with the right message for various audiences. Rather than hunch, which isn’t the most reliable plan, these focus groups were key to extrapolation, the best process we had for knowing the world before data became large and ubiquitous.

What’s truly changed is the amount of information that we now have on every imaginable demographic. Today’s systems track and store every purchase by every consumer in a way that turns us from extrapolation to intimate knowledge instead. Look no further than Netflix’s House of Cards, the very popular television program that was constructed using analytics of the massive amount of consumer patterns that Netflix has at their disposal. They didn’t need to focus group the concept to know it would work…the data told them it would work. And it did.


Beyond ending extrapolation, Big Data provides an enormous incentive for companies to push products and services out more quickly. Manufacturers can measure and iterate product development based on a stream of feedback that can now be captured and analyzed very quickly. Big Data has, effectively, lowered the upfront cost of product and service development and at the same time, increased the likelihood for longer-term success. This pattern changes the foundation of business models and allows more players to enter markets that have higher risk and reward.

Process change

Anything that changes the business model also changes business processes, meaning that Big Data kicks off a corporate-wide remodeling project, in effect. Companies that don’t remodel are likely to fall behind quickly as their competition iterates its way to better sales and lower costs. This isn’t some farfetched prediction. It is happening slowly across the marketplace in manufacturing, retail, healthcare, travel, and others and is gathering steam as the stories emerge.

The value of Big Data is found in these use cases, not necessarily in the “amazing insights” that are far more fuzzy and a lot less measurable.

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}