Big Data is already old news (i.e. its growth, its dynamism, and the rapid proliferation of its sources). Even the consequences and challenges of the Big Data explosion are now familiar and essentially solved (i.e. gathering, managing, storing, and processing). We have the tools to do these things at scale and with great efficiency, mostly by leveraging cloud computing and storage.
The Past Is Passed: Now, Turn Data Into Insight
The real question for today and tomorrow is this: How do you turn Big Data into truly meaningful insights that put you ahead of your competition?
The old ways of analyzing data — econometric models and the like — simply cannot deal with today’s data deluge. They can’t deal with continuous change in the explanatory variables or unstructured data like images or social media posts. And even if you managed to build a good model today using these traditional analytical tools, it will probably be totally out of date within just a few months — and trashing lots of good work and generating unactionable insights.
Even some of the newer approaches, which can handle Big Data sources, are increasingly problematic. Many companies have learned to mine social media data for sentiment analysis, but the biggest generators of social media no longer allow direct access to raw data, so this doesn’t work anymore. In the past, companies used crawlers to glean e-commerce data, but today most large e-commerce players like Amazon are smarter, faster, and more protective, so it’s much harder to get information at scale.
But here’s new news in Big Data: We are seeing the development and deployment of powerful new solutions every day that transform new volumes of data, new kinds of data, and new sources of data into insights much more intelligently. I’m talking about neural networks that can learn, reason, and adapt. Many of them can mimic great data analysts and scientists faster and at scale and without the risks of fatigue, attrition, knowledge transitions, etc.
This is how you transform Big Data into Big Profits. But — and this is a very big “but” — the know-how required to put neural networks to work is not commonplace. The good news is that by properly focusing and narrowing on a problem/opportunity area, the challenge can be very profitably managed.
A Specific Example: Bringing Intelligence to Life
Let me describe a recent Brillio engagement that demonstrates the kind of neural network-driven Big Data solution that is not just possible, it’s real and unbelievably powerful.
One of our clients is one of the largest consumer packaged goods (CPG) companies in the world. In the past, their field personnel would audit shelves in stores, a labor- and time-intensive effort to perform (a completely manual process) and to collate the data, massage it, and analyze it. They wanted to understand precisely and accurately what’s happening at the store level in something like real time.
Now, when field employees visit stores, they just need to open an app on their phone and take a few quick pictures of the shelves. The images are automatically uploaded to the cloud and analyzed in real-time — the system “reads” and interprets the photos and generates insights for both the field professional and the central team at the headquarters:
“Execution Violation Warning: Product X pricing should reflect the agreed campaign discount.”
“Product A is not placed on the right side of the aisle; please warn store personnel.”
“Competitor item Z has twice the amount of space they should have; take actions A, B, and C.”
“There’s a promotion on Product Y, so stock 20% more of the product, and you should come back tomorrow instead of next Tuesday.”
The system now has a wealth of new data for further analysis:
“Why is Product X selling more on Mondays in these markets?”
“What’s the impact of competitor promotion on sales of Product Y?”
“Is this planogram working better for me in this type of store or the other one? Why? By how much?”
These in-store audits used to take up to half an hour; now they take three minutes. And look at all the benefits of this system: capturing millions of pictures each day, automatically translating every image into analyzable and structured data, saving labor costs, enabling field professionals to focus on more value-added functions, providing real-time feedback, and enabling longer-term, in-depth analysis.
Data-Driven Insights Make Better Decisions
The field people can do their jobs better, which leads to higher sales. And analysts and managers in the head office have more and better data-driven rationales for making decisions and negotiating contracts. They can now say to their retail partners: “We paid for promotional efforts in 1,000 stores across the country, but our shelving information shows that the promotions are being implemented in just 742 stores.” Often, companies like our client will have a strong “sense” that execution isn’t as strong as it should be; now they can back their instincts with facts and insights: the basis for decisions and change.
Here are some of the technological components of this system: The in-store/field personnel are using a customized app based on Brillio Shelf Scanner for CPG to capture images of shelves at every store. These images are continuously uploaded (along with geolocation and related data) into a cloud database, where Brillio’s Image Analytics technology turns them into actionable data, which is pooled in a data lake using the Brillio Data Platform. The system uses Machine Learning through our Brillio AI Platform’s neural network to uncover insights from this rich new data stream — and the Machine Learning gets better and better at generating insights over time. (One of the biggest challenges of Machine Learning applications is the design of the training and feedback processes so that the system produces increasingly useful results.)
Machine Learning is valuable because of its ability to generate unlooked-for insights — and to continue to create fresh insights based on the constantly changing competitive landscape. But it’s also compelling because it can easily transfer the approach into different markets and different languages. The Shelf Scanner app is reading labels, price tags, and other information, and does not require being reprogrammed for different languages. More important, the Shelf Scanner app populates the data platform with good, real-time data and then generates insights that go beyond what even a great data scientist can produce.
I started out by saying that Big Data is old news. Of course, I don’t really believe that. I believe in the power of Big Data more than ever because I see how many clients are starting to use it effectively — like our CPG client and many others. The future lies not in learning to use Big Data but to put it to its best use — and that is going to demand the most sophisticated technologies like Machine Learning.