What's Preventing Big Data Success?
The most common issue preventing companies from realizing the benefits of big data is the inability to evolve from legacy technology.
Join the DZone community and get the full member experience.Join For Free
To gather insights on the state of big data in 2018, we talked to 22 executives from 21 companies who are helping clients manage and optimize their data to drive business value. We asked them, "What are the most common issues you see preventing companies from realizing the benefits of big data?" Here's what they told us.
- Depending on traditional systems. The people aspect is real, as expertise is needed to leverage big data systems. How to enable existing employees to use the data. Find a mix of people who can solve problems together. Have a desired end state but start small. Get wins. Stay focused. Have thoughtful, methodical implementation.
- Inability to deal with the shifting sands and technical debt of legacy systems and new software.
- Willingness to embrace the cloud. Understand that there are multiple ways to approach. It is not feasible to keep supporting legacy enterprise systems. They are not able to scale with the influx of data.
- Setting up the right backbone infrastructure (i.e. storage, transport, compute, failover). Getting data delivered from servers to analyze. How to deal with datasets. Scale, complexity, modeling.
Lack of Knowledge
- They don’t understand the cloud. They will do a “lift and shift” adopting Infrastructure as a Service without gaining any efficiency because they do not understand the benefits. They’ll kill their IT department and end up outsourcing management of the cloud to a third-party provider, still not understand the potential efficiency gains. More like Salesforce where they use the cloud for features, scalability, performance, and storage savings. Elastic cloud will scale up and down. You must use an SQL servid4r network and other components to scale instantly. Public cloud providers are now providing cognitive and AI/ML.
- They like the promise of big data but do not understand specific use cases. There’s lack of buy-in by the different lines of business or specific business drivers. Lack of understanding of the best technology for the job, be it a data lake, platform, cloud, or software. It’s a complex decision and one that changes daily with all of the new solutions being introduced. It’s not a good idea to rebuild a data warehouse in a Hadoop data lake. Skillsets are less of an issue because of public cloud toolsets but you still need to understand the use case and the best tools to accomplish your goals.
Business Problem Definition
Data Quality and Management
- Complexity in the technology stack. Retailers want real-time information from shopping carts and 12 months of purchasing history. Stitch three or four systems together. More moving parts result in more opportunities for breakage and latency. Help simplify the data pipeline for greater availability. Data architect the enterprise so that it’s able and ready to scale.
Here’s who we spoke to:
Opinions expressed by DZone contributors are their own.