Where's Big Data Going?
Where's Big Data Going?
Here are the opinions of 22 executives about what the biggest opportunities are in the continued evolution of big data.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
To gather insights on the state of big data in 2018, we talked to 22 executives from 21 companies who are helping clients manage and optimize their data to drive business value. We asked them, "Where do you think the biggest opportunities are in the continued evolution of big data?" Here's what they told us:
- Intelligence via ML. Smart homes become smarter. Cars get smarter. Start leveling business and personal lives by making lives easier.
- Big data is here to stay. It’s the biggest technological revolution along with the internet and mega-computing. You can see the true value with Amazon’s and Netflix’ recommendation engines. There will be a natural evolution of AI/ML voice interfaces changing how people operate and interact with machines to reduce friction. This creates positive change in how we work with each other and how companies operate. Big data has come to represent a new and significant step in the evolution of computers; prior steps are represented by the invention of the automated computers in the 50’s, the design of computer communications and the internet in the 70’s, the commercial web in the 90’s, and the social media revolution in the 2000s. Automated systems can generate lifelike images and create written content that can be of quality compared to what an expert writer would generate, and we can interact with systems just using our voice (if you don’t believe me, ask Alexa). In short, big data represents a unique transformational opportunity for humanity, together with Artificial Intelligence (AI) and Machine Learning (ML).
- We believe stream processing is the next big thing in big data. Businesses can no longer compete in today’s environment if they are waiting to receive daily, weekly, or monthly “reports” on the health of their business. It’s an untenable situation, and the companies who are winning, and will continue to win, are the data-savvy companies like Netflix, Alibaba, and Uber. They understand instantly what is happening with their business and how to react to a changing reality. With stream processing, data is processed instantly, which means businesses can react to changing dynamics and new situations in the moment to detect fraud, spot supply chain issues before they impact the customer (and bottom line), provide more personalized service to customers to keep them happy and build loyalty, and so forth. The impact of this can’t be understated.
- Big data initiatives must be driven by business outcomes. Cloud should be used more to optimize IT spending and above all allow the company to focus on business problems to solve. It still takes time today to analyze the data and get actionable insights. The big data evolution will be on increased real-time views while data protection and privacy by design are fully integrated.
- I will start with characterizing Big Data Analytics to be the processing of the maximum possible amount and types of data in the time allotted where decisions are made. Given that characterization, the biggest opportunities will be:
- 1) Continually exploiting hardware advances to make better decisions faster using even more data in the allotted time. Examples include a) Persistent Memory (3D XPoint/Optane/HPE Persistent Memory), b) Multicore CPUs with hardware transactions and SIMD instructions, c) Many Integrated Core (MIC) and general accelerated computing. Think Intel Xeon Phi, NVIDIA GPUs, and FPGAs. This cannot be understated in my opinion. Too many software solutions don’t truly exploit concurrency and parallelism available in modern hardware. And finally, d) Exploiting faster connectivity and interconnectivity options for locally on servers and connected servers.
- 2) Leveraging Big Data Analytics for predicting outcomes or behavior in order to increase beneficial opportunities or mitigate risk.
- 3) Skilled individuals with both broad technical capabilities to achieve the aforementioned.
- Data is continuing and will continue to grow. Everything will be integrated providing sets of data that can solve specific business problems.
- Business’ ability to unify analysis and operations while adding intelligence. 90% of success with ML is data management. Developers – data fabric exposes interfaces so they can move containers anywhere and access as local. Microservices work the same way. Publish and subscribe are part of the same fabric.
- We’re at a tipping point with the performance and architecture perspective. More real-time, automated intelligence and analysis will be in applications. There's an opportunity to innovate more and faster by converging microservices into big data. Decentralized analytics and transactions together in one platform. More data-driven microservices.
- It becomes more pervasive so companies can continue to become more responsive to customers in real time. Next generation apps will continue to scale even faster.
- We want everything to be self-service at work and in our personal lives. Bringing self-service to a more sophisticated user. Respecting the security controls of the business.
- Big data is just becoming data. There will be no differentiation. 1) Strata Hadoop is now just Strata Data. According to Gartner, Hadoop is obsolete. Big data technology is changing rapidly. Now integrated into enterprise-grade solutions. Data lake technology will evolve to store and analyze later. 2) Data management becomes more important – governance, management, individual distributed processing while storing the data everywhere across a diverse landscape.
- Big data is just getting bigger and faster. If you’re not already involved in big data, you’re late and in jeopardy of being passed by your competitors. Data is the number one business driver in every industry. Investment in data management technology will increase over time. ML, DL need GPU databases to solve problems.
Here’s who we spoke to:
Opinions expressed by DZone contributors are their own.