Where's Big Data Going?

To gather insights on the state of big data in 2018, we talked to 22 executives from 21 companies who are helping clients manage and optimize their data to drive business value. We asked them, "Where do you think the biggest opportunities are in the continued evolution of big data?" Here's what they told us:

AI/ML

• Cognitive and AI/ML accessible through public cloud providers are going to provide higher level services that add business value for clients. Use a vendor’s AI and apply to business needs. Nine months ago, Microsoft supported 35 languages. Today, they support 52. Cognitive services for speech recognition can now identify who is speaking. 
• We are yet to realize the problems we’ll solve with big data in healthcare. Disease diagnosis with AI/ML will be able to detect patterns humans never could. 
• More data provides more opportunities to use AI/ML to augment resources. Embrace those technologies to provide automated insights.
• Static data goes away; data is always in motion. Files are less important as streams become more important. The infrastructure gets figured out. Storage and analytics are led by Google and Facebook’s work with AI/ML.
• There are more cases of insights from large data to make decisions using AI/ML. There are also more use cases with insights from big data. 
• Intelligence via ML. Smart homes become smarter. Cars get smarter. Start leveling business and personal lives by making lives easier.
• Big data is here to stay. It’s the biggest technological revolution along with the internet and mega-computing. You can see the true value with Amazon’s and Netflix’ recommendation engines. There will be a natural evolution of AI/ML voice interfaces changing how people operate and interact with machines to reduce friction. This creates positive change in how we work with each other and how companies operate. Big data has come to represent a new and significant step in the evolution of computers; prior steps are represented by the invention of the automated computers in the 50’s, the design of computer communications and the internet in the 70’s, the commercial web in the 90’s, and the social media revolution in the 2000s. Automated systems can generate lifelike images and create written content that can be of quality compared to what an expert writer would generate, and we can interact with systems just using our voice (if you don’t believe me, ask Alexa). In short, big data represents a unique transformational opportunity for humanity, together with Artificial Intelligence (AI) and Machine Learning (ML).

Real-Time

• We believe stream processing is the next big thing in big data. Businesses can no longer compete in today’s environment if they are waiting to receive daily, weekly, or monthly “reports” on the health of their business. It’s an untenable situation, and the companies who are winning, and will continue to win, are the data-savvy companies like Netflix, Alibaba, and Uber. They understand instantly what is happening with their business and how to react to a changing reality. With stream processing, data is processed instantly, which means businesses can react to changing dynamics and new situations in the moment to detect fraud, spot supply chain issues before they impact the customer (and bottom line), provide more personalized service to customers to keep them happy and build loyalty, and so forth. The impact of this can’t be understated. 
• Big data initiatives must be driven by business outcomes. Cloud should be used more to optimize IT spending and above all allow the company to focus on business problems to solve. It still takes time today to analyze the data and get actionable insights. The big data evolution will be on increased real-time views while data protection and privacy by design are fully integrated. 
• I will start with characterizing Big Data Analytics to be the processing of the maximum possible amount and types of data in the time allotted where decisions are made. Given that characterization, the biggest opportunities will be:
  • 1) Continually exploiting hardware advances to make better decisions faster using even more data in the allotted time. Examples include a) Persistent Memory (3D XPoint/Optane/HPE Persistent Memory), b) Multicore CPUs with hardware transactions and SIMD instructions, c) Many Integrated Core (MIC) and general accelerated computing.  Think Intel Xeon Phi, NVIDIA GPUs, and FPGAs. This cannot be understated in my opinion.  Too many software solutions don’t truly exploit concurrency and parallelism available in modern hardware. And finally, d) Exploiting faster connectivity and interconnectivity options for locally on servers and connected servers.
  • 2) Leveraging Big Data Analytics for predicting outcomes or behavior in order to increase beneficial opportunities or mitigate risk.
  • 3) Skilled individuals with both broad technical capabilities to achieve the aforementioned. 
• Greater speed and reusability of data management processes with greater trust in the data. Less manpower required to manage big data.

Integration

• Data is continuing and will continue to grow. Everything will be integrated providing sets of data that can solve specific business problems. 
• Business’ ability to unify analysis and operations while adding intelligence. 90% of success with ML is data management. Developers – data fabric exposes interfaces so they can move containers anywhere and access as local. Microservices work the same way. Publish and subscribe are part of the same fabric. 
• We’re at a tipping point with the performance and architecture perspective. More real-time, automated intelligence and analysis will be in applications. There's an opportunity to innovate more and faster by converging microservices into big data. Decentralized analytics and transactions together in one platform. More data-driven microservices. 
• Making it easy to unify data no matter where it may be stored and run analytics in real-time at memory speeds with any analytics framework.

Other

• It becomes more pervasive so companies can continue to become more responsive to customers in real time. Next generation apps will continue to scale even faster.
• We want everything to be self-service at work and in our personal lives. Bringing self-service to a more sophisticated user. Respecting the security controls of the business.
• Big data is just becoming data. There will be no differentiation. 1) Strata Hadoop is now just Strata Data. According to Gartner, Hadoop is obsolete. Big data technology is changing rapidly. Now integrated into enterprise-grade solutions. Data lake technology will evolve to store and analyze later. 2) Data management becomes more important – governance, management, individual distributed processing while storing the data everywhere across a diverse landscape.
• Big data is just getting bigger and faster. If you’re not already involved in big data, you’re late and in jeopardy of being passed by your competitors. Data is the number one business driver in every industry. Investment in data management technology will increase over time. ML, DL need GPU databases to solve problems.

Here’s who we spoke to: