Big Data Predictions for 2019

A big data expert and IT industry executive shares his thoughts on topics such as major platforms, open source projects, and more.

· Big Data Zone · Interview

In addition to the minds inside Unravel Data, we have a team of advisors that have a phenomenal track record of entrepreneurship, leadership, and engineering. Two of them had some keen perspectives to share on what 2019 for big data. Over the course of the next two blogs, we’ll introduce you to Herb Cunitz and Tasso Argyros as they outline their take on what’s in store for Big Data in 2019.

Customer and vendor activity continued to accelerate in 2019 and all indications are that we’ll continue to see innovation advance at pace and to see how modern data applications are helping companies of all types be more competitive and drive more value to the bottom line. Over the next few weeks, we thought it would be informative to solicit some points of view on what’s next for Big Data in the coming year from people who have been in it and leading that change for the past number of years. Our First guest is Herb Cunitz, principal owner of AccelG2M.

Herb has been on the front lines of enterprise software and big data for many years and has grown and managed global field operations for multiple private, venture-funded technology companies including Hortonworks, SpringSource, and Vitria. He serves on several boards of directors and advises companies on how to rapidly accelerate growth and dominate categories.

Big data was in a constant state of change in 2018—new players, products, technology, AI, and vendor shakeups all lead the charge. Will 2019 see a similar course or will we see surprising growth and new use cases? We sat down recently with Herb; the following is an excerpt of our conversation.

Unravel: 2018 was an interesting year for big data—what is your take on how the year played out?

HC: It started with three major players on the platform side and companies coming to the realization that the market would continue to grow and grow well, but the rapid growth would be driven by continued usage among larger companies rather than every company adopting big data. We started to see more focus on the larger companies at the top of the pyramid—Cloudera focusing on the Fortune 1000, and Hortonworks doing the same—and then figuring out how you get that platform adopted. In mid-2017-2018, we saw the focus on platform adoption, and that is now shifting to ‘what can I do with that platform and those investments?’

Additionally, migration to the cloud gained momentum. The big data market started as an on-premises market, and now it’s transitioned to ‘how can I leverage the cloud and where else can I monetize and derive more value as more workloads move to the cloud?’

Furthermore, I see hybrid cloud solutions staying here for a long time to come. While cloud was at one time seen as a panacea, we’ve realized there are some things that can’t and shouldn’t move to the cloud—but there are a lot of workloads that will, and a lot of new workloads are getting formed and incubated in the cloud. But hybrid is not going away. For a long time to come, we’ll see the focus center on managing and monitoring workloads in both environments, and organizations will want to figure out how to deliver a common infrastructure control plane across both environments to see how their workloads are doing.

Unravel: The technology stack is in constant flux – new innovations and yesterday’s innovations dying out. If you were to place a bet on a breakout technology for 2019—Spark, Kafka, Flink, etc.—where would you put your money?

HC: There are open source projects gaining more interest. For instance, Spark, but it has had interest for years, and has already had its breakout. On the streaming side, Kafka had a breakout in 2018, and we’ll continue to see more of it as more workloads leverage streaming data and organizations wonder how they can take advantage of that data. In general, it is the technology that gives companies the ability to monitor, manage, control, and gain visibility into workloads running across the hybrid cloud—that’s where we’ll see the breakout technologies. And they’ll be providing opportunities that never existed before.

Another notable technology is serverless computing, which is getting significant interest, but not a lot of actual production usage. The world is still trying to figure out how to consume containers and Kubernetes, and we can’t just bypass that step and go straight to serverless. The more immediate need is the answer to ‘how do I get containers working? More specifically, how do I move from a virtualized environment to a containerized environment and do so securely and at scale?’

Still, some functions will move to serverless, but I don’t see that as the highest priority. We have to digest the whole container world first. Also, we have to define how Kubernetes relates to big data for production environments vs. the experimental projects we see today.

Unravel: What does big data adoption look like in 2019 and where will we see it?

HC: Smaller companies will continue to adopt big data. The larger early adopters, such as large financial service institutes and telcos, are able to leverage big data in full production now and have gained a significant competitive advantage. Now that Hadoop has become more productized, big data will move downstream to smaller companies with more success.

With larger companies, we’re seeing they have two growth vectors in relation to big data—one, large projects that have a lot of data and need more capacity, and two, a need for new use cases. Smaller companies have one vector, which is growing data, and have fewer use cases.

Unravel: What role will AI and automation play in big data in the coming year?

HC: You can liken this to Maslow’s Hierarchy of Needs. We’ve got the foundation covered, and above that is what companies can do with it, then what applications can be built and what use cases can be deployed—which we saw a lot of in 2018. Now that we have hybrid, the focus is all about how to optimize those environments. As the broader industry progresses, we’ve moved up the hierarchy pyramid where we can now tackle the optimization, management, and visibility aspects. Three years ago the mass market wasn’t ready for this, but it is now.

Unravel: Will there be any big vendor shake-ups in 2019?

HC: We just witnessed platform consolidation through Cloudera and Hortonworks, and the next tier of consolidation will be the companies that are built around and on top of the platform stack—but we’ll still have to see how that plays out. Cloudera is positioned to do well—they continue to grow, scale, and excel in their space, along with Amazon and Microsoft. But they need to continue to figure out how they differentiate their services in the cloud.

The notable incumbents (i.e., IBM, HPE) attempted to position themselves as leaders in big data and largely failed. I don’t see them re-entering the space to try it again. The next tier of incumbents is emerging, such as Cloudera, and they’ll continue either through their own innovation or by acquiring others to accelerate time to market.

Unravel: How do you see organizations reacting to the increasing big data skills gap and demand for talent?

HC: I see smaller companies using outsourced services, and bigger companies contracting for managed services.

A big skill set need is in data science and interestingly, we’ve seen universities develop the requisite degree programs and focus more on them than they ever had before. As a result, they are producing more graduates in this field, which is a big benefit to our industry. Three years ago, we weren’t seeing the abundance of data science programs or graduates that we are today.

Unravel: What do you think about the investment community when it comes to innovation funding—will there be a hot spot in 2019?

HC: The majority is still in the Bay Area, but I’m also seeing New York strengthening with some interesting newcomers. Internationally, we’re seeing more growth in Israel and India, too.

Unravel: Your closing thoughts on the 2019 market?

HC: The market has moved to a point where platforms are in place, skill sets are in place, and there’s enough experience to know how to deploy and manage big data. Now, companies want to optimize, manage, and control that environment, and they want to do that with the technologies that best suit their needs—thus anything that can work across multiple environments, multiple planes, and multiple vendors is an essential tool that enterprises will spend money on.

Published at DZone with permission of Kunal Agarwal, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.