Software Ate the World and Now the Models Are Running It

Along with our data ecosystem partners, we are seeing unprecedented demand for solutions to complex, business-critical challenges in dealing with data.

Consider this. Data Engineers walk into work every day knowing they’re fighting an uphill battle. The root of the problem – or at least one problem – is that modern data systems are becoming impossibly complex. The burgeoning amount of data being processed in organizations today is staggering, where annual data growth is often measured in high double-digit percentages. Just a year ago, Forbes reported that 90% of the world’s data was created in the previous two years.

And with that data growth has come rapid growth in the number of applications for ingesting, correlating, and analyzing that data. Each component of a data pipeline is by nature a specialist, and it takes lots of specialists to make data deliver results – and, more importantly, insights. This is a problem that touches virtually every corner of the world of business. And the pressure to perform and make data “work” is unrelenting.

In our own research from November 2018, Unravel found that three-quarters of businesses expect their big data stack to drive profitable business applications by the end of 2019 – but only 12% were seeing this value at the time.

The media is rife with stories prophesying magnificent discoveries to be made when data converges with artificial intelligence-driven models. Some of these discoveries have been made, but many more are still to come. Too often, these discoveries are over the horizon or well beyond the horizon, as data practitioners struggle with data systems that create more hurdles than they knock down.

Old Technologies Cannot Solve New Problems

Model-driven insights from data is what every business aspires to. The need for reliable and scalable application performance spawned the development of Application Performance Management (APM) and log management tools, two pioneering technologies in the race to make sense of new, multi-tier web architectures. The problem is that those technologies fell short because they were not designed and built for modern data systems. From the standpoint of the Data Engineer, the metrics and graphs those technologies deliver fall flat, when the team needs actual recommendations and answers to the issues faced multiple times every day.

“It’s clear that enterprises continue to struggle with dealing with the enormous amount of data that fuels their businesses. Legacy approaches have failed, and they need to modernize their systems or risk being made irrelevant,” said Venky Ganesan, managing director, Menlo Ventures.

Dealing With Overwhelming Complexity

Although it might be trite, it’s worth mentioning that every business is becoming a data business. That’s why most businesses consider data management systems such as Spark, Kafka, Hadoop, and NoSQL as their critical systems of record.

Data pipelines are so complex that they are outgrowing our ability to manage them. That’s because these systems have so many interdependencies that solutions lie beyond human intuition or deduction. And that’s why Unravel talks a lot about the importance of full-stack visibility for optimizing the performance of data-driven applications. We obsess over the need to explore, correlate, and analyze everything in your big data environment, search for dependencies and issues, understand how data and resources are being used, and discover how to troubleshoot and remediate issues.

And we believe in the promise of AI. That’s why Unravel integrated a powerful AI engine to deliver recommendations that drive more reliable performance in modern data applications.

Cloud Complicates Everything

As businesses migrate their data-focused applications and their data to the cloud, they face the fact that many cloud platforms provide only minimal siloed tools for managing these workloads.

In response, we unveiled its newest version of the Unravel platform, which focuses squarely on the unique requirements of hosting data-focused applications in the cloud. That release took the AI, machine learning, and predictive analytics that are the hallmarks of the platform and enabled users to assess which apps are the best candidates to move to the cloud – based on the customer’s own defined criteria.

The release also gave users the tools to validate the success of their cloud migration and predict capacity based on their specific application workloads. At the time, I noted that many unknowns around cost, visibility and migration had prevented this transition to the cloud from occurring more quickly. But that is no more.

Continuous Improvement

Continuous improvement: although the term is dated, the concept is still as timely as ever. And it’s a mantra of many businesses today that are never content, even with their highest achievements.

Continuous improvement is just the latest growth driver in modern data systems as well, and it’s being built on models. In turn, these models are built on closed-loop data. “When built right, these models create a reinforcing cycle: Their products get better, allowing them (businesses) to collect more data, which allows them to build better models, making their products better, and onward,” said Steven Cohen and Matthew Granade of Point72 Ventures, an investor in Unravel Data.

If anything is keeping CIOs from meeting their OKRs, number one on that list is likely data system complexity. Well, complexity is here to stay! In our data-driven world, gains come when we deal with the inevitable complexities and move beyond them. At Unravel, we think big data can do better, and we’re here to help it along by radically simplifying the way you do data operations, how your models perform and ensuring big data lives up to your expectations – both today and tomorrow.