How Microservices and Fast Data Are Converging

Mark Brewer is CEO of Lightbend, the company known for bringing Reactive to JVM application development. With Strata Data Conference in New York, DZone caught up with Brewer to hear more about trends he is seeing around microservices in the enterprise, and to learn about the 2.0 version of Lightbend’s Fast Data platform.

As the company behind the Scala language, Lightbend was very early on making new abstractions for microservices and data-driven applications available to the broader JVM ecosystem. Talk us through that a little bit.

Neither the term “microservices” nor “cloud-native” existed yet when Lightbend founders Martin Odersky (Scala language creator) and Jonas Bonér (Reactive Manifesto author and creator of Akka) were evolving new abstractions for distributed systems. That applies to other technologists around distributed computing that recognized that both the nature of distributed systems themselves as well as the new requirements for those systems to handle complex data issues were pushing up against the limitations of both Java the language, and Java EE the application infrastructure.

The Reactive movement was a key arrival because it put some definition to the problem set that you start facing with distributed systems, and Scala from a language perspective introduced a new set of abstractions for how you deal with very complicated things like state, concurrency, asynchronous and other table stakes requirements for systems created as microservices, to run across clusters of commodity hardware. I think today we are seeing a lot of those new patterns making their way into the evolution of Java at the language layer, as well as the new directions for Jakarta EE for how the application stack around Java becomes more cloud-native in its native capabilities.

Talk through the data component of what you’re seeing around microservices today. What’s the fundamental difference that you’re seeing around application design as it pertains to handling data?

Well, there was a very long period of time where the average enterprise developer’s interaction with datastores was somewhat limited, and they thought of a database as an external resource that got called intermittently. And then with the Big Data era, we saw the emergence of a new class of applications written mostly for analytics use cases, fueled by batch processing.

But what’s happening now is on a whole new level of developer interaction with data, and data as a first-class system design consideration. Where batch jobs seldom lasted for more than a few hours, today’s distributed systems -- driven by real-time data, fast data, streaming data, whatever you want to call it -- run continuously. That they are always-on has huge design, scale and run-time considerations for the behavior of the applications, the business value of features that can be exposed, the uptime of systems. They need to be responsive, elastic, resilient. They need built-in conventions for how they handle failure. They need to be Reactive.

So at Lightbend, we see this major convergence between teams that have been doing large-scale big data in data science domains with the teams building large-scale distributed systems -- you put those two interests together and you get the world of fast data applications. It's a very interesting convergence that we see as the foundation to AI/ML, predictive analytics, and all of the most innovative new use cases that demand real-time access to data as it arrives. This whole conversation around data speeds has elevated beyond the analytics and executive dashboard conversations and is being driven into what’s inside of containers, and how the application logic supports data.

So tell us about the Lightbend Fast Data Platform, which hit its 2.0 milestone today, announced at the Strata event.

It started a few years ago when we recognized a class of customers was using our technologies--particularly Akka, which is a foundational distributing computing framework--in conjunction with other complementary technologies. A common stack called the SMACK stack emerged, which is Spark, Mesos, Akka, Cassandra, Kafka.

We saw how hard it was for enterprises to get started with this combination of building microservices specifically for fast data use cases. There’s the choice of many different streaming technologies, dependencies between these moving parts, and a lot of work to do just to stand these systems up, let alone monitor and manage them in real time.

So we decided to take our Reactive Platform, and add to it a curated set of the open source streaming and machine learning technologies that are typically used to build these streaming applications and create a complete distribution. That’s our Fast Data Platform--and the idea is that if you are looking to build a streaming application, if you have use cases where you want to be able to take big data and actually process it in real-time or near real-time, there is now a platform that is available for you to dramatically reduce the risk of failure because you don't need to worry about all these components working together.

That means you don't have to go out and hire super expensive, really hard to find experts in all of the underlying frameworks. And that's really what our value proposition is to global 2000 enterprises. Most of them have invested in the data science but are struggling to turn that into a system they run the business on 24 by 7.

The big news around the new release of Fast Data Platform is that you now support Kubernetes. What’s complementary to the set of concerns addressed by Reactive and Fast Data Platform, and what Kubernetes handles with the orchestration of containers?

As a starting point, containers are the de facto platform for microservices deployment. Developers have fully embracing microservices for distributed systems, and microservices and containers go hand-in-hand. And Kubernetes has clearly won out as the orchestration platform for containers, with the broadest industry support, including the most open source commits and the most complimentary technologies that keep getting layered on to the cloud-native stack.

With our embrace of Kubernetes as the deployment target for microservices for Fast Data use cases, we are bringing the most distributed computing friendly programming model to the world of containers. Reactive is the set of principles that deals with failure, high availability and complex data state issues at the application infrastructure layer--it’s everything fine-grained that needs to be guaranteed inside of the containers. Kubernetes is the coarse-grained management of the containers themselves. In Fast Data platform, we’ve brought them together to address a broad set of concerns into a single unified platform.