Systems Integration and Streaming Data at Reactive Summit

We interviewed Enno Runne, team lead on the Alpakka project at Lightbend, about his career, the evolution toward instantaneous data via Alpakka, and his upcoming talk at Reactive Summit next week.

First of all, let's talk about what your background is, and what initially drew you to systems integration.

I've been in the IT business in Sweden since 20 years back, and I've had a lot of different roles and consulting jobs and I've worked with a lot of different companies. Whenever you you work with systems, you come across systems integration, so it hasn't really been my focus area. But quite often, I happen to get in touch with it. Back in the Java Enterprise Edition days, it was important, and I've been working with it a lot around Akka and the Akka space. Apache Camel filled the gap there for a while. I'm quite excited for the next step of having integration going on all the time — going from this old-fashioned process of "we transfer the file from one system to the next and read it there" to having something that's combined. That actually leads into the talk I'm giving.

Could you say more about what you'll cover?

I'll have quite concrete examples, starting off with transferring files and from that, show how Akka Streams works. From having to transfer files going into something that's mostly linked to messages and how we kind of, like a strangler pattern, can move from an existing integration that might still be finalized to something that's transferring data more on-demand.

What would you say are the biggest challenges people encounter in moving to streaming data?

To really make use of it, you have to fight all the way. The stream needs to come out of one system in in a timely basis, and other systems must be ready to make use of it in a reactive way. I guess the biggest challenges are with the number of integrations we end up with. The streaming itself is more about what the systems can do, and how much live data they have. Keeping track of all the integrations needed is an issue. That's not new. That's really an old systems integration problem, to understand how much these systems connect together. But what we get with the record speeds that the online streaming has, are the properties of back pressure. They can understand, okay, this system can't handle this much, so I need to slow down. That's what we, with the Alpakka project, bring to existing technologies. Reactive integration is about bringing reactive space principles to cross-system communication.

What does the future of streaming data look like to you?

It will be the predominant way to integrate systems. Getting data fast from one system to another will be more and more in need. Systems won't be able to work with yesterday's exchange rate. As a user, you don't think about systems, of course, but as a service provider, you have lots of systems that need to have the same view of a customer's data. So as a user, you might edit some data there, and look at it on another page. In the background, the two systems we have go from there to disrupt the data when the change happens.

The expectation is going to be more and more instantaneous, and things are going to also grow increasingly complex.

Exactly. And we'll have to do that with the old systems. We can't just revise everything at once. We still have to comply with the existing ones.

Is there anything else that you think we missed talking about, that you'd like to mention now?

With the Alpakka project, which I'm leading, we integrate Reactive Streams and Akka Streams, specifically with technologies that have been around — some of them — for awhile, like databases and messaging technologies, so that you integrate with them seamlessly. The Alpakka connectors translate the properties of the connected systems to orchestrate as much as possible. For example, they help you integrate with Apache Kafka, in a streaming way, to Akka Streams. You can also integrate with MQTT, which is quite big in the IoT landscape. So that's what we do.

Thank you so much. I appreciate your time, and I hope you have a good afternoon.

All right. You too.