I do a lot of presentations these days at meetups and conferences with one focus: How to leverage Apache Kafka and Kafka Streams to apply analytic models (built with H2O, TensorFlow, DeepLearning4J and other frameworks) to scalable, mission-critical environments.
As many attendees have asked me, I created a video recording about this talk (focusing on live demos).
I also see many Confluent customers talking about their challenges to deploy analytic models to a mission-critical, scalable production environment. This is a completely different story than "just" developing a great, accurate model in R or Python. Educating them how Apache Kafka and Kafka Streams can help here is a key task for me these days! This leads to many very interesting and disrupting use cases. I will blog more about this in the next months. For example, I will show an example where I train a neural network with the concept of autoencoders to build analytic models. Some use cases for this include anomaly detection for predictive maintenance, fraud, and customer churn. These neural networks will then be deployed and monitored with Apache Kafka and its Streams API.
Abstract of the Session: Apache Kafka + Machine Learning
Intelligent real-time applications are a game changer in any industry. This session explains how companies from different industries build intelligent real-time applications. The first part of this session explains how to build analytic models with R, Python, or Scala. No matter which language you favor, you can leverage open-source machine learning/deep learning frameworks like TensorFlow, DeepLearning4J, or H2O.ai. The second part discusses the deployment of these built analytic models to your own applications or microservices. Here, you leverage the Apache Kafka cluster and the Kafka Streams API instead of setting up a new, complex stream processing cluster. The session focuses on live demos. It also teaches lessons learned for executing analytic models in a highly scalable, mission-critical and performant way.
Key Takeaways for the Audience
- Insights are hidden in historical data (for example, on big data platforms such as Hadoop).
- Machine learning and deep learning can help you find these insights by building analytics models.
- Stream processing uses these models (without redeveloping) to act in real-time.
- See different open-source frameworks for machine learning and stream processing like TensorFlow, DeepLearning4J, or H2O.ai to build analytic models.
- Apache Kafka, its Streams API, and machine learning can be combined to build, apply, and monitor analytic models.
- Understand how to leverage Kafka Streams to use analytic models in your own streaming microservices.
- Learn best practices for building and deploying analytic models in real-time leveraging the open-source Apache Kafka Streams platform.
Code Examples on GitHub
You can find the Java code examples and analytic models for H2O and TensorFlow in my GitHub project. Just clone the repository and run
maven clean package. Then, take a look at the unit tests to understand how to apply analytic models with Apache Kafka's Streams API.
Video Recording: Apache Kafka + Kafka Streams + H2O.ai + TensorFlow
Finally, here we go with the video recording:
As always, I appreciate any comments (feedback, questions, criticism, anything!). Have fun watching the video.