DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Apache Kafka Streams to Build Highly Scalable Machine Learning Microservices

Apache Kafka Streams to Build Highly Scalable Machine Learning Microservices

Learn how to build analytic models with R, Python, and Scala, how to leverage these built analytic models in your own real-time streaming applications, and more.

Kai Wähner user avatar by
Kai Wähner
CORE ·
May. 25, 17 · Opinion
Like (7)
Save
Tweet
Share
7.78K Views

Join the DZone community and get the full member experience.

Join For Free

I started at Confluent in May 2017 to work as Technology Evangelist focusing on topics around the open-source framework Apache Kafka. I think Machine Learning is one of the hottest buzzwords these days, as it can add huge business value in any industry. Therefore, you will see various other posts from me around Apache Kafka (messaging), Kafka Connect (integration), Kafka Streams (stream processing), and Confluent's additional open source add-ons on top of Kafka (Schema Registry, Replicator, Auto Balancer, etc.). I will explain how to leverage all this for Machine Learning and other Big Data technologies in real-world production scenarios.

Read this if you wonder why am so excited about moving (back) to open source for messaging, integration and stream processing in the big data world.

In the following blog post, I want to share my first slide deck from a conference talk representing Confluent, a software architecture user group in Leipzig, Germany organized a two-day event to discuss Big Data in practice.

Apache Kafka Streams, Machine Learning, and Deep Learning

This is the abstract of the slide deck.

Big Data and Machine Learning are key for innovation in many industries today. Large amounts of historical data are stored and analyzed in Hadoop, Spark, or other clusters to find patterns and insights, e.g. for predictive maintenance, fraud detection or cross-selling.

This first part of the session explains how to build analytic models with R, Python, and Scala leveraging open-source Machine Learning/Deep Learning frameworks like Apache Spark, TensorFlow, or H2O.ai.

The second part discusses how to leverage these built analytic models in your own real-time streaming applications or microservices. It explains how to leverage the Apache Kafka cluster and Kafka Streams instead of building an own stream processing cluster. The session focuses on live demos and teaches lessons learned for executing analytic models in a highly scalable and performant way.

The last part explains how Apache Kafka can help to move from a manual build and deployment of analytic models to continuous online model improvement in real time.

Apache Kafka Streams and Machine Learning / Deep Learning

How to Build Analytic Models and Deployment to Real Time Processing

Here is the slide deck:


More blog posts with more details and specific code examples will follow in the next weeks. I will also do a web recording for this slide deck and post it on Youtube.

kafka Machine learning Stream processing Big data Open source Build (game engine)

Published at DZone with permission of Kai Wähner, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Top 12 Technical Skills Every Software Tester Must Have
  • Connecting Your Devs' Work to the Business
  • How and Why You Should Start Automating DevOps
  • Memory Debugging: A Deep Level of Insight

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: