Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Machine Learning Infrastructure for Extreme Scale With the Apache Kafka Open-Source Ecosystem

DZone's Guide to

Machine Learning Infrastructure for Extreme Scale With the Apache Kafka Open-Source Ecosystem

See how to build ML models at extreme scale and productionize the built models in mission-critical real-time apps by leveraging open-source components in the public cloud.

· AI Zone ·
Free Resource

Enable your enterprise to add AI to your existing infrastructure with EdgeVerve’s Business Applications built on AI platform Infosys Nia™. Register for our webinar to learn more.

I had a new talk presented at Codemotion Amsterdam 2018 this week. I discussed the relation of Apache Kafka and machine learning to build a machine learning infrastructure for extreme scale.

The long version of the title:

" Deep Learning at Extreme Scale (in the Cloud) With the Apache Kafka Open-Source Ecosystem: How to Build a Machine Learning Infrastructure With Kafka, Connect, Streams, KSQL, etc."

As always, I want to share the slide deck. The talk was also recorded. I will share the video as soon as it is published by the organizer.

The room was full — no free seats. There's a lot of interest in this topic. I talked to many attendees who have huge challenges bringing analytic models into mission-critical production. No scalability and missing flexibility were other challenges many people had when using, for example, just a Python environment to build an analytic model.

Apache Kafka as Key Component in a Machine Learning Infrastructure

The open-source Apache Kafka ecosystem helps in many situations of a machine learning process — data integration, data ingestion, data preprocessing, model deployment, monitoring, etc. Many companies have built a Kafka ML infrastructure already. Take a look at tech giants like Netflix with Meson or Uber with Michelangelo, just to mention two examples.

Tech giants and many other companies use Apache Kafka at very large scale already. So, you do not need to worry about this. You just need to evaluate how you integrate it into your existing environment and projects. See examples like LinkedIn (> 4.5 trillion messages per day) or Netflix (peak of 6 Petabyte per day).

Machine learning and Apache Kafka are both parts of their core infrastructure. I see similar scenarios at most traditional companies (like banks, telcos, retailer) these days. They all build a central nervous system around Apache Kafka and apply analytic models in several scenarios and business processes.

Abstract of the Talk: Machine Learning and Deep Learning With Apache Kafka

This talk shows how to build machine learning models at extreme scale and how to productionize the built models in mission-critical real-time applications by leveraging open-source components in the public cloud. The session discusses the relation between TensorFlow and the Apache Kafka ecosystem — and why this is a great fit for machine learning at extreme scale.

The machine learning architecture includes Kafka Connect for continuous high volume data ingestion into the public cloud, TensorFlow leveraging Deep Learning algorithms to build an analytic model on powerful GPUs, Kafka Streams for model deployment and inference in real-time, and KSQL for real-time analytics of predictions, alerts, and model accuracy.

Sensor analytics for predictive alerting in real-time is used as a real-world example from Internet of Things scenarios. A live demo shows the out-of-the-box integration and dynamic scalability of these components on Google Cloud.

Machine learning infrastructure based on the Apache Kafka and Confluent open-source ecosystem:

Apache Kafka and Confluent Open Source Ecosystem for Machine Learning and Deep Learning

Key Takeaways for the Audience

  • Data scientists and developers have to work together continuously (org + tech!)
  • Mission-critical, scalable production infrastructure is key to the success of machine learning projects.
  • Apache Kafka ecosystem + cloud = machine learning at extreme scale (ingestion, processing, training, inference, monitoring)

Slide Deck: Apache Kafka + Machine Learning at Extreme Scale

Here is the slide deck of my talk:

As always, I appreciate any feedback.

Adopting a digital strategy is just the beginning. For enterprise-wide digital transformation to take effect, you need an infrastructure that’s #BuiltOnAI. Register for our webinar to learn more.

Topics:
kafka ,machine learning ,deep learning ,open source ,tensorflow ,mxnet ,confluent ,ksql ,stream processing ,ai

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}