Closed Loop: Machine Learning and Real Time Stream Processing
A summary of Kai Wähner's talk at Voxxed Zurich about machine learning and real time stream processing with Apache Spark, TIBCO, Apache Storm, and other tools.
Join the DZone community and get the full member experience.Join For Free
In March 2016, I had a talk at Voxxed Zurich about “How to Apply Machine Learning and Big Data Analytics to Real Time Processing”.
Finding Insights With R, H20, Apache Spark MLlib, PMML, and TIBCO Spotfire
“Big Data” is currently a big hype. Large amounts of historical data are stored in Hadoop or other platforms. Business Intelligence tools and statistical computing are used to draw new knowledge and to find patterns from this data, for example for promotions, cross-selling or fraud detection. The key challenge is how these findings can be integrated from historical data into new transactions in real time to make customers happy, increase revenue or prevent fraud.
Putting Analytic Models Into Action With Apache Storm, Flink, Spark Streaming, or TIBCO StreamBase
“Fast Data” via stream processing is the solution to embed patterns – which were obtained from analyzing historical data – into future transactions in real-time. The following slide deck uses several real world success stories to explain the concepts behind stream processing and its relation to Apache Hadoop and other big data platforms. I discuss how patterns and statistical models of R, Apache Spark MLlib, H20, and other technologies can be integrated into real-time processing using open source stream processing frameworks (such as Apache Storm, Spark Streaming or Flink) or products (such as IBM InfoSphere Streams or TIBCO StreamBase). A great benefit is if a framework or tool already offers out-of-the-box connectors to execute analytic models without need for rebuilding the model respecively rewriting any code.
Slide Deck and Video Recording From Voxxed Zurich 2016
Here is the slide deck:
The video recording of this talk is available on Youtube:
Published at DZone with permission of Kai Wähner, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.