Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now. Brought to you in partnership with Hortonworks.
Apache Spark and Machine learning can solve a lot of problems and can discover patterns leading to rules replacement. But for some simple cases and for legacy rules requires, you may still need to hand develop rules and implement in Java. For those cases, Drools can be used with Spark. Besides machine learning, CEP or complex event processing can be used as a replacement for standard IF-THEN-ELSE style hardcoded rules.
Siddhi Complex Event Processing Engine
Use Complex Event Processing (CEP) instead of rules:
Decision CEP Engine - complex event processing platform build on spark streaming
Spark plus Drools
Break up your kafka messages
Rules / File Matching
Business Rules - JRules Data Mining
There's a ton of great presentations, webinars and short online courses for learning Spark. Here are a few that I recommend.
More Free Spark Training
Apache Spark Essentials
Build and Monitor Spark Applications
Create Data Pipelines with Spark
MapR Free Spark Training
When implementing a SQL Datawarehouse on Hadoop with Spark, here are a few useful starters.
SQL Data Warehouse With Spark
Building a Datawarehouse
Spark Streaming for Robust Apps
Spark Summit Keynote
Memory-centric Distributed Storage System
Alluxio (Tachyon) - in memory
New Fast SQL Project
Alluxio is proving to be a great general purpose in memory file system.
Hortonworks Sandbox is a personal, portable Apache Hadoop® environment that comes with dozens of interactive Hadoop and it's ecosystem tutorials and the most exciting developments from the latest HDP distribution, brought to you in partnership with Hortonworks.
big data,spark,apache spark,spark sql