Apache Spark and Machine learning can solve a lot of problems and can discover patterns leading to rules replacement. But for some simple cases and for legacy rules requires, you may still need to hand develop rules and implement in Java. For those cases, Drools can be used with Spark. Besides machine learning, CEP or complex event processing can be used as a replacement for standard IF-THEN-ELSE style hardcoded rules.
There's a ton of great presentations, webinars and short online courses for learning Spark. Here are a few that I recommend.
More Free Spark Training
When implementing a SQL Datawarehouse on Hadoop with Spark, here are a few useful starters.
SQL Data Warehouse With Spark
Memory-centric Distributed Storage System
Alluxio is proving to be a great general purpose in memory file system.