Scala Big Data Frameworks and Tools

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools I've gathered up below:

Livy is an open sourceREST interface for interacting with Spark from anywhere and used by Apache Zeppelin and other tools. Using Livy with Zeppelin is useful documentation to install and operate Livy with Apache Zeppelin.

I found a lot of interesting examples, tutorials, starter code and samples using Scala for Big Data:

Scala with MQTT IoT on Raspberry Pi Example (GitHub) and if you don't know about IoT, take a Free IoT Class.

Akka HTTP for scalable HTTP microservices and RESTful JSON web services. Akka-HTTP Microservice with Websockets (GitHub) is a nice tutorial and article for using Akka-HTTP with Websockets.  An alternative to Akka HTTP is Spray Demo for Scala REST API Development, Spark-Spray-Akka-Couchbase (GitHub) and Spark-Spray Starter (GitHub) are good examples of Spray. It's still debated if Spray will be deprecated and moved to Akka HTTP. It has not happened yet, not sure it ever will. Spray is rock solid and enterprise scale so it's still a worthy investment for development.

I found three great Github repos for learning Spark:

• An Example of Spark Streaming with Twitter
• Sparkathon Code
• Introduction to Spark Streaming (In-Depth Tutorial)
• And Annotated Spark Streaming Code

For those trying to determine what streaming framework to use, check out this Article on All Forms of Streaming. Once you get your data from streaming, you will want to access and display it, an example Web application in Scala, Play Example App (Scala/SBT). Scalable Probabilistic Machine Learning in Java which works with Apache Flink, is one of the best choices for Streaming in Java.  I found a nice article and code example of combining Kafka, Spark Streaming and Stanford Core NLP:  Kafka + Spark Streaming + Stanford NLP (GitHub).  

Many Big Data applications are console applications, this is a nice introduction to writing Modern Scala Console Apps. That article will give you a good start.   Finally a quick look at an interesting deep learning framework: Keras: Deep Learning library for TensorFlow and Theano is a minimalist, highly modular neural networks library (GitHub).