Over a million developers have joined DZone.

Scala Big Data Frameworks and Tools

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools the author has gathered up in this article.

· Big Data Zone

Learn how you can maximize big data in the cloud with Apache Hadoop. Download this eBook now. Brought to you in partnership with Hortonworks.

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools I've gathered up below:

Livy is an open sourceREST interface for interacting with Spark from anywhere and used by Apache Zeppelin and other toolsUsing Livy with Zeppelin is useful documentation to install and operate Livy with Apache Zeppelin.

I found a lot of interesting examples, tutorials, starter code and samples using Scala for Big Data:

Scala with MQTT IoT on Raspberry Pi Example (GitHub) and if you don't know about IoT, take a Free IoT Class.

Akka HTTP for scalable HTTP microservices and RESTful JSON web services. Akka-HTTP Microservice with Websockets (GitHub) is a nice tutorial and article for using Akka-HTTP with Websockets.  An alternative to Akka HTTP is Spray Demo for Scala REST API DevelopmentSpark-Spray-Akka-Couchbase (GitHub) and Spark-Spray Starter (GitHub) are good examples of Spray. It's still debated if Spray will be deprecated and moved to Akka HTTP. It has not happened yet, not sure it ever will. Spray is rock solid and enterprise scale so it's still a worthy investment for development.

I found three great Github repos for learning Spark:

For those trying to determine what streaming framework to use, check out this Article on All Forms of Streaming. Once you get your data from streaming, you will want to access and display it, an example Web application in Scala, Play Example App (Scala/SBT)Scalable Probabilistic Machine Learning in Java which works with Apache Flink, is one of the best choices for Streaming in Java.  I found a nice article and code example of combining Kafka, Spark Streaming and Stanford Core NLP:  Kafka + Spark Streaming + Stanford NLP (GitHub).  

Many Big Data applications are console applications, this is a nice introduction to writing Modern Scala Console Apps. That article will give you a good start.   Finally a quick look at an interesting deep learning framework: Keras: Deep Learning library for TensorFlow and Theano is a minimalist, highly modular neural networks library (GitHub).

Hortonworks DataFlow is an integrated platform that makes data ingestion fast, easy, and secure. Download the white paper now.  Brought to you in partnership with Hortonworks

Topics:
spark ,big data ,scala ,spray ,akka ,zeppelin

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}