Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Scala Big Data Frameworks and Tools

DZone's Guide to

Scala Big Data Frameworks and Tools

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools the author has gathered up in this article.

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools I've gathered up below:

Livy is an open sourceREST interface for interacting with Spark from anywhere and used by Apache Zeppelin and other toolsUsing Livy with Zeppelin is useful documentation to install and operate Livy with Apache Zeppelin.

I found a lot of interesting examples, tutorials, starter code and samples using Scala for Big Data:

Scala with MQTT IoT on Raspberry Pi Example (GitHub) and if you don't know about IoT, take a Free IoT Class.

Akka HTTP for scalable HTTP microservices and RESTful JSON web services. Akka-HTTP Microservice with Websockets (GitHub) is a nice tutorial and article for using Akka-HTTP with Websockets.  An alternative to Akka HTTP is Spray Demo for Scala REST API DevelopmentSpark-Spray-Akka-Couchbase (GitHub) and Spark-Spray Starter (GitHub) are good examples of Spray. It's still debated if Spray will be deprecated and moved to Akka HTTP. It has not happened yet, not sure it ever will. Spray is rock solid and enterprise scale so it's still a worthy investment for development.

I found three great Github repos for learning Spark:

For those trying to determine what streaming framework to use, check out this Article on All Forms of Streaming. Once you get your data from streaming, you will want to access and display it, an example Web application in Scala, Play Example App (Scala/SBT)Scalable Probabilistic Machine Learning in Java which works with Apache Flink, is one of the best choices for Streaming in Java.  I found a nice article and code example of combining Kafka, Spark Streaming and Stanford Core NLP:  Kafka + Spark Streaming + Stanford NLP (GitHub).  

Many Big Data applications are console applications, this is a nice introduction to writing Modern Scala Console Apps. That article will give you a good start.   Finally a quick look at an interesting deep learning framework: Keras: Deep Learning library for TensorFlow and Theano is a minimalist, highly modular neural networks library (GitHub).

Hortonworks Community Connection (HCC) is an online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub.  Join the discussion.

Topics:
spark ,big data ,scala ,spray ,akka ,zeppelin

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}