DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > Scala Big Data Frameworks and Tools

Scala Big Data Frameworks and Tools

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools the author has gathered up in this article.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 11, 16 · Big Data Zone · Opinion
Like (3)
Save
Tweet
6.01K Views

Join the DZone community and get the full member experience.

Join For Free

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools I've gathered up below:

Livy is an open sourceREST interface for interacting with Spark from anywhere and used by Apache Zeppelin and other tools. Using Livy with Zeppelin is useful documentation to install and operate Livy with Apache Zeppelin.

I found a lot of interesting examples, tutorials, starter code and samples using Scala for Big Data:

Scala with MQTT IoT on Raspberry Pi Example (GitHub) and if you don't know about IoT, take a Free IoT Class.

Akka HTTP for scalable HTTP microservices and RESTful JSON web services. Akka-HTTP Microservice with Websockets (GitHub) is a nice tutorial and article for using Akka-HTTP with Websockets.  An alternative to Akka HTTP is Spray Demo for Scala REST API Development, Spark-Spray-Akka-Couchbase (GitHub) and Spark-Spray Starter (GitHub) are good examples of Spray. It's still debated if Spray will be deprecated and moved to Akka HTTP. It has not happened yet, not sure it ever will. Spray is rock solid and enterprise scale so it's still a worthy investment for development.

I found three great Github repos for learning Spark:

  • An Example of Spark Streaming with Twitter
  • Sparkathon Code
  • Introduction to Spark Streaming (In-Depth Tutorial)
  • And Annotated Spark Streaming Code

For those trying to determine what streaming framework to use, check out this Article on All Forms of Streaming. Once you get your data from streaming, you will want to access and display it, an example Web application in Scala, Play Example App (Scala/SBT). Scalable Probabilistic Machine Learning in Java which works with Apache Flink, is one of the best choices for Streaming in Java.  I found a nice article and code example of combining Kafka, Spark Streaming and Stanford Core NLP:  Kafka + Spark Streaming + Stanford NLP (GitHub).  

Many Big Data applications are console applications, this is a nice introduction to writing Modern Scala Console Apps. That article will give you a good start.   Finally a quick look at an interesting deep learning framework: Keras: Deep Learning library for TensorFlow and Theano is a minimalist, highly modular neural networks library (GitHub).

Big data Scala (programming language) Framework

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • SQL CTE: How to Master It in One Sitting With Easy Examples
  • How BDD Works Well With EDA
  • Terraform Controller: Cloud Resource Self-Service
  • How to Modify Java Command-Line Arguments

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo