DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
The Latest "Software Integration: The Intersection of APIs, Microservices, and Cloud-Based Systems" Trend Report
Get the report
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Scala Big Data Frameworks and Tools

Scala Big Data Frameworks and Tools

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools the author has gathered up in this article.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 11, 16 · Opinion
Like (3)
Save
Tweet
Share
6.16K Views

Join the DZone community and get the full member experience.

Join For Free

There are a lot of Scala Big Data Frameworks and tools available for querying, programming, batch, streaming and various data engineering tasks. Check out some tools I've gathered up below:

Livy is an open sourceREST interface for interacting with Spark from anywhere and used by Apache Zeppelin and other tools. Using Livy with Zeppelin is useful documentation to install and operate Livy with Apache Zeppelin.

I found a lot of interesting examples, tutorials, starter code and samples using Scala for Big Data:

Scala with MQTT IoT on Raspberry Pi Example (GitHub) and if you don't know about IoT, take a Free IoT Class.

Akka HTTP for scalable HTTP microservices and RESTful JSON web services. Akka-HTTP Microservice with Websockets (GitHub) is a nice tutorial and article for using Akka-HTTP with Websockets.  An alternative to Akka HTTP is Spray Demo for Scala REST API Development, Spark-Spray-Akka-Couchbase (GitHub) and Spark-Spray Starter (GitHub) are good examples of Spray. It's still debated if Spray will be deprecated and moved to Akka HTTP. It has not happened yet, not sure it ever will. Spray is rock solid and enterprise scale so it's still a worthy investment for development.

I found three great Github repos for learning Spark:

  • An Example of Spark Streaming with Twitter
  • Sparkathon Code
  • Introduction to Spark Streaming (In-Depth Tutorial)
  • And Annotated Spark Streaming Code

For those trying to determine what streaming framework to use, check out this Article on All Forms of Streaming. Once you get your data from streaming, you will want to access and display it, an example Web application in Scala, Play Example App (Scala/SBT). Scalable Probabilistic Machine Learning in Java which works with Apache Flink, is one of the best choices for Streaming in Java.  I found a nice article and code example of combining Kafka, Spark Streaming and Stanford Core NLP:  Kafka + Spark Streaming + Stanford NLP (GitHub).  

Many Big Data applications are console applications, this is a nice introduction to writing Modern Scala Console Apps. That article will give you a good start.   Finally a quick look at an interesting deep learning framework: Keras: Deep Learning library for TensorFlow and Theano is a minimalist, highly modular neural networks library (GitHub).

Big data Scala (programming language) Framework

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Data Stream Using Apache Kafka and Camel Application
  • Master Spring Boot 3 With GraalVM Native Image
  • Kubernetes-Native Development With Quarkus and Eclipse JKube
  • 10 Most Popular Frameworks for Building RESTful APIs

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: