DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Coding
  3. Languages
  4. Why Java in Big Data? What about Scala?

Why Java in Big Data? What about Scala?

Here's what to keep in mind when comparing and determining what language to use with big data applications and data access.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 10, 16 · Opinion
Like (16)
Save
Tweet
Share
16.04K Views

Join the DZone community and get the full member experience.

Join For Free

Why Java? Why not? What about Scala? Or Python. I use all three for various parts of Big Data projects. Use the best tool for the job. A lot of things can be orchestrated and managed without any coding through Apache NiFi 1.0. Some things like TensorFlow are best done in Python, while Spark and Flink jobs could be Scala, Python, or Java. Apache Beam is Java only (Spotify added a Scala interface, but it's not official yet. If you are a really strong Java 8 developer and code clean, you can write Hadoop Map Reduce, Kafka, Spark, Flink, Apex. Apache NiFi is written in Java and so is most of Hadoop, so it's Big Data scale. Spark and others are written mostly in Scala.

Ecosystem

Scala and Java share a ton of libraries, as they run on the JVM. Python has its own huge ecosystem, but for many Hadoop things the JVM languages have a bit of an advantage. You can run JPython on the JVM, but I really haven't seen that used for Big Data, Spark, or Machine Learning. I am wondering if anyone is doing this? Please comment here. Python has TensorFlow and some nice Deep Learning and Machine Learning libraries. They are also starting to get more Universities teaching Python instead of Java. Not too many Universities are teaching Scala.

Tools

Scala and Java, for the most part, share IDEs (Eclipse, NetBean, and IntelliJ being the main ones). Java support is older and a bit more robust, but they are pretty even. Scala in IntelliJ is very nice. Apache Zeppelin supports Scala and Python, but no Java. For that point alone, I am preferring Scala at this point for most of my Big Data engineering and data exploration.

Ease of Development, Readability, Verbosity

Java and Python are very easy to develop and read the code. Java is very verbose and wordy, which is either a big pro or a big con depending on your opinion. Python is easy for administrators, cloud, DevOps, and console-oriented engineers. Java and Scala require the JDK, a build tool, libraries, and some heavy setup. Python needs Python 2.7/3 installed and PiP, these are often there on most Linux servers and AWS.

Libraries, Frameworks, GitHub Examples, and Packages

Check for yourself, but there's a ton for all three (Python. Scala. Java.)

Build and Dependency Tools

Maven, SBT, Gradle along with auto install dependencies are all there for Java and Scala. Python generally just needs PiP. It does take a while to build a JAR from Java and Scala applications with any number of libraries (which you always need a few dozen at least). But the running application has always been faster in my usage. So build time vs run time, really depends on how many times you are running the program and are you reusing it.

Java (programming language) Big data Scala (programming language) Python (language)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Deploying Java Serverless Functions as AWS Lambda
  • Spring Cloud: How To Deal With Microservice Configuration (Part 1)
  • What Is Testing as a Service?
  • How Observability Is Redefining Developer Roles

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: