Why Java in Big Data? What about Scala?
Why Java in Big Data? What about Scala?
Here's what to keep in mind when comparing and determining what language to use with big data applications and data access.
Join the DZone community and get the full member experience.
Join For FreeVerify, standardize, and correct the Big 4 + more– name, email, phone and global addresses – try our Data Quality APIs now at Melissa Developer Portal!
Why Java? Why not? What about Scala? Or Python. I use all three for various parts of Big Data projects. Use the best tool for the job. A lot of things can be orchestrated and managed without any coding through Apache NiFi 1.0. Some things like TensorFlow are best done in Python, while Spark and Flink jobs could be Scala, Python, or Java. Apache Beam is Java only (Spotify added a Scala interface, but it's not official yet. If you are a really strong Java 8 developer and code clean, you can write Hadoop Map Reduce, Kafka, Spark, Flink, Apex. Apache NiFi is written in Java and so is most of Hadoop, so it's Big Data scale. Spark and others are written mostly in Scala.
Ecosystem
Scala and Java share a ton of libraries, as they run on the JVM. Python has its own huge ecosystem, but for many Hadoop things the JVM languages have a bit of an advantage. You can run JPython on the JVM, but I really haven't seen that used for Big Data, Spark, or Machine Learning. I am wondering if anyone is doing this? Please comment here. Python has TensorFlow and some nice Deep Learning and Machine Learning libraries. They are also starting to get more Universities teaching Python instead of Java. Not too many Universities are teaching Scala.
Tools
Scala and Java, for the most part, share IDEs (Eclipse, NetBean, and IntelliJ being the main ones). Java support is older and a bit more robust, but they are pretty even. Scala in IntelliJ is very nice. Apache Zeppelin supports Scala and Python, but no Java. For that point alone, I am preferring Scala at this point for most of my Big Data engineering and data exploration.
Ease of Development, Readability, Verbosity
Java and Python are very easy to develop and read the code. Java is very verbose and wordy, which is either a big pro or a big con depending on your opinion. Python is easy for administrators, cloud, DevOps, and console-oriented engineers. Java and Scala require the JDK, a build tool, libraries, and some heavy setup. Python needs Python 2.7/3 installed and PiP, these are often there on most Linux servers and AWS.
Libraries, Frameworks, GitHub Examples, and Packages
Check for yourself, but there's a ton for all three (Python. Scala. Java.)
Build and Dependency Tools
Maven, SBT, Gradle along with auto install dependencies are all there for Java and Scala. Python generally just needs PiP. It does take a while to build a JAR from Java and Scala applications with any number of libraries (which you always need a few dozen at least). But the running application has always been faster in my usage. So build time vs run time, really depends on how many times you are running the program and are you reusing it.
Developers! Quickly and easily gain access to the tools and information you need! Explore, test and combine our data quality APIs at Melissa Developer Portal – home to tools that save time and boost revenue. Our APIs verify, standardize, and correct the Big 4 + more – name, email, phone and global addresses – to ensure accurate delivery, prevent blacklisting and identify risks in real-time.
Like This Article? Read More From DZone
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}