Why Java in Big Data? What about Scala?

Why Java? Why not? What about Scala? Or Python. I use all three for various parts of Big Data projects. Use the best tool for the job. A lot of things can be orchestrated and managed without any coding through Apache NiFi 1.0. Some things like TensorFlow are best done in Python, while Spark and Flink jobs could be Scala, Python, or Java. Apache Beam is Java only (Spotify added a Scala interface, but it's not official yet. If you are a really strong Java 8 developer and code clean, you can write Hadoop Map Reduce, Kafka, Spark, Flink, Apex. Apache NiFi is written in Java and so is most of Hadoop, so it's Big Data scale. Spark and others are written mostly in Scala.

Ecosystem

Scala and Java share a ton of libraries, as they run on the JVM. Python has its own huge ecosystem, but for many Hadoop things the JVM languages have a bit of an advantage. You can run JPython on the JVM, but I really haven't seen that used for Big Data, Spark, or Machine Learning. I am wondering if anyone is doing this? Please comment here. Python has TensorFlow and some nice Deep Learning and Machine Learning libraries. They are also starting to get more Universities teaching Python instead of Java. Not too many Universities are teaching Scala.

Tools

Scala and Java, for the most part, share IDEs (Eclipse, NetBean, and IntelliJ being the main ones). Java support is older and a bit more robust, but they are pretty even. Scala in IntelliJ is very nice. Apache Zeppelin supports Scala and Python, but no Java. For that point alone, I am preferring Scala at this point for most of my Big Data engineering and data exploration.

Ease of Development, Readability, Verbosity

Java and Python are very easy to develop and read the code. Java is very verbose and wordy, which is either a big pro or a big con depending on your opinion. Python is easy for administrators, cloud, DevOps, and console-oriented engineers. Java and Scala require the JDK, a build tool, libraries, and some heavy setup. Python needs Python 2.7/3 installed and PiP, these are often there on most Linux servers and AWS.

Libraries, Frameworks, GitHub Examples, and Packages

Check for yourself, but there's a ton for all three (Python. Scala. Java.)

Build and Dependency Tools

Maven, SBT, Gradle along with auto install dependencies are all there for Java and Scala. Python generally just needs PiP. It does take a while to build a JAR from Java and Scala applications with any number of libraries (which you always need a few dozen at least). But the running application has always been faster in my usage. So build time vs run time, really depends on how many times you are running the program and are you reusing it.