The Rise and Fall of Scala

Five years ago, Scala seemed like the next big thing in programming languages because it elegantly enabled functional programming within an object-oriented paradigm. Today, Scala’s popularity seems to be fading, with companies like LinkedIn and Yammer moving away from it. The TIOBE index (www.tiobe.com) of software language popularity ranked Scala at #13 in 2012; now it’s fallen to #32 in August 2016, being used by less than .6% of the programming community.

Here’s another ominous sign for Scala: Lightbend, its parent company, is now releasing new frameworks with a Java API before the Scala version. Anecdotally, as CTO of a leading software product engineering company, I meet many software development managers, and I know of at least two who have made the painful decision to abandon Scala after more than a year of adoption. What happened? What gave Scala its initial popularity boost, and what caused its decline? Are there any use cases for which Scala is still the best choice?

To understand the initial popularity of Scala, we must first understand the evolution of modern programming paradigms. First, there was procedural programming, where programs were viewed as a series of statements that should be executed one after another. Then came object-oriented programming, where programs were viewed as actors that knew how to perform operations on objects and converse with each other in order to accomplish tasks.

Functional programming, by contrast, views a program as a mathematical function which is evaluated to produce a result value. That function may call upon nested functions, which in turn may call upon more nested functions. A nested function evaluates to produce a result. From there, that result is passed on to the enclosing function, which uses the nested function values to calculate its own return value. To enable functions to easily pass data to and from other functions, functional programming languages typically define data structures in the most generic possible way, as a collection of (any) things. They also allow functions to be passed to other functions as if they were data parameters. A function in this paradigm is not allowed to produce any side effects such as modifying a global variable that maintains state information. Instead, it is only allowed to receive parameters and perform some operations on them in order to produce its return value. Executing a functional program involves evaluating the outermost function, which in turn causes evaluation of all the nested functions, recursively down to the most basic functions that have no nested functions.

Why is functional programming a big deal?

• Clarity: Programming without side effects creates code that is easier to follow – a function is completely described by what goes in and what comes out. A function that produces the right answer today will produce the right answer tomorrow. This creates code that is easier to debug, easier to test, and easier to re-use.
• Brevity: In functional languages, data is implicitly passed from a nested function to its parent function, via a general-purpose collection data type. This makes functional programs much more compact than those of other paradigms, which require substantial “housekeeping” code to pass data from one function to the next.
• Efficiency: Because functions do not have side effects, operations can be re-ordered or performed in parallel in order to optimize performance, or can be skipped entirely if their result is not used by any other function.

Functional programming languages have existed for decades, beginning with John McCarthy’s LISP language created at MIT in the 1950s. However, these were always viewed as niche languages, of interest mainly to academicians and theoreticians. Scala also began as an academic project, created in 2001 at the Ecole Polytechnique Federale de Lausanne by Martin Odersky. However, Scala designers made several critical decisions that positioned Scala to be the breakthrough language that would bring functional programming to the mainstream:

• Scala combines functional programming with object-oriented programming. As a multi-paradigm language, it can serve as a bridge for object-oriented programmers entering the world of functional programming.
• Scala code runs in the Java Virtual Machine (JVM). This means it can be readily deployed on any machine that runs Java (around 85% of PC’s). It also means Scala code can theoretically interoperate with Java code, providing a bridge for a Java development team to ease into Scala.
• Scala is syntactically similar to Java, and, like Java, performs type-checking at compilation time rather than at runtime, thus eliminating the possibility of runtime errors caused by type incompatibility. These similarities reduce the initial learning curve for Java programmers.
• Scala has built-in support for pattern matching, where arbitrary data types can be matched based on value patterns, in order to perform distinct operations on each matched pattern.
• Scala includes Akka as a standard library, enabling support for rich concurrency models. This makes it easy for programmers to implement complex creation or processing of streaming data as a graph of actors, where each actor processes data in parallel.

It’s no wonder then that Scala was initially greeted with such enthusiasm and was viewed as the language that would bring functional programming to the mainstream. Yet in the words of William H. Calvin, “You can always spot the pioneers by the arrows in their backs.” Scala was surely a pioneer in popularizing functional programming. So why now has the tide turned on Scala, to the point where today its developer base is steadily shrinking?

• The Java programming language introduced functional programming constructs beginning with Java 8, released in early 2014. There are subtle differences in the ways Scala and Java support functional programming, and the argument can be made that Scala’s approach is superior. But, Java has surpassed Scala as the preeminent functional programming language, because programmers already know Java. This is reminiscent of the arc of Adobe Flex and Microsoft Silverlight, which had a sizable following among Web UI programmers, until the release of HTML 5, which provided enough Web UI features to become the dominant technology.
• Scala is a difficult language to master because its principles are based on mathematical type theory, which is fully understood by only the most academic and mathematically minded programmers. Furthermore, many language features of Scala, including implicits and macros, may cause the control of the program to flow unexpectedly to other parts of the code base, which makes it difficult for most programmers to follow or debug their code. An excellent programmer who comprehends all this will be more productive in Scala than in Java, but an average programmer’s productivity, as measured by implemented functionality, will probably decline when transitioning from Java to Scala. This is not just a short-term decline due to a learning curve — the decline has been observed in some development teams a full year after adopting Scala.
• Unlike Java, Scala has a flexible syntax, and typically offers many ways to achieve the same end result. Rather than making Scala more usable for average programmers, the Scala community seems to spend a lot of time arguing about which of several functionally equivalent solutions is the right one. These debates generate more heat than light, and prevent the emergence of tried-and-true implementation patterns that exist in other, more restrictive languages like Java.
• Scala has not done a good job of maintaining compatibility, either with earlier versions of Scala or with Java.

Because of these issues, Scala will likely never evolve into a mainstream programming language like Java. However, there are still specific use cases where Scala is such an ideal fit that it should be the programming language of choice:

• Big Data manipulation: Scala’s strengths are highly suited to the Big Data programming model, where a task takes an immutable collection as input, transforms the collection with map and reduce operations, and generates a new result collection. For Big Data tools such as Spark, the advantages of using Scala are overwhelming, given that a Scala version of a program will usually be five-10 times shorter than the equivalent Java program.
• Creating domain-specific languages: Many problems are best solved by giving users a domain-specific language (DSL) that they can use to write scripts. For example, suppose your users are asking for a tool that will enable them to schedule and run automated QA tests. Since you cannot predict all the combinations of tests they will need to run, you need to give them a scripting language where they can define the order and location of these tests for any scenario. Scala is uniquely suited for development of DSLs, thanks to features such as pattern matching, syntactic flexibility, and operator overloading.

To summarize: Scala played a key role as a catalyst in popularizing functional programming, and it exerted a strong influence on the design of functional programming in Java. Scala will probably never become the next big programming language. But, it will be around for years to come as the language of choice for niche problem domains such as Big Data programming.