On static compilation of Groovy

I want to talk today about two weakness of Groovy and possible solutions for these problems.

The following "manifesto" starts the main page of the Groovy web site. Please read it:

Groovy...

• is an agile and dynamic language for the Java Virtual Machine

• builds upon the strengths of Java but has additional power features inspired by languages like Python, Ruby and Smalltalk

• makes modern programming features available to Java developers with almost-zero learning curve

• supports Domain-Specific Languages and other compact syntax so your code becomes easy to read and maintain

• makes writing shell and build scripts easy with its powerful processing primitives, OO abilities and an Ant DSL

• increases developer productivity by reducing scaffolding code when developing web, GUI, database or console applications

• simplifies testing by supporting unit testing and mocking out-of-the-box

• seamlessly integrates with all existing Java objects and libraries

• compiles straight to Java bytecode so you can use it anywhere you can use Java

All that is true. I can put my signature on each and every word here and as one of the Groovy Core developers and the Co-Founder of the first ever Groovy company G2One, Inc. (now part of SpringSource and a division of VmWare), I put a lot of effort into making it happen.

What this manifesto doesn't say, and is very important to note, is, as all Groovy developers know, there is a price to pay

• there is no compile time check
• there is 5 to 15 times performance slowdown compared to the code written in Java or Scala

In general, the equation we have is:

Someone will probably argue that this is not a problem, that the productivity gain which comes out of using Groovy compensates for the disadvantages above, that Groovy became much faster compared to what it used to be, that you can use Java for critical parts of your code, etc.

I don't want to go into this discussion (especially as the one responsible for a big part of performance gain). I think it is almost counter-productive. I only want to mention that, for example, due to performance reasons Groovy is almost useless today for multi-core programming (even with brillian GPars library - disclosure: I am one of developers of GPars).

What I want ot talk about is:

These very important questions have been asked many times in the Groovy community. Though I am not sure if their answers, which were also verbalized several times by several people, were heard. So, now we come to maybe the most interesting part of the story.

Why is Groovy slow and unable to be staticly compiled. Well, because every call is dynamically dispatched.

Ok, totally understandable... But why? Because the beautiful runtime metaprogramming capabilities of Groovy can change behavior of any method (formally speaking, every call on on every call site) Technically, it means that if we found (and knew it for sure) that at some point we call existing method of a class with rightly typed arguments we still can not use bytecode calling this method and even can't be sure that returned value will be of expected type.

Think about it for a second:

Runtime metaprogramming requires dynamic dispatch

• Dynamic dispatch does not allow compile time checks

• Dynamic dispatch is slow

All together, we pay a huge price for only one (a very important one) feature of the language. But still... for only one feature.

Imagine for a second that we don't use run-time metaprogramming in some piece of code and we have a keyword or annotation to mark such a piece (method or class or whole module or file extension) to be statically compiled. Does it solve our problem?

I claim it does:

• It is possible to write staticly typed compiler for Groovy keeping all goodies of the beutiful language

• It is possible to implement very powerful type inference of local variable and closures, so we don't need to provide verbose type information more than necessary and/or useful for documentation purposes
  

• It is possibly to have compile-time metagrogramming (properies, categories, default groovy methods, mixins, ast transformations - all but runtime meta-programmings)

• It is possible to support Scala-like traits (also knows as interfaces with default implementation), which is extremly strong meta-programming tool
  

• It is even possibly to compile piece of code in so called mixed mode (resolve and call statically what we can and call the rest dynamically). This mode is great for example for mixing computation with building markup or populating UI elements with results of computation

• It is even possibly that because of type inference staticly compiled code can be a little bit less verbose compare to Groovy (almost no need in 'as' conversion for example)
  

 
There are several reasons I claim so:

• It is not rocket science
• Scala proved solution for type inference part
• There is some work in progress proving each and every of claims above

Let me share with you some  snippets of commented code with you:

/**
  Trait Function1 defines interface of abstract function with one argument.

  It also additionally provide implementation of several useful methods
  Traits are always statically compiled
*/
@Trait
abstract class Function1<T,R> {

   /** 
      Body of the function
   */
    abstract R apply (T param)

    /**
     Creates another function applying another function provided as argument to result of calculation of this one
    */
    public <R1> Function1<T,R1> andThen (Function1<R,R1> g) {
        /*
            here is type inference comes in to play
            Groovy allows us not to specify return explicitly AND
            compiler knows that return type is Function1 (interface with one abstract method 
            'apply' because the rest methods have default implementation), 
            so it can compile the whole new class and return of instance of this class

            It is very interesting to notice that we don't even need 
            to specify type of parameter 'arg' compiler is smart enough to deduct it

            Another interesting thing to notice is use of [] operator. As we define 
            default implementation of getAt method (Groovy convention for []) we can use it
            And the last but also important note: In pure Groovy we would write {...} as Function1. Static compiler has enough information to allow us to skip 'as'
          */
        { arg -> g[ apply(arg) ] }
    }

   /**
     Creates another function applying this one to result of another function 
     provided as argument
   */
    public <T1> Function1<T1,R> composeWith (Function1<T1,T> g) {
        /*
            very similar to previous method
        */
        { arg -> apply(g [arg] }
    }

    /**
       Provides [] - syntax for Function1
   */
    R getAt (T arg) {
        apply(arg)
    }
}

Now we can try to use this code:

    /*
       Method to convert iterator to another iterator applying given function to every element 
    */ 
    static <T,R> Iterator<R> map (Iterator<T> self, Function1<T,R> op ) {
        [ next: { op[self.next()] }, hasNext: { self.hasNext() }, remove: { self.remove() } ]
    }

This code looks probably too short. It's done intentionally to suggest to the reader to imagine typical Java code to achieve exactly the same result. Here is the same code with comments:

   /*
       Method to conver iterator to another iterator applying given function to every element 
    */ 
    static <T,R> Iterator<R> map (Iterator<T> self, Function1<T,R> op ) {
        /*
            Again as above compiler has enough information to understand that provided map expression should be compiled in to instantiation of a new class
        */
        [
          // no need to specify return types of methods because compiler knows
          // also compiler knows that next () accep no arguments, so correct method will be created
          // op [] again at our service 
          next: { op[self.next()] }, 
          
          hasNext: { self.hasNext() }, 

          remove: { self.remove() } 
       ]
    }

OK, we are almost there. Let us test what we achieved:

        /*
           we don't define type of iter variable
           isn't it obvious that it is Iterator<String>?

           Look:
               [0, 1, 2] is ArrayList<Integer>
               [0, 1, 2].iterator () is Iterator<Integer>
               map {...} called with Function1<Integer,String> (String is return type of provided closure as well as 'it' parameter is Integer)
        */ 
        def iter = [0, 1, 2].iterator().map { (it + 5).toString() }
        def res = []
        // we use normal Groovy Truth but staticly compiled. So iter instead of iter.hasNext ()
        while (iter) {
            // and yes, toUpperCase method is at our service
            res << iter.next ().toUpperCase()
        }
        assertEquals (["0", "1", "2"], res )

Let me stop here. There are a lot more things to say about the unlimited opportunities that static compilation of Groovy opens. I will probably blog about it in the future if my research progresses well.

I strongly believe that Groovy can become the main-stream programming language used both in performance-critical and complicated concurrent applications and utilizing both the power of compile time type checking and dynamic genesis of Groovy. Here is my conclusion: