The Covariant Return Type Abyssal

A few month ago I was working in a customer project where we were building a large smartclient sales-system for a german insurance company. We were using Riena as the application framework, eclipse for developing, and Java 6 as the platform. Besides the usual problems it was quite fun, but one day we stumbled about a problem involved with covariant return types. Eh, what's that? Since Java 1.5 you can use a more specific return type when overriding a method, so you can avoid nasty casts in class hierarchies. By example: let's say, we have a class A with a method getValue() which returns an object of type X:

public class X {
     ...
}

public class A {

     private X value;

     public X getValue() {
         return value;
     }
}

So, if I derive a class B from A, I may override getValue() and use a subclass of X as the return type:

public class Y extends X {
     ...
}

public class B {
...

     @Override
     public Y getValue() {
         return value;
     }
}

That's fine, since Y 'is' an X. So what's the point here?. The problem we stumbled about will hit you, if you use bean-introspection. I don't need no stinkin' bean-introspection! Really? Ever used data-binding? Ah, so you do ;-)

If I use bean-introspection to retrieve the property value from an object of type B, and ask for its type, I will get... X. Say what? But.. it is an Y?!? Yep, but that's what happens. There are some open problems in this area. I never had any problems with generics, so when is that gonna bite you? Let's say Y extends X with a property name:

public class Y extends X {
     private String name;

     public String getName() {...}

     public void setName(String name) {...}
}

And now you use data-binding in conjunction with that property:

B b = ...

ridget.bind(b, "value.name")

Data-Binding uses bean-introspection to retrieve the property value from B, and on the type of value it tries to retrieve the property name. And that's the moment the bug enters the stage: Due to the problem described above, bean-introspection retrieves X as the type of value. But since the property name is not defined in X (but in Y), you will get a funny exception saying there is no such property name :-)

But if I call B.getValue(), I will get a Y. Well, at least the compiler does not complain if I treat it as a Y...and I can call methods defined in Y, like getName(). So is this some kind of compiler magic? Well, there really is a method Y getValue()... but there is also a method X getValue()! If you call getMethods() on class B, you will get:

public X B.getValue()
public Y B.getValue()

What the...? Where does this method X getValue() come from?!? That was generated by the compiler. Ah... :-| But why? That whole topic covariant return types came up with generics... and the type erasure needed for compatibility reasons. Yeah no, is clear. Ok, so we have to dig a lil' deeper here. Let's say our class A would have been defined using generics:

public class A <T extends X> {

     private T value;

     public T getValue() {
         return value;
     }
}

public class B extends A<Y> {}

If you use reflection to get the methods from A and B, they will both have ONE method X getValue(). After all I expected B to have a method Y getValue()? So what's going on if I call getValue() and treat the result as a Y?!? That's all compiler sugar, means: the compiler does all that type checking by evaluating the generics information, and inserts casts in your code where necessary. Look at the following snippet:

     public static void main(String[] args) {
         B b = new B();
         Y y = b.getValue();
     }

At the byte-code level the (only) method X getValue() is called, and the result X is casted to Y:

   public static void main(java.lang.String[] args);
      0  new B [29]
      3  dup
      4  invokespecial B() [31]
      7  astore_1 [b]
      8  aload_1 [b]
      9  invokevirtual B.getValue() : X [174]  // X getValue() called
     12  checkcast Y [133]                     // cast to Y
     15  astore_2 [y]
     16  return

At runtime all generics information has been erased, and only the raw type X is used. This is done for compatibility reasons. It's the same thing if you use collections: If you define a List<String> in your source, the compiler will transform that to an untyped List of Objects. So why was the type of our generic type T in class A compiled to X, and not to Object? That's because we defined a lower bound of our type T:

public class A <T extends X> {
...

The collections like List do not define any lower bound, so the raw type is Object:

public interface List<T> ...

Until now you don't even notice all that stuff; you just use it and it feels quite natural. But things change, if we override getValue() in B:

public class B extends A<Y> {

     @Override
     public Y getValue() {
        ...
     }
}

Now the compiler creates a method Y getValue() as expected, which overrides the code of superclass A. For the Java language everything is fine. But from the JVM's point of view Y getValue() is something different then than X getValue(). So if you assign an object of type B to a reference of type A, and call getValue(), the JVM will search for a method X getValue() which is defined in A:

     B b = new B();
     A ba = b;
     ba.getValue(); // -> JVM wants to invoke X getValue()

So here is a gap. And what's our fancy compiler doing? He corrects this flaw by gerating a synthetic method X getValue():

public class B extends A {
...

   public Y getValue();
     0  aload_0 [this]
     ...


   public bridge synthetic X getValue();
     0  aload_0 [this]
     1  invokevirtual B.getValue() : Y [21]   // delegate to Y getValue()
     4  areturn
       Line numbers:
         [pc: 0, line: 1]

}

That's a bridge method. In this case it's nothing but a delegate to our overridden method Y getValue(). So here we have two methods getValue(). (By the way: 'bridge' and 'synthetic' are both attributes of the method that can be queried using reflection.) This subject gets clearer, if you think about overriding a method with generic parameters:

public class A <T extends X> {

     public void setValue(T value) {
       ....
     }
...
}

public class B extends A<Y> {

     @Override
     public void setValue(Y value) {
        ...
     }
}

So again the compiler correctly creates a method with signature setValue(Y) for the method. Again, this would be a different method than the one defined in A, so a bridge method is inserted to fill the gap:

public class B extends A {
...

   public void setValue(Y value);
     0  aload_0 [this]
     ...

   public bridge synthetic void setValue(X arg0);
     0  aload_0 [this]
     1  aload_1 [arg0]
     2  checkcast Y [21]                         // cast to Y
     5  invokevirtual B.setValue(Y) : void [23]  // call setValue(Y)
     8  return
       Line numbers:
         [pc: 0, line: 1]
}

Usually, you don't have to even think about this stuff, because the compiler manages all this for you, and gives you warnings or even errors if you leave the path of typesafetyness.

But despite of that, there are still some pitfalls waiting. One point is: there is no defined order in the reflection API (e.g. getMethods()) for delivering the methods of a class. Means: depending on when and how the methods are queried, you will get either X getValue() or Y getValue() first! And here we have a JDK bug entering the stage: bean-introspection does not pay attention to this, so if you query the type of property value in class B you will get either X OR Y! We had exactly this situation in a customer project. When we started the application from our IDE, everything was fine. But in the deployed application, we got that data-binding error described above :-/

But there are even more of those funny problems. And not in theory, we got the next problem in the same project. Following scenario: we wanted to ease up certain things in order to get rid of boiler plate code. E.g. instead of program async execution handish using threads, we appreciated an annotation based solution. Means: you mark a method as @Async, and it will be executed asynchronously while the UI event-loop was still running. We used a (Class-)proxy for interception, which was analyzing the annotations and performing the asynchronous execution if necessary. Now let's take our example and mark the overridden method getValue() as @Async:

public class B extends A<Y> {

     @Override
     @Async
     public Y getValue() {
        ...
     }
}

Now the problem. Sometimes it was executed asynchronously, sometimes not :-/ The reason was, that the annotation was sometimes present, and sometimes not?!? If you think about the reason for bridge methods, you may already know the answer. It depends, on HOW you call getValue(). Have a look:

  7:     B b = new B();
  8:     A ba = b;
  9:
 10:     b.getValue();  // Executed asynchronously.
 11:     ba.getValue(); // In this case not :-(

Again the bridge methods are the cause of the problem. If you set a breakpoint in the overridden method Y getValue() in class B, you will get the following execution stack in the debugger for the call b.getValue():

Thread [main] (Suspended)
     B.getValue() line: 7
     BridgeMethodTest.main(String[]) line: 10

On the second call - ba.getValue() - it looks like this:

Thread [main] (Suspended)
     B.getValue() line: 7
     B.getValue() line: 1    // <- bridge-method  X getValue()
     BridgeMethodTest.main(String[]) line: 11

So in the call ba.getValue() we hit the bridge method. Think about it: For the compiler the object ba is an A, and in A the method X getValue() is defined, so that's what the compiler is using -> our bridge method. And since it is a synthetic method, which doesn't have any source-code representation, line: 1 is shown as line number information... which feels quite curious, since there is nothing in line 1 in our source. Ok, but what about the sometimes missing annotation? Again, this is a JDK bug: when the compiler generates the bridge method, it does not copy any annotations from the original method, means: the bridge method is not annotated. So our proxy will not find any annotation in this case and therefore will not execute the call asynchronously :-/

After all, you will hardly get into these traps, so you don't have to care too much about it. But it's quite useful to know these oddities... at least to get the point why the debugger stops in line 1 ;-) By the way, the bean-introspection bug has been fixed in Java 7, but the missing annotations are still a problem. And for sure there will be some more funny problems in this area.

When you look into an abyss,
the abyss also looks into you.
Friedrich Nietzsche

Resources

• http://bugs.sun.com/view_bug.do?bug_id=6788525
• http://bugs.sun.com/view_bug.do?bug_id=6888164
• http://cr.openjdk.java.net/~briangoetz/lambda/Defender%20Methods%20v4.pdf