In the Java EE umbrella every piece of technology is standardized under a JSR (Java Specification Request). The Expert Groups of the JSRs have to deliver the specification, a Reference Implementation, and a TCK (Technology Compatibility Kit). For most of the JSRs, the TCK is licensed as closed-source software and is not available for the public.
That has the implication that any vendor who wants to implement a specific technology has to explicitly apply/ask for/buy the TCK in order to test their implementation and get officially certified. The problem with this situation is clearly that it raises the barrier for potential vendors. Or, in other words, it would be a benefit for vendors to more easily join in the game. Having that said it would then also be more favorable for users of Java EE to have more competition among implementations.
If we follow the thought experiment of having solely OSS in the JSRs, another big change could be how users and developers interact with vendors. If somebody finds a bug in an implementation, then they could not only report the bug to the (certified) vendor but also report (and contribute!) to the TCK directly—as the tests obviously don’t cover that specific case for certified implementations. Then the providers would be notified about the change and some potential maintenance release of the TCKs could be delivered from time to time to check whether implementations are still certified. As a result, we all could ensure better software quality as anybody would be able to improve the tests of what is standardized in the specs.
Unfortunately only very few JSRs are fully open source currently, e.g. Bean Validation.
Other thoughts? Feedback is very welcome.