Enabling Modelling 2.0 With Xtext

With the Eclipse Helios release just around the corner, Xtext will be providing their 1.0 release of their highly successful framework for creating DSLs. I spoke with Sven Efftinge to find out more about this release.

DZone:  First, just in case anyone doesn't know about Xtext, could   you   explain it?

Sven Efftinge: Of course. Xtext is what we call a language development framework. That is an open set of libraries and domain-specific languages (DSLs) used to implement a complete infrastructure for any kind of language, whether it's regular programming languages like Java or DSLs.  It covers all aspects of a compiler or interpreter, such as parsing, validation and linking, and it provides you with a full-blown Eclipse-based IDE for your language.

Xtext is an open-source project developed at Eclipse.org. It is driven by itemis, which is a strategic member at  Eclipse. I have the pleasure to lead a  fantastic team of highly  motivated and skilled developers, and  itemis is funding us to work  almost full time on Xtext.

DZone: So what are the highlight features of Xtext 1.0?

Sven: Since the last release, we've implemented over 80 new features and   fixed several hundreds of bugs. Besides greatly improved  performance  especially within the linker, we have introduced the possibility to bind and   integrate with any JVM language. That means it is a matter of  minutes  to add references to Java elements such as types, fields,  methods and  even annotations. In the IDE this results in a tight integration with  Eclipse's Java Development Tools (JDT) while at  runtime Xtext uses  Java reflection to look up referenced Java  elements. That's pretty  cool and something a lot of users now make  use of.

The most important addition in Xtext 1.0 is the indexing infrastructure. Based on a description of your project structure  Xtext  indexes all contained files and remembers the names and  types of your  elements as well as any cross references. As a  result Xtext now lets  you do namespace based linking of any  element in any file in your  project. In the IDE this information  is used to provide a lot of new  cool features, like a global "Open Model Element" dialog, where you  can search for elements similar  to the "Open Type" dialog in JDT. In fact, by default Xtext will  leverage the Java classpath mechanism to  describe your project  structure. That is you can have your Xtext files  in jars, class  folders or even OSGi bundles and Xtext will reuse the  Java classpath configuration to determine the boundaries of your  project. Anyway, leveraging JDT is just a convenient default, you  can  use the index without JDT.

DZone: What are the limitations of Xtext?

Sven: You can implement almost any kind of programming language or DSL with Xtext. There is one exception, that is if you need to use so  called  'Semantic Predicates' which is a rather complicated thing I  don't think is worth being explained here. Very few languages really need this concept. However the prominent example is C/C++.  We want to look into that topic for the next release.

DZone: I see that Itemis are doing iPhone apps now. Is Xtext a  basis  for  this?

Sven: Yes, partly. We have developed a small domain-specific language to   develop a very common type of data-driven mobile application. If  you  search the different app stores it is surprising how often you  find  this kind of application. With 'Applause', which is the name  of that DSL, you can very easily describe such applications and generate an  iPhone as well as an Android app from it. It's pretty  impressive.

In addition to that we do a lot of different things on modern mobile platforms. Mostly we use this new platform as a great new  client  technology for enterprise software systems. We already did  a couple of cool projects and are currently working on a big one.

DZone: We did a poll recently  and it seems that about 66% of developers who answered,  don't   use  model driven development. Can you convince them otherwise?

Sven: I think the term model-driven is used in very different ways. I myself don't like riding that old horse anymore, because there is too much dogmatic stuff going on with it. Especially the ideas  behind MDA seem very strange and impractical to me and the technologies and languages coming from the OMG are complicated and  also not very well  designed. I never use UML for programming nor  do I use OCL or QVT. I  also try to to avoid the typical modeling  terminology but instead use  simple and existing terminology from  general programming and the language community. I try to focus on  writing good software because  that is what all this should be used for in the end.
What I can say is, that being able to write languages and code generators or interpreters is a very valuable addition to every developer's toolbox.

With Xtext we try to make these tasks as easy and as much fun as   possible. And I think we made some progress here :-) Obviously, as part of Eclipse Modeling (EMP) we cannot and don't  want  to completely omit the term "Modeling", but I think that not  only  Xtext but also a lot of other technologies at EMP are very  helpful and  pragmatically designed technologies. They actually  don't have that much to do with the general perception of  "Modeling". That is why I  call what we do "Modeling 2.0" from time to time. I hope this will  give people a hint, that it doesn't fit  into the old idea of "Modeling".

DZone: So for anyone who is convinced, can you give us an easy  three  step  plan to getting started with modelling?

Sven: You mean Modeling 2.0? :-)

There are two different very cool things you can do with EMF, which  is  the core framework at EMP. You can use EMF in any software  system at  runtime as your domain model technology, similar to how  people use  JavaBeans. EMF is just much nicer, has much richer  semantics and comes with a lot of great additional technology. For instance CDO can really  be seen as  a superior replacement for any  object-relational database  mapper. It is highly scalable, flexible  and gives you the deep semantics of EMF. If you are interested in  that I suggest to watch the webinar on CDO.

Xtext on the other hand uses EMF as the in-memory representation  of  languages. These models are usually called Abstract Syntax  Trees (AST)  but in Xtext they are implemented using EMF. These in- memory EMF  models repesent the main data the different aspects of  a language  infrastructure work with and it is and was a very good  decision to  leverage EMF here. It turned out that the semantics  and features EMF  provides are a fantastic match to what we needed  in Xtext. Also by
implementing EMF API any EMF client can transparently use Xtext  under  the hood. Which for instance allows to combine Xtext with  other  existing EMF-based
tools such as the Graphical Modeling Framework (GMF).

If you want to learn how implementing programming languages feels  like  in 2010, I recommend to have a look at our website, watch the  webinar  or read the introductory examples in the documentation.  You'll be surprised how easy language development can be.


DZone: It seems that a lot of the focus for this release has been to   improve the editor support, such as content assist and other  useful  editor features. Can you give us a rundown of these features? Which  was the most difficult to implement?

Sven: Yes, we have done a lot of UI features. An Xtext language IDE  should  really feel like Eclipse's JDT and the framework should  come with  great default semantics as well as easy to use API for  all the  different aspects. During the last year we vastly improved  content assist, added support for quick fixes, rewrote the whole validation  framework as well as the scoping framework. The editor  now supports  folding, bracket matching, and auto edit. We have  support for semantic  highlighting and quick outline. I'm sure I  missed a couple of UI features, but we have a very nice new &  noteworthy page which not only contains descriptions and screenshots but also  small  screencasts whenever appropriate.

DZone: The index infrastructure seems to have some additions too.  Could  you explain these changes and their benefits?

Sven: The index infrastructure was a tough one. Actually we implemented  it  three times before we got it right. For the last  implementation  Sebastian and I closed any communication tools and  locked up the doors  for three weeks, to get it done. We are very  happy with the result, we  like the core abstractions a lot, and  the framework has become an  important basis for other cool features we have implemented later.

A very hard thing to solve was the computation of the transitive  hull  of all affected resources. Imagine you have a broken  reference to  something called 'MyBookingService' because in the  declaration you misspelled the name ('MyBokingService'). In that  case you of course  get an error marker that the reference can not  be resolved. If you now  open the declaration and fix the typo, we  need to know that the  previously broken reference can be fixed but  of course we cannot rebuild the whole workspace to find any  possible occurrences. So we  remember which elements have asked for  which names and inform them as  soon as such a name has changed. As  a language designer you don't have  to deal with this complex  problem, the index infrastructure solves  this for you.
Another very cool thing is the "Dirty Editor State" support. That   means, that the builder state is shadowed by the state of all  editors  containing unsaved changes. JDT tries to do the same but  it doesn't  work as reliable as in Xtext. :-)

DZone: Also, the performance figures look impressive - linking is 40   times faster, and less builders invoked! How did you get those  improvements?

Sven: Actually we hadn't found the time to do comprehensive profiling  for  the previous 0.7 release. So there were a lot of unused  opportunities  to improve performance. The most compelling  performance gain was done  in the linker phase, which previously  was also the most expensive phase. Now it's on par with parsing  which is very fast. We  accomplished this by introducing a generic  caching utility for EMF  models, which is based on EMF's adapters.  The builder framework was  introduced in M4 / M5. We optimized it  in M6. I can proudly say, that scalability of Xtext 1.0 is in a  pretty good shape.