Recently I attended a local rheinJUG meeting in Düsseldorf. While the topic of the session was Eclipse e4, the night’s sponsor itemis provided some handouts on Xtext which got me very interested. The reason is that currently at work we are developing a mobile Java application (J9, CDC/Foundation 1.1 on Windows CE6) for which we needed an easy to use and reliable way for configuring navigation through the application.

In a previous iteration we had – mostly because of time constraints – hard coded most of the navigational paths, but this time the app is more complex and doing that again was not really an option. First we thought about an XML based configuration, but this seemed to be a hassle to write (and read) and also would mean we would have to pay the price of parsing it on every application startup.

Enter Xtext: An Eclipse based framework/library for building text based DSLs. In short, you just provide a grammar description of a new DSL to suit your needs and with – literally – just a few mouse clicks you are provided with a content-assist, syntax-highlight, outline-view-enabled Eclipse editor and optionally a code generator based on that language.

Getting started: Sample Grammar

There is a nice tutorial provided as part of the Xtext documentation, but I believe it might be beneficial to provide another example of how to put a DSL to good use. I will not go into every step in great detail, because setting up Xtext is Eclipse 3.6 Helios is just a matter of putting an Update Site URL in, and the New Project wizard provided makes the initial setup a snap. I assume, you have already set up Eclipse and Xtext and created a new Xtext project including a generator project (activate the corresponding checkbox when going through the wizard). In this post I am assuming a project name of com.danielschneller.navi.dsl and a file extension of .navi.

When finished we will have the infrastructure ready for editing, parsing and generating code based on files like these:

navigation rules for MyApplication

mappings {
 map permission AdminPermission to "privAdmin"
 map permission DataAccessPermission to "privData"

 map coordinate Login to "com.danielschneller.myapp.gui.login.LoginController" in "com.danielschneller.myapp.login"
 map coordinate LoginFailed to "com.danielschneller.myapp.gui.login.LoginFailedController" in "com.danielschneller.myapp.login"
 
 map coordinate MainMenu to "com.danielschneller.myapp.gui.menu.MainMenuController" in "com.danielschneller.myapp.menu"

 map coordinate UserAdministration to "com.danielschneller.myapp.gui.admin.UserAdminController" in "com.danielschneller.myapp.admin"
 map coordinate DataLookup to "com.danielschneller.myapp.gui.lookup.LookupController" in "com.danielschneller.myapp.lookup"
}


navigations {
 define navigation USER_LOGON_FAILED
 define navigation USER_LOGON_SUCCESS
 define navigation OK
 define navigation BACK
 define navigation ADMIN
 define navigation DATA_LOOKUP
}


navrules {
 from Login
  on navigation USER_LOGON_FAILED
   go to LoginFailed
  on navigation USER_LOGON_SUCCESS
   go to MainMenu

 from LoginFailed
  on navigation OK
   go to Login

 from MainMenu
  on navigation ADMIN
   go to UserAdministration
   with AdminPermission
  on navigation DATA_LOOKUP
   go to DataLookup
   with DataAccessPermission
  
 from UserAdministration
  on navigation BACK
   go to MainMenu

 from DataLookup
  on navigation BACK
   go to MainMenu


}

As you can see it is a nice little language for defining coordinates in an application, meaning a specific GUI for a certain task and the possible navigation paths between them. Optionally a navigation path can be tagged to require one or more permissions to work. So for example one possible navigation path shown in the above sample is from the applications main menu, identified by the identifier MainMenu and represented in code by the com.danielschneller.myapp.gui.menu.MainMenuController class in the com.danielschneller.myapp.menu OSGi bundle to a GUI identified as DataLookup, implemented by com.danielschneller.myapp.gui.lookup.LookupController in the com.danielschneller.myapp.lookup bundle.

For this path to be taken, the application must request the DataLookup navigation path and the currently logged in user be assigned the DataAccessPermission. What exactly that means is not the focus of this tutorial, suffice it to say that we somehow need to get the information contained in this specialized language into our Java application in some shape or form that can be evaluated at runtime. In the following example all information will be transformed into a HashMap based data structure. For our little mobile application this has several advantages over the XML option mentioned earlier:

• No XML parsing necessary on application startup, saving some performance
• Validation of the navigation rules ahead of time, preventing parse errors at runtime
• No libraries needed to access the information – by putting everything in a simple HashMap we do not have to rely on any non-standard classes whatsoever

First thing I did when I started with Xtext was define a sample input file such as the one above. Then – following its general structure – I began to extract a formal grammar for it. Of course, the first draft of the sample data was not perfect, over the course of a few iterations I refined some of the syntax, but in the end this is the grammar definition I came up with. It is heavily commented to allow you to copy it out and still leave the documentation intact:

grammar com.danielschneller.navi.NavigationRules with org.eclipse.xtext.common.Terminals

generate navigationRules "http://com.danielschneller/fw/funkmde/navi/NavigationRules"

/*
 * The top level entry point for the file.
 * "Root" is just a name as good as any, but
 * makes the meaning quite clear.
 */
Root:
    // first thing in the file is a "keyword",
    // followed by an attribute that will be
    // accessible as "name" later and allow
    // definition of an ID type of thing.
 'navigation rules for' name=ID
 
 // after the keyword and "name" attribute
 // three sections follow, each assigned
 // to an attribute for later reference
 // (called "mappingdefs", "transitiondefs"
 // and "ruledefs").
 // Their types are defined later in the file.
 mappingsdefs=Mappings
 transitiondefs=TransitionDefinitions
 ruledefs=NavigationRules
// semicolon ends the definition of "Root"
;



// mappings section >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/*
 * Definition of the "Mappings" type used in
 * the "Root" type. 
 */
Mappings:
    // first the keyword "mappings" is expected,
    // then an open curly
 'mappings' '{'
 // after that a collection of "Mapping"s is
 // expected. The "+=" means that they will
 // all be collected in a collection type element
 // called "mappings" for future reference.
 // The "+" at the end means "at least one, but
 // more is just fine".
 (mappings+=Mapping)+
 // finally the "Mappings" type requires a closing
 // curly brace.
 '}'
// semicolon ends the definition of "Mappings"
;

/*
 * Definition of a single "Mapping", those we are
 * collecting in the "mappings" attribute of the
 * "Mappings" type.
 */
Mapping:
    // each mapping starts with the keyword "map"
    // and is followed by an element of type "MappingSpec"
 'map' MappingSpec
;

/*
 * Definition of a "MappingSpec" element. This is
 * actually just a "parent type" for two more specific
 * kinds of "MappingSpec":
 */
MappingSpec:
    // no keywords are defined here, a "MappingSpec"
    // can be either a "PermissionMappingSpec" or a
    // "CoordinateMappingSpec". Any of these will be
    // fine where a "MappingSpec" is asked for.
 PermissionMappingSpec | CoordinateMappingSpec
;

/*
 * Definition of a "PermissionMappingSpec" element.
 */
PermissionMappingSpec:
    // first the keyword "permission" is required.
    // then a "name" attribute is expected of type ID.
    // Following the name the "to" keyword is expected,
    // followed by a string that is stored in the "value"
    // attribute
 'permission' name=ID 'to' value=STRING
;

/*
 * Definition of a "CoordinateMappingSpec" element.
 * The definition is very similar to the "PermissionMappingSpec"
 * but has more attributes.
 */
CoordinateMappingSpec:
    // first the keyword "coordinate", then an ID stored as "name",
    // the keyword "to", followed by a string stored as "controllername",
    // next the keyword "in" and finally another string, memorized as
    // "bundleid"
 'coordinate' name=ID 'to' controllername=STRING 'in' bundleid=STRING
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< mappings section



// >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navigations section
/*
 * Definition of the "TransitionDefinitions" type used in
 * the "Root" type. 
 */
TransitionDefinitions:
    // first, this element is introduced with the "navigations"
    // keyword, followed by an open curly brace.
 'navigations' '{'
    // after that a collection of "TransitionDefinition"s is
 // expected. The "+=" means that they will
 // all be collected in a collection type element
 // called "transitions" for future reference.
 // The "+" at the end means "at least one, but
 // more is just fine".
 (transitions+=TransitionDefinition)+ 
 // the element ends with a closing curly brace
 '}'
;

/*
 * Definition of a "TransitionDefinition" element. This
 * one is very simple.
 */
TransitionDefinition:
    // the keyword "define navigation" is required first,
    // then a "name" attribute of type ID is expected.
 'define navigation' name=ID
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navigations section


// >>>>>>>>>>>>>>>>>>>>>>>>>>>>> navrules section
/*
 * Definition of the "NavigationRules" element.
 */
NavigationRules:
    // Element starts with the keywords "navrules" and
    // open curly.
 'navrules' '{'
 // collection attribute called "rules", consisting
 // of one or more occurrences of a "Rule" element.
 (rules+=Rule)+
 // element finishes with a closing curly keyword
 '}'
;


/*
 * Definition of a "Rule" element as used in the "NavigationRules"
 * element.
 */
Rule:
    // first the "from" keyword, then a reference  to one of the
    // coordinate mappings defined earlier. This time no new
    // definition of a coordinate is required, but one of those
    // that have been listed before. So the type here is put in
    // square brackets
 'from' source=[CoordinateMappingSpec]
 // following the source specification, one or more "Destination"
 // type elements are expected, collected in a collection attribute
 // named "destinations"
 (destinations+=Destination)+
;

/*
 * Definition of a "Destination" type. These are collected
 * in a "Rule".
 */
Destination:
    // first comes an "on navigation" keyword. After that a
    // reference to one of the Transition elements defined
    // in the "navigations" section is required and stored
    // in the "transition" attribute.
    // after that follows a "go to" keyword and a reference
    // to a coordinate mapping, stored in the "target" attribute.
    // finally - as with the "destinations" collection attribute
    // in the "Rule" element - a "permissions" collection is
    // defined to store none or more (*) "PermissionReference"
    // elements.
 'on navigation' transition=[TransitionDefinition]
 'go to' target=[CoordinateMappingSpec]
 (permissions+=PermissionReference)*
;

/*
 * Definition of a "PermissionReference" type. This is used
 * in the "permissions" collection of a "Destination".
 */
PermissionReference:
    // first, a "with" keyword is expected. After that a
    // "permission" attribute stores a reference to one of
    // the previously defined permission mappings from the
    // "mappings" section.
 'with' permission=[PermissionMappingSpec]
;
// <<<<<<<<<<<<<<<<<<<<<<<<<<<<< navrules section

This is what XText can digest and create an editor plugin and outline view for. Just save this as navigationRules.xtext – when you created the XText project in Eclipse using the wizard it should have been prepared for you.

Copying and pasting this into a .xtext file in Eclipse will provide you with syntax highlighting, code completion and syntax checking, making it easy to play around with grammar files.

Once done, right click the .mwe2 file lying next to the grammar file in the Package Explorer view and select Run As MWE2 Workflow from the context menu. This will take a moment and generate several classes, both in the current (XText) project and the accompanying ...ui project.

Next, right click the Xtext project and select Run As Eclipse Application from the context menu. This will bring up another Eclipse instance with the newly created support for navigation rules files (with a .navi suffix) installed.

To try it out, just create a new project and in that a new file. Make sure its name ends in .navi. When asked, make sure to accept adding the Xtext nature to the project. You will be presented with a new, empty editor that already has an error marker in it. This is because according to our grammar definition, an empty file does not comply to all the rules we specified. Try hitting the code-completion shortcut (Ctrl-Space) twice and see what happens:

The first code-completion fills in the navigation rules for part. According to the grammar this is the only valid text at the beginning of a file, so it is automatically inserted. Hitting Ctrl-Space again will tell you that now you need a Name of type ID. Just go ahead and try out the completion. It will help you create a syntactically sound navigation rules file. Notice that the Problems View tells you what is currently wrong. Also notice, that one you reach a part where references are expected by the grammar (e. g. when defining source and destination coordinates in a navigation rule) you will get suggestions based on what you entered earlier.

This is what the whole sample from above looks like in the editor:

While you are still fleshing out and fine tuning your grammar definitions, you will probably close this Eclipse instance and reopen it, once you repeated the Run As MWE2 Workflow steps in the main instance. In the long run I suggest you create a Feature and an Update Site project to allow easier distribution and updates of the intermediate iterations.

Generating Code

This is a sample of what the generated output is going to look like:

public class NaviRules {

    private Map navigationRules = new Hashtable();

    // ... 

    public NaviRules() {
        NaviDestination naviDest;
        // ========== From Login (com.danielschneller.myapp.gui.login.LoginController)
        // ========== On   USER_LOGON_FAILED
        // ========== To   LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController in com.danielschneller.myapp.login) 
        naviDest = new NaviDestination();
        naviDest.action = "USER_LOGON_FAILED";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginFailedController";
        naviDest.targetBundleId = "com.danielschneller.myapp.login";
        
        store("com.danielschneller.myapp.gui.login.LoginController", naviDest);
        
        // ========== On   USER_LOGON_SUCCESS
        // ========== To   MainMenu (com.danielschneller.myapp.gui.menu.MainMenuController in com.danielschneller.myapp.menu) 
        naviDest = new NaviDestination();
        naviDest.action = "USER_LOGON_SUCCESS";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.menu.MainMenuController";
        naviDest.targetBundleId = "com.danielschneller.myapp.menu";
        
        store("com.danielschneller.myapp.gui.login.LoginController", naviDest);
        

  // =============================================================================
        
        // ========== From LoginFailed (com.danielschneller.myapp.gui.login.LoginFailedController)
        // ========== On   OK
        // ========== To   Login (com.danielschneller.myapp.gui.login.LoginController in com.danielschneller.myapp.login) 
        naviDest = new NaviDestination();
        naviDest.action = "OK";
        naviDest.targetClassname = "com.danielschneller.myapp.gui.login.LoginController";
        naviDest.targetBundleId = "com.danielschneller.myapp.login";
        
        store("com.danielschneller.myapp.gui.login.LoginFailedController", naviDest);
        
        // .... and so on ...
    }
}

The support class NaviDestination is omitted but is generally just a value holder struct type class.

When creating the Xtext project using the wizard earlier we created a third Eclipse project, ending in ...generator. Its src folder contains three subdirectories called model, templates and workflow. Put the sample .navi file into the model directory. It will serve as the input for the generator.

Create the first template

Code generation is based on templates. Xtext leverages the Xpand template engine. In the templates directory create a new Xpand template using the context menu. Call it NaviRules.xpt, open it and insert the following:

«REM»
    import the namespace defined in our DSL model
«ENDREM»
«IMPORT navigationRules»

«REM»
    Define a template called "main" for elements of
    type "Root". The minus sign at the end takes care
    of not adding a newline at the end of it.
«ENDREM»
«DEFINE main FOR Root-»

«ENDDEFINE»

As there is only one instance of a Root element in a navigation rules file, this will be the main entry point - hence the name. There is no need to call it main, but it seems fitting.

Now between the DEFINE and ENDDEFINE insert what is to be generated: As shown above, we need a new Java source file called NaviRules.java:

...
«DEFINE main FOR Root-»
«FILE "NaviRules.java"-»

«ENDFILE-»
«ENDDEFINE»
...

Again, the contents to be generated is put in between the FILE and ENDFILE brackets. Anything not enclosed in «» will be used verbatim in the output file. So first of all, put in the static parts of the Java file. What I did was first write the source for a single navigation rule by hand, made sure it compiled and then copied over the relevant parts into the template piece by piece:

...
«FILE "NaviRules.java"-»
import java.util.*;

public class NaviRules {

    public static class NaviDestination {
        String action;
        List requiredPermissions = new ArrayList();
        String targetClassname;
        String targetBundleId;
        NaviDestination() {};

        public final List getRequiredPermissions() {
            return new ArrayList(requiredPermissions);
        }

        // let Eclipse generate getters, setters,
        // equals and hashCode methods for this
    }

    private Map navigationRules = new Hashtable();
    
«ENDFILE-»
...

Now, this is nothing special so far. To fill in the elements from the navigation rules DSL file put in the following:

...
    private Map navigationRules = new Hashtable();

    public NaviRules() {
        NaviDestination naviDest;
«REM»
    Iterate all elements in the "rules" collection attribute
    of the "ruledefs" attribute of the "Root" element. Call
    each iterated element (which is of type "Rule") "rule" and
    expand the "ruletmpl" template for it here.
«ENDREM»
        «FOREACH ruledefs.rules AS r»«EXPAND ruletmpl FOR r»«ENDFOREACH»
    }
...

In the class constructor we first define a local variable naviDest of the previously declared type. Then - as the comment states - the FOREACH instruction will iterate over all Rule type elements. This might not seem to be completely obvious at first. Remember at this point in the template the current scope is the "Root" element from the navigation rules file. It has an attribute called ruledefs as per the grammer definition. This attribute is of type NavigationRules which in turn has a collection attribute called rules, containing of Rule type objects. Inside the loop the current element can then be adressed by the template variable name r. The loop body (between FOREACH and ENDFOREACH) contains another Xpand instruction to expand a template called ruletmpl which will be declared next.

Don't worry, even though this is a little difficult at first - switching contexts between the Java and the template scopes is made significantly easier in Eclipse, because the Xpand template editor will syntax color (static parts are blue) and also assist you with code completion inside the Xpand template parts. Ctrl-Spacing your way through it will make things more obvious than they are when reading an example.

Now for the ruletmpl template. Place it below the ENDDEFINE statement belonging to the main template:

...
«ENDFILE-»
«ENDDEFINE»

«DEFINE ruletmpl FOR Rule-»
        // ========== From «source.name» («source.controllername»)
        «FOREACH destinations AS d»«EXPAND destTmpl(source) FOR d»«ENDFOREACH»
        // =============================================================================
        
«ENDDEFINE»

You see the same idea used again: Static parts that get transferred into the output file 1:1 and Xpand statements that fill in data from the navigation rules definition file. In this case you see references to the attributes of the Rule element. As per the FOREACH instruction in the previous template, the one at hand will be repeated for every instance of Rule in our source file. Inside this definition the current scope is that of Rule, so with «source.name» the name attribute of the CoordinateMappingSpec object referenced as source in a Rule is taken first, then the controllername attribute likewise.

Next up another FOREACH loop iterates the one or more possible Destinations of each Rule. Instead of just applying a template (destTmpl) for every Destination we also pass in the corresponding CoordinateMappingSpec stored in the source attribute of the Rule. This is then used in the following template:

...

«DEFINE destTmpl(CoordinateMappingSpec source) FOR Destination-»
        // ========== On   «transition.name»
        // ========== To   «target.name» («target.controllername» in «target.bundleid») 
        naviDest = new NaviDestination();
        naviDest.action = "«transition.name»";
        naviDest.targetClassname = "«target.controllername»";
        naviDest.targetBundleId = "«target.bundleid»";
        «FOREACH permissions AS p»«EXPAND permTmpl FOR p»«ENDFOREACH»
        store("«source.controllername»", naviDest);
        
«ENDDEFINE»

«DEFINE permTmpl FOR PermissionReference-»
        naviDest.requiredPermissions.add("«permission.value»");
«ENDDEFINE»

In this innermost templates the attributes of the CoordinateMappingSpec objects source and target are accessed and put into place to be assigned to the members a NaviDestination Java object instance per Destination. There is only one more (very simple) template for the PermissionReference elements. With this, the Xpand file is complete.

Set Up The Generator Workflow

The wizard initially created a NavigationRulesGenerator.mwe2 file in the workflow folder. Open it and replace its contents with the following:

module workflow.NavigationRulesGenerator

import org.eclipse.emf.mwe.utils.*

var targetDir = "src-gen"
var fileEncoding = "Cp1252"
var modelPath = "src/model"

Workflow {
    component = org.eclipse.xtext.mwe.Reader {
        path = modelPath
        // this class has been generated by the xtext generator 
        register = com.danielschneller.navi.NavigationRulesStandaloneSetup {}

        load = {
            slot = "root"
            type = "Root"
        }
    }

    component = org.eclipse.xpand2.Generator {
        metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}

        expand = "templates::NaviRules::main FOREACH root"
        outlet = {
            path = targetDir
        }
        fileEncoding = fileEncoding
    }
}

The most interesting parts of this workflow file are the load section in the Reader component and the expand and outlet sections in the Generator component:

The first one will connect a so-called slot with the Root element from our navigation rules.

The second one will trigger the evaluation of the main template in the NaviRules.xpt file in the templates folder and feed any Root instances it finds in the *.navi files from the src/model (modelPath) into it.

Now it is time for some actual generation.

Run the generator workflow

Right click the MWE2 file you just edited and select the Run As MWE2 Workflow command from the context menu. The Eclipse console will show this output:

0    [main] DEBUG        org.eclipse.xtext.mwe.Reader  - Resource Pathes : [src/model]
431  [main] DEBUG xt.validation.ResourceValidatorImpl  - Syntax check OK! Resource: file:/Users/ds/ws/ws36_xtext/com.danielschneller.navi.dsl.generator/src/model/MyApp.navi
1013 [main] INFO         org.eclipse.xpand2.Generator  - Written 1 files to outlet [default](src-gen)
1014 [main] INFO  .emf.mwe2.runtime.workflow.Workflow  - Done.

Then have a look at the newly generated contents of the src-gen source folder. If everything went alright, you should find a fresh NaviRules.java file placed there, based on the contents of your navigation rules file and the Xpand templates. Try and make some changes to the template, then re-run the workflow. You will see the changes reflected in the generated source file.

Generate a second source File

In the templates directory add another Xpand template file Navigation.xpt with the following content:

«IMPORT navigationRules»;

«DEFINE main FOR Root-»

«FILE "Navigation.java"-»
public final class Navigation {

«FOREACH ruledefs.rules.destinations.transition.collect(e|e.name).toSet().sortBy(e|e) AS t»«EXPAND actionTmpl FOR t»«ENDFOREACH»

    private final String name;
    
    private Navigation(String aName) {
        name = aName;
    }
    
    public String getName() {
        return name;
    }

}
«ENDFILE-»
«ENDDEFINE»

«DEFINE actionTmpl FOR String-»
    /** Constant for Navigation «this» */
    public static final Navigation «this» = new Navigation("«this»");

«ENDDEFINE»

This is a template for a type-safe enumeration that can be used in Java 1.4 - remember I had to do this for JavaME.

Notice the FOREACH loop in this case. It demonstrates that not only simple iterations are possible, but that Xpand allows more complex operations as well. In this case it will collect the names of all the navigation transitions from all the Destinations in the navigation rules. These are of type String. They are made unique by converting them to a Set datastructure and then finally sorted in their natural order. The resulting list of sorted strings is then iterated, each one - called t - is passed to the actionTmpl template. It is very simple, just placing the string itself («this») into a single line of Java source code.

Of course, strictly speaking this is a rather complicated procedure to get the same information we could also have taken from the TransitionDefinitions element in the rules definition. However I think it serves as a nice example for additional Xpand capabilities. For a full description of its possibilities, have a look at the Xpand Reference in the Eclipse documentation.

To use the new template, add another section to the MWE2 workflow definition:

component = org.eclipse.xpand2.Generator {
    metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}
    expand = "templates::Navigation::main FOREACH root"
    outlet = {
        path = targetDir
    }
    fileEncoding = fileEncoding
}

Running it again will produce a slightly different output, making clear that two files have been generated. This is what comes out in the src-gen folder as Navigation.java:

public final class Navigation {

    /** Constant for Navigation ADMIN */
    public static final Navigation ADMIN = new Navigation("ADMIN");

    /** Constant for Navigation  BACK */
    public static final Navigation BACK = new Navigation("BACK");

    /** Constant for Navigation DATA_LOOKUP */
    public static final Navigation DATA_LOOKUP = new Navigation("DATA_LOOKUP");

    /** Constant for Navigation  OK */
    public static final Navigation OK = new Navigation("OK");

    ...

More...

This was just about my first experiments with Xtext. I am sure there is plenty more to be done with it. For more reading, please have a look at this very nice Getting started with Xtext tutorial by Peter Friese of Itemis.

From http://www.danielschneller.com/2010/08/code-generation-with-xtext.html