ANTLR and Jetbrains MPS: Parsing files and Displaying Their ASTs Using Tree Notation

itemis did it again: they just released a new, very cool plugin for jetbrains mps. this one allows you to define new tree editors.

they look like this:

in this post we are going to see:

• how to use antlr parsers inside mps
• how to represent the parsed ast using the tree notation

in particular, we are going to use the antlr grammar which parses… antlr grammars. how meta is that? the very same approach could be used for every antlr grammar, of course.

as always the code is available on github .

dependencies

first of all, you need to install jetbrains mps. grab your free copy here .

to use the tree notations you should install the mbeddr platform. just go here , download a zip and unzip it among the plugins of your mps installation.

all set, time to do some programming.

packaging antlr to be used inside mps

in a previous post , we discussed how to use an existing antlr grammar in java projects using gradle. we will apply that technique here as well.

we start by download the grammar from here: https://github.com/antlr/grammars-v4/tree/master/antlr4

we just do some minor changes by directly including lexbasic into antlrv4lexer. note that we also need the lexeradaptor .

to simplify usage we create a facade:

package me.tomasetti.mpsantlr.parser;

import me.tomassetti.antlr4.parser.antlrv4lexer;
import me.tomassetti.antlr4.parser.antlrv4parser;
import org.antlr.v4.runtime.commontokenstream;
import org.antlr.v4.runtime.tokenstream;

import java.io.*;
import java.nio.charset.standardcharsets;

public class antlr4parserfacade {

    public antlrv4parser.grammarspeccontext parsestring(string code) {
        inputstream inputstream = new bytearrayinputstream(code.getbytes(standardcharsets.utf_8));
        return parsestream(inputstream);
    }

    public antlrv4parser.grammarspeccontext parsefile(file file) throws filenotfoundexception {
        return parsestream(new fileinputstream(file));
    }

    public antlrv4parser.grammarspeccontext parsestream(inputstream inputstream) {
        try {
            antlrv4lexer lexer = new antlrv4lexer(new org.antlr.v4.runtime.antlrinputstream(inputstream));
            tokenstream tokens = new commontokenstream(lexer);
            antlrv4parser parser = new antlrv4parser(tokens);
            return parser.grammarspec();
        } catch (ioexception e) {
            throw new runtimeexception("that is unexpected", e);
        }
    }

}

now we need a build file:

buildscript {
    repositories {
        maven {
            name 'jfrog oss snapshot repo'
            url  'https://oss.jfrog.org/oss-snapshot-local/'
        }
        jcenter()
    }

}

repositories {
    mavencentral()
    jcenter()
}

apply plugin: 'java'
apply plugin: 'antlr'
apply plugin: 'idea'


dependencies {
    antlr "org.antlr:antlr4:4.5.1"
    compile "org.antlr:antlr4-runtime:4.5.1"
    testcompile 'junit:junit:4.12'
}

generategrammarsource {
    maxheapsize = "64m"
    arguments += ['-package', 'me.tomassetti.antlr4.parser']
    outputdirectory = new file("${project.builddir}/generated-src/antlr/main/me/tomassetti/antlr4/parser".tostring())
}

task fatjar(type: jar) {
    manifest {
        attributes 'implementation-title': 'antlr4-parser',
                   'implementation-version': '0.0.1'
    }
    basename = project.name + '-all'
    from { configurations.compile.collect { it.isdirectory() ? it : ziptree(it) } }
    with jar
}

you may want to run:

• gradle idea to create a jetbrains idea project
• gradle fatjar to create a jar which will contain our compiled code and all the dependencies

good. now to use this parser in mps we start by creating a project. in the wizard, we select the runtime and sandbox options. once we have done that we should copy our fat jar under the models directory of the runtime solution. in my case i run from the directory of the java project this command:

cp build/libs/parser-all.jar ../languages/me.tomassetti.mpsantlr/runtime/models/

now we need to make mps aware of that jar. lets’s select the sandbox solution and first add the jar to the models:

then we add it to the libraries:

now the content of the jar should appear among the stubs of the runtime solution.

creating mps nodes from ast nodes

now we are going to build a new concept named antlrimporter. we will use it to select and import antlr grammars into mps:

the concept structure will be pretty simple:

we also need concepts for the ast nodes we are going to import. first of all, we will define the abstract concept astnode . then we will define two subconcepts for the terminal and non-terminal ast nodes.

now let’s take a look at the editor for the antlrimporter.

the first swing component is a button which opens a file chooser. in this way, we can easily select a file and set the property path . or we can edit it manually if we prefer.

once we have selected a file we can import it by clicking on the second button

the import logic is in importmodel , a method in the behavior of antlrimporter.

good. that is it. with that, we can parse any antlr grammar and get it into mps. now we just have to use a nice representation. for this, we have chosen the tree notation.

using the tree notation

the tree notation is surprising easily to use.

let’s start by adding com.mbeddr.mpsutil.treenotation.styles.editor to the dependencies of the editor aspect of our language.

we will also need the com.mbeddr.mpsutil.treenotation to be among the used languages.

the editor for nonterminalnode consists of a single tree cell. the top part of the tree cell represents this node. we will use the rulename to represent it. the bottom part instead contains the children to be displayed in the tree.

we can put the cursor on the tree drawing between the top and the bottom part (the “ /|\ ” symbol) and open the inspector. there we can use style attributes to customize the appearance of the tree.

we just decide to show the tree from left-to-right instead of top down. then we decide to add more spaces between the parent and the children when there are too many children. in this way, the lines do not overlap too much.

this is how it looks without the property:

this is how it looks with the property set:

there are other properties that can be used to control the color and the thickness of the lines. you can also add shapes at the extremes of the lines. for now, we do not need these features, but it is nice to know they are there.

the editor for terminalnode is very simple:

conclusion

over the years mps has become more stable and easier to use, it has reached the point at which it can greatly improve your productivity. projectional editing is an idea that has been around for a while and there are other implementations available like the whole platform . however, mps is now quite mature as a product.

i think we are still missing the following:

• processes and best practices : how best to manage dependencies with other mps projects? how best to integrate with java libraries?
• examples : there are surprisingly few applications which are publicly available. after all, many users develop dsls for their specific usages and do not intend to share them. however, this means we have few opportunities to learn from each other.
• extensions : the mbeddr team is doing an amazing job providing all sorts of goodies as part of the mbeddr platform. however, they seem the only ones producing reusable components and sharing them.

now's the time for the community to start working together to see we can achieve. in my opinion, these are going to be very interesting times.

it would be interesting to hear how others are using mps, please get in touch and share your experiences.