Over a million developers have joined DZone.

The Story of javac AST Visualization on NetBeans

· Java Zone

Learn more about how the Java language, tools and frameworks have been the foundation of countless enterprise systems, brought to you in partnership with Salesforce.

The story started as soon as I was accepted to be part of the NetBeans First Patch Program. I associated myself with Bug 159203 (No Null pointer access hint like Eclipse in the Netbeans editor). I started searching for documentation on how Hints in the NetBeans Java Editor work. It turns out that the Java Editor uses a javac AST (Abstract Syntax Tree) representation behind the scenes and therefore the issue will be about understanding javac's AST.

So I started looking for tools able to visualize the javac's AST in a user-friendly way. To my big surprise, I was not able to find such a kind of tool neither as a standalone application nor as a NetBeans plug-in. Then I decided to develop such a tool by myself because it can be useful for other programmers interested in AST and because this will give me practical experience with javac's AST as well.


The Development Process

In the beginning, I just wanted to get things done and I didn't really care about the design. This ends up with a codebase where the core algorithm and data structures were tied to NetBeans visualization APIs and Swing application builds for the user interaction. Starting from commit 7 you could actually find an interesting comment in the source code:

"//TODO hacking class from architecture perspective !"
Even with a small number of classes and LOC in general, it soon started to look like Spaghetti code.

Then I make a decision to spend some time with Modularizing and getting rid of hacks. During the multiple refactorings I managed to get rid of introduced leaking abstractions and the API has changed multiple times over the time (mostly small changes, but not necessarily source code backward compatible). The overall source base was divided into 2 modules:

  • ast-visualizer-core (responsible for generating customized AST)
  • ast-visualizer-netbeans (takes advantage of VisualLibrary API to visualize the produced data structure from core)
Both are now published as Maven Central artifacts. This "mavenization" also helped with better management of dependencies. Later I have also added Gradle build scripts for both artifacts.


Faced Challenges

As with every project, you will have to face challenges. The very first and the most important challenge I have to cope with was to get familiar with Compiler Tree API. Most of the parts were clear for me and for those that were not, Java Language Specification was a good consultant (Compiler Tree API is 1:1 mapping from grammar described in JLS).

Because of lack of documentation it turns out that I was using the API in an improper way. What an unpleasant surprise! This causes me a headache when I started programming interaction with javax.lang.model package. With the help from Jan Lahoda (NetBeans Java Editor guru) it was clarified where the problem was and fixes were incorporated.

I have also spent some time with studying and implementing Reingold & Tilford's Tidier Layout. The idea was also to port the algorithm to be "first-class citizen"" in NetBeans VisualLibrary API. I was almost finished (I just thought I was almost finished) when the Efficient and Customizable TreeLayout Algorithm in Java article was published. I was able to incorporate the layout literally within few minutes.

abego Tree Layout was the real world example of "Oh happy day" situation
The only problem was the fact that abego Tree Layout was not already mavenized that time, so I could not used it properly as part of my Maven artifacts. It was absolutely clear for me to contact the author and help him with Mavenizing the abego Tree Layout project. The author was immediately interested and these days anybody can use the abego Tree Layout Maven Central artifacts.


Application Description

Now we finally get to the description of the javac AST visualization application. The core artifact is used to produce "customized AST". You might be thinking "Why is it named customized?". Well, for visualization purposes we need "small customization" of original AST produced by javac compiler. The customization process consist of:

  1. traversing javac's AST
  2. adding additional data
  3. removing unnecessary (a.k.a. useless) data

Traversing javac's AST is trivial using the Visitor Pattern. However the AST has interactions with javax.lang.model elements in a way where those elements are not instances of com.sun.source.tree.Tree interface. Speaking intelligibly, those elements will not be automatically traversed by the visitor and you have to write additional code to handle such information. That was the example of adding additional data. Why we also need to remove data? Well, there is one specific example for that. ModifiersTree is a tree node modeling modifiers & annotations. The weird part is the fact that it will not be null (the way how Visitor treats empty tree nodes) even if it is practically empty (meaning it contains no modifiers and no annotations). From my point of view, it does not really make sense to visualize such a node. That is the reason for removing it from AST.

Describing the process in general takes us to the set of most important APIs used in the core:

  1. com.sun.source.tree.* (Compiler Tree API)
  2. javax.tools.* (Java Compiler API)
  3. javax.lang.model.* (Annotation processing modeling API)

For clarification: Compiler Tree API is Oracle (former Sun) API for modeling AST produced by standard (Oracle) JDK compiler (different compilers will almost certainly produce different ASTs, for instance javac vs ecj). Java Compiler API is API for programmatic compiling of java source files. Annotation processing modeling API is API modeling code elements which can be annotated (used mostly because of annotation processing).

Because the GUI is built on top of the NetBeans Platform, it is built with respect to good programming practices (modularity, loosely coupled communication, etc). Because of modularity, it was divided into the following modules:

The purpose of each module, its loosely coupled communication (bidirectional) and the most important dependencies from the architecture point of view are modelled as follows:

This was in fact simplified scheme, at least from the dependencies point of view. Much more NetBeans APIs are used of course (Lookup API is the ultimate example).


The Result

After all those words, let's see the final real world application screenshot, click to enlarge it:

Oh, and by the way, all the code was open sourced. Howdy, howdy AST hacker! :)



Discover how the Force.com Web Services Connector (WSC) is a code-generation tool and runtime library for use with Force.com Web services, brought to you in partnership with Salesforce.


The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}