A simple way to test Eclipse-based editors

After two releases of the open source version of protobuf-dt and ten Google-internal ones, code complexity grow to a point that more comprehensive testing is needed. Even though the practices in this post can be used to test different aspects of an Eclipse editor, we’ll focus on testing scoping.

Verifying that scoping works correctly involves:

1. Specifying and parsing protocol buffer-content to use for testing
2. Find a specific element in the parsed file
3. Verify that scoping returns all the possible elements that can match the element found in step #2

An additional requirement is to make tests as lightweight as possible (e.g. do not require to instantiate a full Eclipse for running tests.)

1. Specifying and parsing protocol buffer-contents to use

My first attempt was to put all the protocol buffer code in a StringBuilder and then passed it to my XtextRule to get an AST back, as follows:

@Rule public XtextRule xtext = createWith(integrationTestSetup());
 
@Test public void should_provide_Property_fields_for_custom_field_option() {
  StringBuilder proto = new StringBuilder();
  proto.append("package com.google.proto;                   ")
       .append("import 'google/protobuf/descriptor.proto';  ")
       .append("                                            ")
       .append("extend google.protobuf.FieldOptions {       ")
       .append("  optional int32 code = 1000;               ")
       .append("  optional int32 info = 1001;               ")
       .append("}                                           ")
       .append("                                            ")
       .append("message Person {                            ") 
       .append("  optional boolean active = 1 [(code) = 68];")
       .append("}                                           ");
  Protobuf root = xtext.parseText(proto);
  // test implementation
}

Needless to say, writing and maintaining this is a real PITA. To make it readable I have to add extra whitespace to each line so the whole thing looks square. In addition, copy/paste code from the protocol buffer editor requires also adding append and quotes to each line, plus formatting. On top of that, it makes the test method too long.

An alternative approach was to have .proto files in the file system. I was not happy with this approach since anybody reading the code will need to go to two places to understand what the test is doing.

Finally, I found a better approach: specify the protocol buffer text as a comment!

I didn’t come up with this idea myself, I’m just borrowing it from CDT. After implementing a simpler, but similar approach, my test looks like this:

// package com.google.proto;
// import 'google/protobuf/descriptor.proto';
//  
// extend google.protobuf.FieldOptions {
//   optional int32 code = 1000;
//   optional int32 info = 1001;
// }
//  
// message Person {
//   optional boolean active = 1 [(code) = 68];
// }
@Test public void should_provide_Property_fields_for_custom_field_option() {
  Protobuf root = xtext.root();
  // test implementation
}

This is a big improvement! First of all, the test method ends up being shorter and easier to read. Second, it is easy to type something in the protocol buffer editor, copy it, and paste it above a test method. Then I just have to highlight it and press Ctrl+/ to have the text converted to a comment.

In this new scenario, my custom XtextRule does the following,:

1. extracts the comment of each test method and creates a method name > comment mapping
2. parses the comment of a test method and creates an AST just before executing the test method, not earlier
3. keeps a reference to the root node of the AST, to be used by the test method itself

It looks good, but what about imported .proto files?

When testing scoping, it is absolutely necessary to verify that imported types are included correctly.

I originally had .proto files to be imported in the file system and stored in Git, which is ugly for the reason I mentioned before.

To make things better, I borrowed another idea from CDT: specify the text and file name of the .proto file to be imported in comments, XtextRule will create the file in the file system just before executing a test method.

Here is an example:

// // Create file custom-options.proto
// 
// package com.google.proto;
//
// import "google/protobuf/descriptor.proto";
//
// extend google.protobuf.FileOptions {
//   optional int32 code = 1000;
//   optional int32 info = 1002;
// }  
 
// package com.google.proto;
//  
// import 'custom-options.proto';
//
// option (code) = 68;
@Test public void should_provide_imported_Property_fields_for_custom_option() {
  // test implementation
}

In the example above we have two comments for a test method. The first one tells my XtextRule to create a file named “custom-options.proto” before executing the test method. The second comment will be the one parsed and whose AST will be stored by the XtextRule (just like in the previous example.)

Neat, isn’t it? :)

2. Find a specific element in the parsed file

This step is necessary to ensure that scoping returns all the possible types that a given reference may be pointing to.

In this example:

// package com.google.proto;
// import 'google/protobuf/descriptor.proto';
//  
// extend google.protobuf.FieldOptions {
//   optional int32 code = 1000;
//   optional int32 info = 1001;
// }
//  
// message Person {
//   optional boolean active = 1 [(info) = 68];
// }
@Test public void should_provide_Property_fields_for_custom_field_option() {
}

my test is going to verify that scoping returns the field options code and info as potential matches for (code). To do so, I first need to find the proper AST element that (code) represents.

Let’s say I want to search for the custom field option info. My original approach would require the following:

1. Find all the elements in the AST of type CustomFieldOption
2. If any result is returned from #1, find the one whose name is “info”

Here is an example:

Protobuf root = xtext.root();
// statically imported from CustomFieldOptionFinder
CustomFieldOption option = findCustomFieldOption(name("info"), in(root));

This solution by itself is not too bad, and it works! Unfortunately it requires one specialized finder per type in the AST. For a relatively simple language like Protocol Buffers, it would require more than 10 finders, which is too much code to maintain just for testing.

Enter CDT with a better approach, which requires only one finder for all the types in the AST. This finder looks for elements this way:

1. Find the first element matching some text
2. Verify that the found element is of the the specified type and has the specified name

For example:

CustomFieldOption option = xtext.find("info", ")", CustomFieldOption.class);

To find the custom option “info” this finder will:

1. concatenates the Strings passed as arguments and find the first element that matches the text “info)”
2. verify that the found element is a CustomFieldOption and has name “info” (the first String argument only)

As you can see, this approach is the complete opposite of my original one.

Since only one finder is needed, I can have my XtextRule create it and pass the root of the AST to it. This way, I don’t have to pass it to the finder every time I perform a lookup.

3. Verifying that scoping returns the correct types

This is actually DSL-specific. My only recommendation here is, if you are using JUnit, write your own Hamcrest matchers to keep test code short and readable, and without duplication.

Putting it all together

Here is how one of my tests for protobuf-dt looks like:

// message Person {
//   optional Type type = 1 [ctype = STRING];
// }
@Test public void should_provide_Property_fields_for_native_field_option() {
  // We have an overloaded version of "find" that takes only one <code>String</code>, 
  // to be used when the text to find and the name to match are the same.
  NativeFieldOption option = xtext.find("ctype", NativeFieldOption.class);
  IScope scope = provider.scope_PropertyRef_property(option.getProperty(), reference);
  Collection<Property> fieldOptions = descriptor().optionsOfType(FIELD);
  assertThat(descriptionsIn(scope), containAll(fieldOptions));
}

Please feel free to click these links to find the code of:

• XtextRule
• ModelFinder
• or just get the project’s source code, as described here

Feedback is always welcome! :)

Future posts

There are a couple of useful things I have done with Xtext that I’d like to blog about in the near future:

• Adding spell checking to comments and strings
• Opening files outside an Eclipse workspace
• Making Xtext work with file names, instead of file extensions

Now it is a matter of finding the time and energy!

From http://alexruiz.developerblogs.com/?p=1962