Using AST Transformations to Write a Testing Library

Being a language geek I always try to write a library that will exercise the language I’m trying to learn more about. You know, something that will heavily use metaprograming or type-system tricks. One of those libraries you can write can be a testing framework.

The first thing I thought when I switched from Groovy to Ruby is: “Why not to port Spock to Ruby?” For those who are not familiar with Spock, it’s an excellent testing framework for Groovy. It uses AST transformations, which makes possible such constructions as:

when:
  def x = 1
  def y = 1

then:
  x + y == 2

Everything in the “then” block is an assertion. Of course, it can do much more than just inserting assertions but this is really the core feature of Spock and I’ve decided to implement something similar in Ruby.

Choosing a Name

Not being a real Star Trek fan I just chose one character I liked more than others and named my project “Picard”.

The Experiment That Made it Much More Interesting

To make it interesting I decided to test “Picard” using the development version “Picard”. Basically, I took the “Eat Your Own Dog Food” rule to its extreme. Every change I make mustn’t break anything in the system. As if something is broken all my tests will be broken and I can’t test the change. This insane idea turned out to be a very interesting experiment as I was forced to be very careful with my code. All iterations were very small; I had to write lots of small temporary adapters to keep the “old” and “new” versions of code working at the same time. If it doesn’t sound like a lot of fun for you just try it.

Finally

require 'picard'

class DemoTest < Test::Unit::TestCase
  include Picard::TestUnit

  def test_simple_math
    given
      x = 1
      y = 2

    expect
      x + y == 3
  end
end

To start using picard you need to mix in Picard::TestUnit module into your TestUnit test case. It will add a special hook that will transform every test method in your test case. For instance, the “test_simple_math” method will be transformed into something like:

def test_simple_math
  given
     x = 1
    y = 2

  expect
    assert_equal 3, (x + y), MESSAGE
end

Where the MESSAGE is:

-----------------------------------------------------------------------------
| File: "/Users/savkin/projects/picard/test/picard/demo_test.rb", Line: 10|
| Failed Assertion: (x + y == 3)                                          |
-----------------------------------------------------------------------------

You might notice a few things here:

1. Picard uses TestUnit, so all your tools we will work with it just fine.
2. Picard is smart enough to insert assert_equal instead of regular assert.
3. Picard generates a very descriptive error message containing not only the file name and the line number of the failed assertion but the assertion itself. In most cases it’s enough information to understand what went wrong so you won’t have to find that exact line number to figure it out.

I Want to Try!

gem ‘picard’

What is Coming Next

There are some things I’m going to add in a week or two:

• The only special case Picard supports right now is ==. If you are using something like x != y in your expect block it will just insert a regular assert which is bad. It’s going to be much smarter than this soon.

• In Spock it’s possible to write data driven tests:

.

expect:
  x + y == z

where:
  x = [1, 10, 100]
  y = [2, 20, 200]
  z = [3, 30, 300]

Basically it will transform it into something like:

.

expect:
  1 + 2 == 3
  10 + 20 == 30
  100 + 200 == 300

Which is totally awesome! I’m going to add a similar feature to Picard soon.ght now Picard is written in Ruby 1.9 but it can parse only 1.8 syntax (which is really weird). It needs to be put in order so it will work properly on 1.8 and 1.9.

Should I Use it in Production

Probably not, but maybe in a few release when it matures you can give it a try.

To Sum Up

In my view, Picard is a nice example of what can be achieved by transforming AST. It’s a very powerful technique, which allows you to change the semantic of the language, is underused in the Ruby community. I think it needs to be taken more seriously. I’d like to see a generic framework that will make it easier to transform AST. Similar to one that exists for Groovy.

From http://victorsavkin.com/post/11124227221/using-ast-transformations-to-write-a-testing-library