/ DevOps Zone
Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Brief comparison of BDD frameworks

DZone's Guide to

Brief comparison of BDD frameworks

JDave, Concordion, Easyb, JBehave, Cucumber are all compared here briefly for your convenience.

by
Sebastian Laskawiec
·
Feb. 24, 14 · DevOps Zone
Free Resource

Comment (12)

Save
Tweet
{{ articles[0].views | formatCount}} Views

The Nexus Suite is uniquely architected for a DevOps native world and creates value early in the development pipeline, provides precise contextual controls at every phase, and accelerates DevOps innovation with automation you can trust. Read how in this ebook.

Some time ago my team was asked to create up to date documentation of our projects. First we thought about Wiki page, it's far batter then specification written in Doc, but it still becomes out of date very quickly. An ideal approach requires changing documentation each time a significant change in code is being made. After some research we have discovered BDD frameworks (BDD – behavior driven development) and this article is about experimenting and choosing the right BDD framework which suites our needs.

The way BDD works is really simple – it is a bridge between business language (User Stories) and automatic tests (JUnit, TestNG). In order to keep the documentation in sync with the code – we need to run it as test in Continuous Integration build (Jenkins, Bamboo). Test specification should be written in business friendly form, for example in some text or HTML file. This way it will be easily accessible by Business Analysts. On the other hand in must fail just like any other test during Surefire or Failsafe execution. After the build we should be able to get user friendly HTML report.

Most of BDD frameworks use common template for whiting tests:

given (some context)

when (something happens)

then (some behavioral validation)

We were experimenting with several BDD frameworks including:

For the purpose of this article, let's imagine we have Favorites Repository. It may store our favorite book or song. For the testing purpose I used one my favorites book – Terry Pratchett's Going Postal. We can test the repository using two scenarios:

Scenario name: Empty Favorites Repository

Given empty Favorites Repository

When there is nothing added to the repository

Then favorite book is null








Scenario: Going Postal added to Favorites Repository

Given empty Favorites Repository

When Going Postal is added to Favorites Repository

Then favorite book is Going Postal


JDave

Although JDave is focused on specification testing, although it's not strictly BDD framework, but it is worth to take a look. JDave's main concept is called Specification. This is a container for Behaviors. Behaviors in turn defines how class behaves in specific context. Here is how we might write our case in JDave specific language:

DefaultFavoritesRepositorySpecification specification

  • Empty repository
    • Has empty books
  • Repository with Going Postal added
    • Has Going Postal book

The code looks like this:

@RunWith(JDaveRunner.class)
@Group("basic")
public class DefaultFavoritesRepositorySpecification extends Specification<DefaultFavoritesRepository> {

    public class EmptyRepository {

        private FavoritesRepository repository = new DefaultFavoritesRepository();

        public FavoritesRepository create() {
            return repository;
        }

        public void hasEmptyBooks() {
            specify(repository.getFavorite(Book.class), must.equal(null));
        }

        public void hasEmptySongs() {
            specify(repository.getFavorite(Song.class), must.equal(null));
        }
    }

    public class RepositoryWithGoingPostalAdded {

        private FavoritesRepository repository;

        public FavoritesRepository create() {
            FavoritesRepository ret = new DefaultFavoritesRepository();
            Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
            ret.putFavorite(book);
            repository = ret;
            return ret;
        }

        public void hasGoingPostalBook() {
            Book goingPostalBook = new Book("9781407035406", "Terry Pratchett", "Going Postal");
            specify(repository.getFavorite(Book.class), must.equal(goingPostalBook));
        }
    }
}

JDave provides integration with Maven report plugin. Here is a screenshot of a sample report:

JDave&apos;s report

JDave's syntax might look odd and complicated, but it might be useful on small scale specification testing. In case of a large scale projects, which needs self documenting tests there are better tools.

Pros/Cons:
+ Integrated with Maven reporting plugin
-  Only for small scale testing
-  Complicated syntax

Easyb

Very nice and easy to learn Groovy based framework designed for BDD tests. Every story is written in text file and then it is tested by Easyb Maven plugin. Story files look very clear and handy, just take a look:

//EmptyDefaultFavoritesRepository.story
import com.github.bdd.core.favorites.*
import com.github.bdd.core.favorites.repository.*

scenario "Empty Favorites Repository", {

    given "empty Favorites Repository",{
        repository = new DefaultFavoritesRepository()
    }

    when "there is nothing added to the repository", {

    }

    then "favorite book is null", {
        repository.database.get(Book).shouldBe null
    }
}
//GoingPostalAddedToFavoritesRepository
import com.github.bdd.core.favorites.*
import com.github.bdd.core.favorites.repository.*

scenario "Going Postal added to Favorites Repository", {

    given "empty Favorites Repository",{
        repository = new DefaultFavoritesRepository()
    }

    when "Going Postal is added to Favorites Repository", {
        book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    then "favorite book is Going Postal", {
        goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.database.get(Book).shouldBe goingPostal
    }

}

Easyb also allows creating HTML reports. With standard configuration they are not attached to Maven site report, but attaching them manually is not a pain. With default CSS files the report looks really nice:

Easyb&apos;s report

Easyb&apos;s report

Easyb supports also Specification testing. Tests are small, clean and should be maintained without pain. However there are some drawbacks. Its based on Groovy, so the developers need to learn at least some of its features. Easyb is based on its own syntax – again another thing to learn. Finally – it's being run by its own Maven plugin, so if you have some complex Surefire or Failsafe configuration – it might be hard to configure them properly.

Pros/Cons:
+ Simple and easy to learn
+ Supports both Story and Specification testing
+ Story and glue code in one place
-  Tests are ran by it's own plugin
-  Groovy based (In many cases it's not a drawback, but just another thing to learn)

JBehave

JBehave is one of the largest frameworks for BDD written for Java. Writing tests process consists of writing stories, mapping them to Java, configuring them and finally – running and viewing reports. Configuration step is done only once. There are many ways of running stories. I decided to integrate it with JUnit (both testing and reporting). All configuration is done via annotations:

@RunWith(AnnotatedEmbedderRunner.class)
@Configure(storyLoader = JBehaveTest.StoryLoader.class,
        storyReporterBuilder = JBehaveTest.ReportBuilder.class)
@UsingEmbedder(embedder = Embedder.class, generateViewAfterStories = true, ignoreFailureInStories = false,
        ignoreFailureInView = true, verboseFailures = true)
@UsingSteps(instances = { EmptyRepositorySteps.class, GoingPostalRepositorySteps.class })
public class JBehaveTest extends InjectableEmbedder {

    @Test
    public void run() {
        List<String> storyPaths = new StoryFinder().findPaths(codeLocationFromClass(this.getClass()), "**/*.story", "");
        injectedEmbedder().runStoriesAsPaths(storyPaths);
    }

    public static class StoryLoader extends LoadFromClasspath {

        public StoryLoader() {
            super(JBehaveTest.class.getClassLoader());
        }
    }

    public static class ReportBuilder extends StoryReporterBuilder {

        public ReportBuilder() {
            this.withFormats(org.jbehave.core.reporters.Format.CONSOLE,
                    org.jbehave.core.reporters.Format.HTML).withDefaultFormats();
        }
    }
}

Stories:

# EmptyFavoritesRepository.story
Scenario: Empty Favorites Repository

Given empty Favorites Repository
When there is nothing added to the repository
Then favorite book is null
# GoingPostalAddedToFavoritesRepository.story
Scenario: Going Postal added to Favorites Repository

Given empty Favorites Repository
When Going Postal is added to Favorites Repository
Then favorite book is Going Postal

Mapping files:

public class EmptyRepositorySteps {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("empty Favorites Repository")
    public void givenEmptyFavoritesRepository() {

    }


    @When("there is nothing added to the repository")
    public void whenThereIsNothingAddedToTheRepository() {

    }

    @Then("favorite book is null")
    public void thenFavoriteBookIsNull() {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
public class GoingPostalRepositorySteps {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("empty Favorites Repository")
    public void givenEmptyFavoritesRepository() {

    }

    @When("Going Postal is added to Favorites Repository")
    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    @Then("favorite book is Going Postal")
    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }
}

Mapping is done using regexp by default. The same text must be used in stories and annotations (it also might be bound to variables). When JBehave doesn't find corresponding mapping – it prints warning message. This message contains also template methods which might be copied to mapping classes. This really speeds up development. Another nice feature is @Pending annotation – steps annotated with it are simply skipped (think about it as “not implemented” or “in progress”).

JBehave reports are highly customizable. All HTML templates might be defined using Freemarker. There is also a default set of templates embedded into JBehave, which is used here:

JBehave&apos;s report

JBehave is a very flexible and expendable framework. It integrates with numbers of different programming languages (Groovy, JRuby, Java), technologies and frameworks. Stories are clear for business world and mapping files allows translating them into Java easily. Reporting functionality is really great. Defining own templates with Freemarker makes it easy to adopt to corporate standards. JBehave's configuration is a bit overwhelming, but it needs to be done only once.

Pros/Cons:
+ Highly configurable
+ Customizable report templates
+ Support for many languages and technologies
-  Configuration is a bit overwhelming

Cucumber

Cucumber was originally written for Ruby. After some time additional plugins and runners were added. Cucumber has much simpler configuration. Just like in JBehave writing test is divided into steps – writing stories, mapping them into Java (also called glue code), running and reporting. Stories look very similar:

Feature: Favorites Repository

Scenario: Empty Favorites Repository
Given empty Favorites Repository
When there is nothing added to the repository
Then favorite book is null

Scenario: Going Postal added to Favorites Repository
Given empty Favorites Repository
When Going Postal is added to Favorites Repository
Then favorite book is Going Postal

All scenarios for one feature reside in one place. The glue code is pretty similar to JBehave's:

@RunWith(Cucumber.class)
@CucumberOptions(format = { "pretty", "html:target/cucumber", "json:target/cucumber.json" }, glue = "pl.payu.dictionary.client.exception.steps")
public class CucumberTest {

}
public class EmptyRepositoryStepDefinitions {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("^empty Favorites Repository$")
    public void Empty_Favorites_Repository() throws Throwable {
        repository = new DefaultFavoritesRepository();
    }

    @When("^there is nothing added to the repository$")
    public void there_is_nothing_added_to_the_repository() throws Throwable {

    }

    @Then("^favorite book is null$")
    public void Favorite_book_is_null() throws Throwable {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
public class GoingPostalRepositoryStepDefinitions {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    @When("^Going Postal is added to Favorites Repository$")
    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    @Then("^favorite book is Going Postal$")
    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }

}

The default reports looks really great:

Cucumber&apos;s report

Cucumber&apos;s report

Cucumber is a very powerful framework for BDD testing. It has (just like JBehave) many useful features like testing by example or parameters. Both features and glue code looks very nice and might be maintained easily. Additionally Cucumber supports many different languages and platforms like Ruby, Java or .NET. Additionally Arquillian support comes out of box and there is also very useful Jenkins plugin for generating reports.

Pros/Cons:
+ Simple but powerful
+ Jenkins reporting plugin
+ Support for many languages
-  Not so easy to embed in your own report templates

Concordion

Concordion is very similar to Cucumber but features might be written in HTML fashion. It also supports many different platforms and languages.

Here are 2 stories written in HTML and an “index” page:

<html xmlns:concordion="http://www.concordion.org/2007/concordion">
    <head>
        <title>Concordion tests</title>
    </head>

    <body>

    <h1>Concordion tests</h1>

    <p>
        <ul>
            <li>
                <a concordion:run="concordion" href="EmptyFavoritesRepository.html">Empty Favorites Repository</a>
            </li>
            <li>
                <a concordion:run="concordion" href="GoingPostalAddedToFavoritesRepository.html">Going Postal Added To Favorites Repository</a>
            </li>
        </ul>
    </p>

    </body>
</html>
<html xmlns:concordion="http://www.concordion.org/2007/concordion">

    <head>
        <title>Favorites Repository</title>
    </head>

    <body>

        <h1>Empty Favorites Repository</h1>

        <p>
            Given empty Favorites Repository <span concordion:execute="emptyFavoritesRepository()" /><br/>
            When there is nothing added to the repository <span concordion:execute="thereIsNothingAddedToTheRepository()" /><br/>
            Then favorite book is null <span concordion:execute="favoriteBookIsEmpty()" /> null.
        </p>

    </body>

</html>
<html xmlns:concordion="http://www.concordion.org/2007/concordion">

    <head>
        <title>Going Postal added to favorites repository</title>
    </head>

    <body>

        <h1>Going Postal added to Favorites Repository</h1>

        <p>
            Given empty Favorites Repository <span concordion:execute="emptyFavoritesRepository()"/><br/>
            When Going Postal is added to Favorites Repository <span concordion:execute="whenGoingPostalIsAddedToFavoritesRepository()"/><br/>
            Then favorite book is <span concordion:execute="thenFavoriteBookIsGoingPostal()"/> Going Postal.
        </p>

    </body>

</html>

The glue code does not contain method annotations. Concordion simply use method names:

@RunWith(ConcordionRunner.class)
public class FavoritesRepositoryTest {

}
@RunWith(ConcordionRunner.class)
public class EmptyFavoritesRepository {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    public void emptyFavoritesRepository() throws Exception {

    }

    public void thereIsNothingAddedToTheRepository() throws Exception {

    }

    public void favoriteBookIsEmpty() throws Exception {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
@RunWith(ConcordionRunner.class)
public class GoingPostalAddedToFavoritesRepositoryTest {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    public void emptyFavoritesRepository() throws Exception {

    }

    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }

}

Default styles and above HTML files doesn't look very nice, but can be easily adjusted to corporation standards:

Concordion&apos;s report

Concordion&apos;s report

Concordion&apos;s report

The main difference between Concordion and Cucumber is how it uses mapping fies. Concordion uses method names for resolution and Cucumber - Regexp expressions. All major features like testing by example or parameters are also implemented.

Pros/Cons:
+ The most customizable implementation for writing scenarios
+ Support for many languages and platforms
- Fewer plugins available

Summary

There are many BDD frameworks on the market. Some of them are designed only for specification testing and some are very flexible and support many different features.

My team is responsible for maintaining several JEE projects. Some of them contains very sophisticated and complicated business logic and all of them are written in pure Java. This is why we discarded JDave and EasyB at the beginning. Next we realized, that we don't need to use our own CSS files in the reports – this is why we discarded Concordion. Finally we had to choose among JBehave and Cucumber. We decided to use Cucumber because of Arquillian support out of box and Cucumber offers simpler bootstrapping. Now we plan to adjust Cucumber reporting plugin to use with Bamboo... Wish us luck!

References

The DevOps Zone is brought to you in partnership with Sonatype Nexus.  See how the Nexus platform infuses precise open source component intelligence into the DevOps pipeline early, everywhere, and at scale. Read how in this ebook. 

Like This Article? Read More From DZone

The Difference Between BDD and TDD
The Role of QA in a DevOps World
Test Pyramid Heresy

Free DZone Refcard

Declarative Pipeline With Jenkins
DOWNLOAD
Topics:
devops ,bdd ,testing

Comment (12)

Save
Tweet
{{ articles[0].views | formatCount}} Views

Opinions expressed by DZone contributors are their own.

DevOps Partner Resources

Take the Forrester DevOps Maturity Assessment Test
Automate Security in Your DevOps Pipeline
Starting and Scaling DevOps in the Enterprise
DevOps is Crippled Without Continuous Integration
Continue your digital transformation with a Blueprint for Continuous Delivery.
The DevOps Journey - From Waterfall to Continuous Delivery
Solve the 4 Major Problems with Text Data
Uncover your DevOps maturity level in four key areas.
The CTO’s Ultimate Guide to Continuous Testing
Redefine application release processes and unify your DevOps strategy.
Optimizing Internal Communications with DevOps
The Ultimate Monitoring Tools Landscape

DevOps Partner Resources

Take the Forrester DevOps Maturity Assessment Test
Automate Security in Your DevOps Pipeline
Starting and Scaling DevOps in the Enterprise
DevOps is Crippled Without Continuous Integration
THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.linkDescription }}

{{ parent.urlSource.name }}
by {{ parent.authors[0].realName || parent.author}}
· {{ parent.articleDate | date:'MMM. dd, yyyy' }} {{ parent.linkDate | date:'MMM. dd, yyyy' }}
· {{ parent.portal.name }} Zone