DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

SBOMs are essential to circumventing software supply chain attacks, and they provide visibility into various software components.

Related

  • WebDriverIO Integration With Cucumber
  • Design Patterns for Scalable Test Automation Frameworks
  • COM Design Pattern for Automation With Selenium and Cucumber
  • The Power of Docker and Cucumber in Automation Testing

Trending

  • One Checkbox to Cloud: Migrating from Tosca DEX Agents to E2G
  • Testing the MongoDB MCP Server Using SingleStore Kai
  • What Is Plagiarism? How to Avoid It and Cite Sources
  • Spring Cloud LoadBalancer vs Netflix Ribbon
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. Brief comparison of BDD frameworks

Brief comparison of BDD frameworks

JDave, Concordion, Easyb, JBehave, Cucumber are all compared here briefly for your convenience.

By 
Sebastian Laskawiec user avatar
Sebastian Laskawiec
·
Feb. 24, 14 · Analysis
Likes (16)
Comment
Save
Tweet
Share
129.3K Views

Join the DZone community and get the full member experience.

Join For Free

Some time ago my team was asked to create up to date documentation of our projects. First we thought about Wiki page, it's far batter then specification written in Doc, but it still becomes out of date very quickly. An ideal approach requires changing documentation each time a significant change in code is being made. After some research we have discovered BDD frameworks (BDD – behavior driven development) and this article is about experimenting and choosing the right BDD framework which suites our needs.

The way BDD works is really simple – it is a bridge between business language (User Stories) and automatic tests (JUnit, TestNG). In order to keep the documentation in sync with the code – we need to run it as test in Continuous Integration build (Jenkins, Bamboo). Test specification should be written in business friendly form, for example in some text or HTML file. This way it will be easily accessible by Business Analysts. On the other hand in must fail just like any other test during Surefire or Failsafe execution. After the build we should be able to get user friendly HTML report.

Most of BDD frameworks use common template for whiting tests:

given (some context)

when (something happens)

then (some behavioral validation)

We were experimenting with several BDD frameworks including:

    • JDave - http://jdave.org/
    • Concordion - http://www.concordion.org
    • Easyb - http://easyb.org/
    • JBehave - http://jbehave.org/
    • Cucumber - http://cukes.info/

For the purpose of this article, let's imagine we have Favorites Repository. It may store our favorite book or song. For the testing purpose I used one my favorites book – Terry Pratchett's Going Postal. We can test the repository using two scenarios:

Scenario name: Empty Favorites Repository

Given empty Favorites Repository

When there is nothing added to the repository

Then favorite book is null








Scenario: Going Postal added to Favorites Repository

Given empty Favorites Repository

When Going Postal is added to Favorites Repository

Then favorite book is Going Postal


JDave

Although JDave is focused on specification testing, although it's not strictly BDD framework, but it is worth to take a look. JDave's main concept is called Specification. This is a container for Behaviors. Behaviors in turn defines how class behaves in specific context. Here is how we might write our case in JDave specific language:

DefaultFavoritesRepositorySpecification specification

  • Empty repository
    • Has empty books
  • Repository with Going Postal added
    • Has Going Postal book

The code looks like this:

@RunWith(JDaveRunner.class)
@Group("basic")
public class DefaultFavoritesRepositorySpecification extends Specification<DefaultFavoritesRepository> {

    public class EmptyRepository {

        private FavoritesRepository repository = new DefaultFavoritesRepository();

        public FavoritesRepository create() {
            return repository;
        }

        public void hasEmptyBooks() {
            specify(repository.getFavorite(Book.class), must.equal(null));
        }

        public void hasEmptySongs() {
            specify(repository.getFavorite(Song.class), must.equal(null));
        }
    }

    public class RepositoryWithGoingPostalAdded {

        private FavoritesRepository repository;

        public FavoritesRepository create() {
            FavoritesRepository ret = new DefaultFavoritesRepository();
            Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
            ret.putFavorite(book);
            repository = ret;
            return ret;
        }

        public void hasGoingPostalBook() {
            Book goingPostalBook = new Book("9781407035406", "Terry Pratchett", "Going Postal");
            specify(repository.getFavorite(Book.class), must.equal(goingPostalBook));
        }
    }
}

JDave provides integration with Maven report plugin. Here is a screenshot of a sample report:

JDave&apos;s report

JDave's syntax might look odd and complicated, but it might be useful on small scale specification testing. In case of a large scale projects, which needs self documenting tests there are better tools.

Pros/Cons:
+ Integrated with Maven reporting plugin
-  Only for small scale testing
-  Complicated syntax

Easyb

Very nice and easy to learn Groovy based framework designed for BDD tests. Every story is written in text file and then it is tested by Easyb Maven plugin. Story files look very clear and handy, just take a look:

//EmptyDefaultFavoritesRepository.story
import com.github.bdd.core.favorites.*
import com.github.bdd.core.favorites.repository.*

scenario "Empty Favorites Repository", {

    given "empty Favorites Repository",{
        repository = new DefaultFavoritesRepository()
    }

    when "there is nothing added to the repository", {

    }

    then "favorite book is null", {
        repository.database.get(Book).shouldBe null
    }
}
//GoingPostalAddedToFavoritesRepository
import com.github.bdd.core.favorites.*
import com.github.bdd.core.favorites.repository.*

scenario "Going Postal added to Favorites Repository", {

    given "empty Favorites Repository",{
        repository = new DefaultFavoritesRepository()
    }

    when "Going Postal is added to Favorites Repository", {
        book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    then "favorite book is Going Postal", {
        goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.database.get(Book).shouldBe goingPostal
    }

}

Easyb also allows creating HTML reports. With standard configuration they are not attached to Maven site report, but attaching them manually is not a pain. With default CSS files the report looks really nice:

Easyb&apos;s report

Easyb&apos;s report

Easyb supports also Specification testing. Tests are small, clean and should be maintained without pain. However there are some drawbacks. Its based on Groovy, so the developers need to learn at least some of its features. Easyb is based on its own syntax – again another thing to learn. Finally – it's being run by its own Maven plugin, so if you have some complex Surefire or Failsafe configuration – it might be hard to configure them properly.

Pros/Cons:
+ Simple and easy to learn
+ Supports both Story and Specification testing
+ Story and glue code in one place
-  Tests are ran by it's own plugin
-  Groovy based (In many cases it's not a drawback, but just another thing to learn)

JBehave

JBehave is one of the largest frameworks for BDD written for Java. Writing tests process consists of writing stories, mapping them to Java, configuring them and finally – running and viewing reports. Configuration step is done only once. There are many ways of running stories. I decided to integrate it with JUnit (both testing and reporting). All configuration is done via annotations:

@RunWith(AnnotatedEmbedderRunner.class)
@Configure(storyLoader = JBehaveTest.StoryLoader.class,
        storyReporterBuilder = JBehaveTest.ReportBuilder.class)
@UsingEmbedder(embedder = Embedder.class, generateViewAfterStories = true, ignoreFailureInStories = false,
        ignoreFailureInView = true, verboseFailures = true)
@UsingSteps(instances = { EmptyRepositorySteps.class, GoingPostalRepositorySteps.class })
public class JBehaveTest extends InjectableEmbedder {

    @Test
    public void run() {
        List<String> storyPaths = new StoryFinder().findPaths(codeLocationFromClass(this.getClass()), "**/*.story", "");
        injectedEmbedder().runStoriesAsPaths(storyPaths);
    }

    public static class StoryLoader extends LoadFromClasspath {

        public StoryLoader() {
            super(JBehaveTest.class.getClassLoader());
        }
    }

    public static class ReportBuilder extends StoryReporterBuilder {

        public ReportBuilder() {
            this.withFormats(org.jbehave.core.reporters.Format.CONSOLE,
                    org.jbehave.core.reporters.Format.HTML).withDefaultFormats();
        }
    }
}

Stories:

# EmptyFavoritesRepository.story
Scenario: Empty Favorites Repository

Given empty Favorites Repository
When there is nothing added to the repository
Then favorite book is null
# GoingPostalAddedToFavoritesRepository.story
Scenario: Going Postal added to Favorites Repository

Given empty Favorites Repository
When Going Postal is added to Favorites Repository
Then favorite book is Going Postal

Mapping files:

public class EmptyRepositorySteps {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("empty Favorites Repository")
    public void givenEmptyFavoritesRepository() {

    }


    @When("there is nothing added to the repository")
    public void whenThereIsNothingAddedToTheRepository() {

    }

    @Then("favorite book is null")
    public void thenFavoriteBookIsNull() {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
public class GoingPostalRepositorySteps {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("empty Favorites Repository")
    public void givenEmptyFavoritesRepository() {

    }

    @When("Going Postal is added to Favorites Repository")
    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    @Then("favorite book is Going Postal")
    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }
}

Mapping is done using regexp by default. The same text must be used in stories and annotations (it also might be bound to variables). When JBehave doesn't find corresponding mapping – it prints warning message. This message contains also template methods which might be copied to mapping classes. This really speeds up development. Another nice feature is @Pending annotation – steps annotated with it are simply skipped (think about it as “not implemented” or “in progress”).

JBehave reports are highly customizable. All HTML templates might be defined using Freemarker. There is also a default set of templates embedded into JBehave, which is used here:

JBehave&apos;s report

JBehave is a very flexible and expendable framework. It integrates with numbers of different programming languages (Groovy, JRuby, Java), technologies and frameworks. Stories are clear for business world and mapping files allows translating them into Java easily. Reporting functionality is really great. Defining own templates with Freemarker makes it easy to adopt to corporate standards. JBehave's configuration is a bit overwhelming, but it needs to be done only once.

Pros/Cons:
+ Highly configurable
+ Customizable report templates
+ Support for many languages and technologies
-  Configuration is a bit overwhelming

Cucumber

Cucumber was originally written for Ruby. After some time additional plugins and runners were added. Cucumber has much simpler configuration. Just like in JBehave writing test is divided into steps – writing stories, mapping them into Java (also called glue code), running and reporting. Stories look very similar:

Feature: Favorites Repository

Scenario: Empty Favorites Repository
Given empty Favorites Repository
When there is nothing added to the repository
Then favorite book is null

Scenario: Going Postal added to Favorites Repository
Given empty Favorites Repository
When Going Postal is added to Favorites Repository
Then favorite book is Going Postal

All scenarios for one feature reside in one place. The glue code is pretty similar to JBehave's:

@RunWith(Cucumber.class)
@CucumberOptions(format = { "pretty", "html:target/cucumber", "json:target/cucumber.json" }, glue = "pl.payu.dictionary.client.exception.steps")
public class CucumberTest {

}
public class EmptyRepositoryStepDefinitions {

    private FavoritesRepository repository = new DefaultFavoritesRepository();;

    @Given("^empty Favorites Repository$")
    public void Empty_Favorites_Repository() throws Throwable {
        repository = new DefaultFavoritesRepository();
    }

    @When("^there is nothing added to the repository$")
    public void there_is_nothing_added_to_the_repository() throws Throwable {

    }

    @Then("^favorite book is null$")
    public void Favorite_book_is_null() throws Throwable {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
public class GoingPostalRepositoryStepDefinitions {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    @When("^Going Postal is added to Favorites Repository$")
    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    @Then("^favorite book is Going Postal$")
    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }

}

The default reports looks really great:

Cucumber&apos;s report

Cucumber&apos;s report

Cucumber is a very powerful framework for BDD testing. It has (just like JBehave) many useful features like testing by example or parameters. Both features and glue code looks very nice and might be maintained easily. Additionally Cucumber supports many different languages and platforms like Ruby, Java or .NET. Additionally Arquillian support comes out of box and there is also very useful Jenkins plugin for generating reports.

Pros/Cons:
+ Simple but powerful
+ Jenkins reporting plugin
+ Support for many languages
-  Not so easy to embed in your own report templates

Concordion

Concordion is very similar to Cucumber but features might be written in HTML fashion. It also supports many different platforms and languages.

Here are 2 stories written in HTML and an “index” page:

<html xmlns:concordion="http://www.concordion.org/2007/concordion">
    <head>
        <title>Concordion tests</title>
    </head>

    <body>

    <h1>Concordion tests</h1>

    <p>
        <ul>
            <li>
                <a concordion:run="concordion" href="EmptyFavoritesRepository.html">Empty Favorites Repository</a>
            </li>
            <li>
                <a concordion:run="concordion" href="GoingPostalAddedToFavoritesRepository.html">Going Postal Added To Favorites Repository</a>
            </li>
        </ul>
    </p>

    </body>
</html>
<html xmlns:concordion="http://www.concordion.org/2007/concordion">

    <head>
        <title>Favorites Repository</title>
    </head>

    <body>

        <h1>Empty Favorites Repository</h1>

        <p>
            Given empty Favorites Repository <span concordion:execute="emptyFavoritesRepository()" /><br/>
            When there is nothing added to the repository <span concordion:execute="thereIsNothingAddedToTheRepository()" /><br/>
            Then favorite book is null <span concordion:execute="favoriteBookIsEmpty()" /> null.
        </p>

    </body>

</html>
<html xmlns:concordion="http://www.concordion.org/2007/concordion">

    <head>
        <title>Going Postal added to favorites repository</title>
    </head>

    <body>

        <h1>Going Postal added to Favorites Repository</h1>

        <p>
            Given empty Favorites Repository <span concordion:execute="emptyFavoritesRepository()"/><br/>
            When Going Postal is added to Favorites Repository <span concordion:execute="whenGoingPostalIsAddedToFavoritesRepository()"/><br/>
            Then favorite book is <span concordion:execute="thenFavoriteBookIsGoingPostal()"/> Going Postal.
        </p>

    </body>

</html>

The glue code does not contain method annotations. Concordion simply use method names:

@RunWith(ConcordionRunner.class)
public class FavoritesRepositoryTest {

}
@RunWith(ConcordionRunner.class)
public class EmptyFavoritesRepository {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    public void emptyFavoritesRepository() throws Exception {

    }

    public void thereIsNothingAddedToTheRepository() throws Exception {

    }

    public void favoriteBookIsEmpty() throws Exception {
        assertThat(repository.getFavorite(Book.class)).isNull();
    }
}
@RunWith(ConcordionRunner.class)
public class GoingPostalAddedToFavoritesRepositoryTest {

    private FavoritesRepository repository = new DefaultFavoritesRepository();

    public void emptyFavoritesRepository() throws Exception {

    }

    public void whenGoingPostalIsAddedToFavoritesRepository() {
        Book book = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        repository.putFavorite(book);
    }

    public void thenFavoriteBookIsGoingPostal() {
        Book goingPostal = new Book("9781407035406", "Terry Pratchett", "Going Postal");
        assertThat(repository.getFavorite(Book.class)).isEqualTo(goingPostal);
    }

}

Default styles and above HTML files doesn't look very nice, but can be easily adjusted to corporation standards:

Concordion&apos;s report

Concordion&apos;s report

Concordion&apos;s report

The main difference between Concordion and Cucumber is how it uses mapping fies. Concordion uses method names for resolution and Cucumber - Regexp expressions. All major features like testing by example or parameters are also implemented.

Pros/Cons:
+ The most customizable implementation for writing scenarios
+ Support for many languages and platforms
- Fewer plugins available

Summary

There are many BDD frameworks on the market. Some of them are designed only for specification testing and some are very flexible and support many different features.

My team is responsible for maintaining several JEE projects. Some of them contains very sophisticated and complicated business logic and all of them are written in pure Java. This is why we discarded JDave and EasyB at the beginning. Next we realized, that we don't need to use our own CSS files in the reports – this is why we discarded Concordion. Finally we had to choose among JBehave and Cucumber. We decided to use Cucumber because of Arquillian support out of box and Cucumber offers simpler bootstrapping. Now we plan to adjust Cucumber reporting plugin to use with Bamboo... Wish us luck!

References

  • Code for all examples- https://github.com/altanis/bdd-examples
  • JDave - http://jdave.org/
  • Concordion - http://www.concordion.org
  • Easyb - http://easyb.org/
  • JBehave - http://jbehave.org/
  • Cucumber - http://cukes.info/
Framework Cucumber (software) Testing Repository (version control)

Opinions expressed by DZone contributors are their own.

Related

  • WebDriverIO Integration With Cucumber
  • Design Patterns for Scalable Test Automation Frameworks
  • COM Design Pattern for Automation With Selenium and Cucumber
  • The Power of Docker and Cucumber in Automation Testing

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: