Getting Started With Cucumber on CircleCI

Cucumber is a behavior-driven development (BDD) tool to develop acceptance tests for web apps. For the purposes of this article, I’ll assume that your CircleCI setup is running smoothly and focus on what Cucumber is, how to use it well, and how to get it integrated with your CircleCI setup.

What Is Cucumber, Anyway?

Cucumber is a BDD tool that you can use to verify your app’s functionality. Cucumber tests are written in a language called Gherkin and express the desired behavior and the expected result of the software in a natural, language-like syntax. BDD differs from traditional automated tests in that the tests are neither code-level unit tests nor application-level scripts to interact with the UI. BDD tests express a set of preconditions for the test, then express the test itself, then the expected result.

Combined with some documentation (which can be easily formatted in a Gherkin-friendly way), a set of Cucumber BDD tests can also serve as a comprehensive set of documentation to review with Product Owners or for developers and QA staff to review in the future. For teams whose main language is not English, Gherkin supports numerous languages.

Given-When-Then

The basic structure of a Cucumber test is the given-when-then format.

• “GIVEN” is the setup for the test. You can use GIVEN to set up the type of data you need for the test. You use GIVEN to set up the state.

• “WHEN” is the action of the test. WHEN describes the action that you want to be performed against the state you set up in the GIVEN portion of the test.

• “THEN” is the expected outcome. You use THEN to state what you want the outcome of the test to be.

Note that GIVEN, WHEN, and THEN all support multiple steps using the “AND” keyword. Here’s an example:

• GIVEN Jane has an active profile
• AND her profile has Sam for a friend
• AND her profile does not have Bill for a friend
• WHEN she enters a new public status update
• THEN Sam should see her new public status update
• AND Bill should not see her new public status update

Features and Scenarios

A giant pile of given-when-then steps would get unwieldy very quickly. Gherkin provides a way to organize tests into features and scenarios. A feature is a high-level category and maps nicely to a user story or whatever your team uses for the fundamental unit of development. A feature is made up of one or more scenarios and a scenario is made up of the given-when-then steps. (The above given-when-then example is a scenario.) You can think of a scenario as a test case and a feature as a logical set of test cases.

Step Definitions and Hooks

The tool requires some back-end coding in order to run its tests properly. Each step in a given-when-then needs a step definition in order to be executed. You can set up the step definitions using a regular expression that will match to your steps, and a block of Ruby code to execute when the regex is matched. Step definitions can be reused across scenarios.

Hooks are how you set up and tear down your test state before and after test runs, particularly to keep your test database in good shape.

Integrating With CircleCI

Integrating your Cucumber tests with CircleCI is very straightforward. Cucumber has a few formatter plugins for test result output. Use the JUnit formatter and configure Cucumber to output to the $CIRCLE_TEST_REPORTS/cucumber directory. The CircleCI documentation page suggests the following YAML be added to your circle.yml:

test:
  override:
    - mkdir -p $CIRCLE_TEST_REPORTS/cucumber
    - bundle exec cucumber --format junit --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml

Pretty simple. The documentation page also shows how to configure your circle.yml file if you prefer the JSON formatter.

This configuration allows CircleCI to see whether or not your Cucumber tests are passing or failing, and you can configure CircleCI on how to respond to failed Cucumber tests.

Cucumber is a powerful tool for expressing the desired behavior of your software system. Integrating it into your CircleCI builds can bring a lot of value to your development process.