Six months of Behat

We thought about using Behat for our acceptance tests after PHP.TO.START  in Turin; Behat is the PHP version of Cucumber. After a while, I reused the objects that our first PHPUnit end-to-end tests were calling to work with the system to prepare a basic FeatureContext able to perform a payment with Onebip; several weeks later, we had executable specifications in place for the first Spain-based tests.

Fast-forward six months

Now we have more than an hour worth of Behat specifications being verified in our integration servers, where four different projects are deployed alongside each other. We took several measures for speeding up the process (and with that our daily deployments):

• improving the hardware of the server and that of our development machines.
• experimentally, run multiple Behat processes in parallel.

The test suite is simulating more than one hundred scenarios, each going from a single purchase to 3 or 4 weeks of time where a subscription is renewed. When you are simulating this much work you're winning over manual testing all the time; however we resisted, at least for now, the movement of older and rarer scenarios to a nightly build that does not get in the way of deployment.

The feedback of a nightly build is very slow, because on average you're going to find out if you broke some test after 12 hours.

Reproducibility

We started with scenarios all similar to each other:

Given I am in ES
And I am a Vodafone user
When I subscribe to ES_SOME_SERVICE
Then I am billed 3.00 EUR
And the merchant is notified with SUBSCRIPTION_ACTIVATED

and we used those to integrate the different components of the distributed system that Onebip is today. Different code is run depending on the country and the mobile phone operator, since each has its own interface for payments from where we take the money. So there is much value in repeating these tests over all new developed countries and connections.

From there, we transitioned also to user experience scenarios. Even if we have just a single user-facing page (the payment one), there are several possible corner cases you may want to test in an end-to-end fashion:

Given I am in ES
And I am a Vodafone user
And I walked in on ES_SOME_SERVICE
And I received a pin
When I walk in again on the same service
And I insert the previously received pin
Then I am billed 3.00 EUR

The difficulties

It's not easy to pull off a stable end-to-end test suites. Behat forces automation since you can't put while() and complex code inside feature files, but it's up to you to perform the correct waiting conditions instead of sleep()ing a fixed amount of time. Wherever there are more than one component involved, asynchronous API get in the way, requiring asynchronous assertions.

Here are the most common issues we encountered with our executable specifications:

• synchronizing cron processes, which must be called manually and filtered to act only on the current test entities (such as the current phonenumber or user subscription). You have to disable all the crons and jobs scheduled at fixed times before running the suite.
• A related issue is the needed investment for simulating business time: injecting a clock in the applications that is persistent on a MongoDB collection and that can be brought forward through an API.
• Maintaining the discipline between backdoors (querying the databases of the different integrated components directly to quickly implements a test) and acting through APIs (more flexible and avoiding backward incompatible changes in the future, even if this is a client that we control totally.)
• Keeping all projects up to date and with versions identical to production; we will extract Behat tests in a separate project since they actually test the whole application, not the single end-user facing one.
• Finding out the project that broke the pipeline! We realized in the last retrospective that all projects need to converge to an integration phase in CI, they're not independent anymore (but they can be deployed independently after that phase, one at a time.)

Writing scenarios: how

Still, even if they're heavy to run (but not to maintain) you want scenarios to test one thing and have one, maximum two reasons to fail. For example, we bundle billing and merchant notifications of the event since they are closely related; but text messages received by the user or retry policies are separated in their own scenarios.

You still want to eliminate much of the duplication inside scenarios. However,  you cannot remove duplication by extracting methods in the feature file, since it's a totally different style than an xUnit test. Then steps can easily by divided into scenarios by not making them in a single battery but splitting them up; however even big steps such as Given I have been subscribed for two weeks should be available, with reasonable defaults with their parameters such as the country of the user.

This choice lets you trasform>

Given I am in IT
And I am a Vodafone user
And I subscribe to SERVICE_NAME
And I am billed
And 7 days have passed
...

into

Given I have been subscribed for 7 days
...

more precisely representing that is really important about the initial conditions.

Like the Etsy guys say , your system should be so easy to use that a bot can navigate in it. For us this was easy on the front end level as our payment page has one or two clicks as the main actions; the complex part was automating the backend interactions with carriers.

Writing scenarios: who

We're still not at the level of writing scenarios directly with the customer. We are writing the user story together for now, which is a big improvement over the "requirements collection" phase where a bunch of Excel files and informal Word documents are passed around.

When we are in the same office with those internal customers it becomes a lot easier, but it's not always possible. Since the customers almost never go to Agile events and conferences (maybe they should), it's the technical team's responsibility to go look for them and start the collaboration that is part of the Manifesto. BDD enables this collaboration by featuring a set of examples as the knowledge produced by the encounter between tech and business people, and that's the next step we want to take.