How to Make Functional and Performance Testing Easy With Open-Source Testing Tools

Functional testing and performance testing are two important parts of testing your website or app and for discovering bottlenecks, errors, and bugs. “Shifting left” and connecting the two types of tests in the Continuous Integration process improves your product’s quality and lowers costs. Running the tests separately takes more time, requires more meticulous work from you, and prevents you from cross-analyzing trends and data through rich reports.

Taurus is an automation testing tool that makes it easy to configure test suites in a human-readable format, YAML, or JSON and integrate them in your Continuous Delivery process. With Taurus, the tests can be run automatically and in parallel.

In this blog post, I will show you how to:

• Define and run functional tests.

• Define and run performance tests.

• Integrate tests in a Continuous Delivery cycle with Jenkins.

Defining and Running Functional Tests: A Selenium Tutorial

Our test scenario is run on the blazedemo.com site. In the scenario, a user comes to the site, selects a flight, and purchases a ticket. Here’s how to do this in three steps:

Step 1: Record the Use Case With Selenium

In this example, the use case we're interested in is a successful ticket purchase.

The simulated user loads the index page, chooses departure and destination, and presses the button to continue to the checkout page. The user enters their credit card data and presses the button to purchase a ticket to the flight. The website should then show them a confirmation page.

You can record a Selenium script with Selenium IDE for this process, and add it to Taurus.

Step 2: Add the Tests to Taurus

You can run the Selenium scripts through Taurus by invoking the Selenium executor and adding them to the config file:

---
execution:
- executor: selenium
  # functional tests
  scenario:
    script: selenium/usecase-purchase.py

Save the script as a YAML file, let’s say, example.yml.

Open your terminal and run. As simple as that, Taurus will run it.

Step 3: Write Tests Natively in Taurus

The YAML or JSON format parsed by Taurus allows you to define each step of our use case above:

---
execution:
- concurrency: 10
  hold-for: 10s
  steps: 4
  scenario: usecase-purchase


scenarios:
  usecase-purchase:
    timeout: 5s
    retrieve-resources: true
    store-cache: false
    store-cookie: false
    default-address: http://blazedemo.com
    headers:
      User-Agent: 'Mozilla/5.0(X11;Linuxx86_64)AppleWebKit/537.36'
      Accept-Language: 'en-US'
      Accept-Encoding: 'gzip,deflate,sdch'
      Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp'
    requests:
    - /index.php
    - url: '/reserve.php'
      method: POST
      headers:
        Content-Type: application/x-www-form-urlencoded
      body:
        fromPort: Philadelphia
        toPort: Rome
    - url: '/purchase.php'
      method: POST
      headers:
        Content-Type: application/x-www-form-urlencoded
      body:
        airline: United Airlines
        flight: '234'
        fromPort: Philadelphia
        price: '432.98'
        toPort: Rome
    - url: '/confirmation.php'
      method: POST
      headers:
        Content-Type: application/x-www-form-urlencoded
      body:
        inputName: John Doe
        address: A Test Street
        city: Anytown
        state: SomeState
        zipCode: '54321'
        cardType: Visa
        creditCardNumber: '7463385993'
        nameOnCard: John Doe
        rememberMe: 'on'

While this step is optional for our example use case, you’ll find it to be extremely beneficial if you need to test performance.

Performance Testing Tutorial

You may want tests to ensure that the pages of your site will work as expected within reasonable response or load times. This means that you’ll decide the expected times for each page you want to check, and set one or more thresholds in Taurus for that page. If the threshold condition is reached, the test will be marked as failed and you can tell Taurus to take an action.

Performance testing using thresholds allows you to know immediately when you introduce a change that is too slow, like a terrible query that will slow down your application (I know, you'd never do such thing, but you can make your pipeline future-proof!).

You can do this in Taurus using the Pass/Fail module. For simplicity, we use the same scenario as above and set values for the average response time of each page. Taurus will check and record the values for each pass. We can add to the above YAML:

reporting:
  - module: passfail
    criteria:
      - avg-rt of IndexPage>10ms for 1s, continue as failed
      - avg-rt of ReservePage>15ms for 1s, continue as failed
      - avg-rt of PurchasePage>15ms for 1s, continue as failed
      - avg-rt of ConfirmationPage>15ms for 1s, continue as failed

In order to use these descriptive labels, we need to define them for the respective pages with the attribute label. So we modify the initial YAML config to read in the pertinent part:

    requests:
      - label: IndexPage
        url: '/index.php'
      - label: ReservePage
        url: '/reserve.php'
      - label: PurchasePage
        url: '/purchase.php'
      - label: ConfirmationPage
        url: '/confirmation.php'

The average response time is only one of the thresholds you can set. You can use others, such as fail:

    criteria:
      - fail of ConfirmationPage>50% for 10s, stop as failed

This is Taurus report screen for these example values:

You can set the values to your liking, to serve your purpose. To find information about all the thresholds implemented in Taurus, and the explanation of each parameter, please see the Pass/Fail page in the documentation.

Adding Tests to Jenkins CI

By adding testing to Continuous Integration platforms like Jenkins, you will be able to see at any time if everything is still okay, for any commit you make. All you need to do is add a few commands to run BZT from the shell. You will be then able to see if your performance tests succeed or fail, giving you feedback every time your team changes something in your web application that can make things slower than the thresholds. This helps significantly because you know exactly which pull request caused the slowing down, and where.

Congratulations! You are now ready to define and run functional and performance tests and to integrate them in the CI cycle.

Learn more about automating functional and performance tests from our free webinar.