Trust Your Pipeline: Automatically Testing an End-to-End Java Application

Disclaimer: This post is part of the same presentation given by myself and Bruno Souza at the JavaOne Conference 2017 in San Francisco.

Although extremely important, we are not talking here about unit testing and integration testing, assuming you already know what it is and already apply it, as a developer concerned about the quality of your code.

Test Automation Pyramid and Agile Testing Quadrants

Creating tests is the easiest and quickest way to test your application and ensure that if any bug appears, you will find it before your client.

Currently, we cannot talk about testing without talking about test automation. The application of the test automation gives a quick feedback to the team and maintains the execution of the regression tests continuously.

The approach we can use to take a greater speed in automating, executing, and having a rapid feedback of the tests is the application of the Test Pyramid, which is a guide for the application of automation in at least three levels : unit, services , and UI (user interface).

The services layer is divided into three parts: component tests, integration tests, and API tests.

The unit testing layer is the most important layer of our application because we will create tests for the code that we are developing and guarantee its work as expected, even after future maintenance (recommending the use of TDD — Test Driven Development). In this layer, we can apply code coverage analysis and static analysis practices in order to intensify the rapid feedback against a defect that may appear.

The service layer (component, integration and API) is extremely important nowadays, with a large focus on API testing. Here we apply mocks, stubs and fakes to give speed in the execution of microservice tests. We also need separate test environment servers, where this can be the closest to a production environment.

The UI (User Interface) layer is also important, especially from a mobile testing perspective where, when a customer encounters an error in an app, it usually removes it. The most important techniques here are automated testing in UI and Visual Regression Testing [3]. In the web part we need browsers to execute the same test in different ones (IE, Chrome, Firefox, Safari). We need the same for mobile automation testing: iOS and Android devices to ensure the compatibility of our app on these two platforms.

The Agile Testing Quadrant, created by Brian Merick and widely disseminated by Lisa Crispin and Janet Gregory in his book “Agile Testing — A Pratical Guide for Testers and Agile Teams” are some practices that can be applied during activity-focused development of tests. The quadrant is a guide: it is not necessary to perform all the existing practices in it, you can choose one or more depending of your context.

Continuous Delivery and Testing Pipeline

There is no way to talk about DevOps without talking about Continuous Delivery (CD). Without it, we could not even talk about DevOps culture. In Continuous Delivery, one of the foundations is Continuous Testing, where we must test all the stages of our development (pipeline) with an initial recommendation applied to unit tests and automated acceptance.

Continuous Delivery enables joint roles between Development, QA, and Operations. An example of evidence-focused collaboration for these roles is:

• Development + QA: Build, Deploy and Test automation at various levels

• QA + Operations: Test Automation and Continuous Feedback through test executions, as well as sanity test executions

• Operations + Development: Automated provisioning of machines/containers required for testing at any level.

Now we continue with the pipeline focused on tests, which can be applied in whole or in parts, being:

• Unit: Unit Tests

• Integration: Integration Tests

  • We can create mocks/fakes/stubs to remove the dependencies and accelerate the test executions.

• Service: Test on service layer (SOAP, REST)

  • Smoke: small execution subset to guarantee that all API’s are working (at least return status different from HTTP 404 or HTTP 500).

  • Contract: a collection of agreements (tests) between a client (consumer) and an API.

  • Functional: tests that want to guarantee the operation against different business rules (happy path, sad path, and alternative flows).

  • Acceptance: evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery.

• Acceptance: acceptance tests (those that focus on end-user usage, commonly called E2E).

  • Smoke: main test suite that will guarantee your business running.

• Functional: tests that will ensure operation against different business rules (happy path, flow of exception, and alternative flows).

  • Smoke: main test suites of happy path, sad path and alternative flows.

From Integration to Functional Testing (end of the pipeline) we have to worry about non-functional tests. Examples of tests are functional: performance, load, security, etc, and it’s extremely necessary to create an entire automated test architecture to support the continuous, automated, and least-maintenance run possible with:

• Screenshots to evidence the execution of each test, or evidence when an error occurs.

• Logs to analyse and see any error occurred.

• Reports to show up a feedback about the test execution.

• Data management for the sensitive data on a test script.

• Parameterize commonly changed data like URL's, endpoints, etc…

Toolbox for Automated API, Web, and Mobile Tests

In order to automate an API, a web page, and a mobile front-end, there are open source tools that will help you to quickly and easily build and run tests.

Rest Assured

A tool for creating automated tests for an API (REST and XML). Rest Assured uses an easily and understood DSL based on Gherkin (Given-When-Then).

In the example below, it is possible to see the API through a local endpoint (simulating a production environment) and a mock endpoint created with Java Spark. Creating a mock API by developing the API fixed data returns can give even greater speed in the execution and validation of the different aspects that secure the tests for microservices, especially about contract tests.

@Test
public void getPersonById() {
    int personID = 
        given()
            .contentType(ContentType.JSON)
            .body(new Person("Elias Nogueira", "RS", "Automate tests")).
        when().
            post("person").
        then().
            extract().
                path("id");

    when().
        get("person/{id}", personID).
    then().
        contentType("application/json").and().
        body("name", equalTo("Elias Nogueira")).and().
        body("address", equalTo("RS")).and().
        body("hobbies", equalTo("Automate tests")).and().
        statusCode(200);
}

Selenium WebDriver

Selenium is the best-known tool for automation of a web page. It also has an easy DSL and is based on four steps for automation:

• Navigation: actions like access a web page, forward, back, and refresh.

• Interrogation: ways to find web elements like by id, name, cssSelectors, and other locators.

• Manipulation: a way to interact with an element like click, fill (sendKeys), clear, and get text.

• Synchronization: ways to wait for some asynchronous actions, like an element that appears after some seconds.

It is a W3C standard and performs actions in web browsers simulating a real browser. For this to be possible, it is necessary to use the browsers drivers.

@Test
public void addPersonSuccessfully() {

    System.setProperty("webdriver.chrome.driver", "/Users/eliasnogueira/Selenium/chromedriver");

    WebDriver driver = new ChromeDriver();
    WebDriverWait wait = new WebDriverWait(driver, 10);
    driver.get("http://localhost:8888/javaone");

    By addButton  = By.id("add");
    wait.until(ExpectedConditions.presenceOfElementLocated(addButton));
    driver.findElement(addButton).click();

    wait.until(ExpectedConditions.presenceOfElementLocated(By.id("back")));
    driver.findElement(By.id("name")).sendKeys("Daenerys Targaryen");
    driver.findElement(By.name("address")).sendKeys("Dragonstone");
    driver.findElement(By.cssSelector("input[ng-model='post.hobbies']")).sendKeys("Break Chains");
    driver.findElement(By.cssSelector(".w3-btn.w3-teal")).click();

    wait.until(ExpectedConditions.presenceOfElementLocated(By.id("address")));
    String dataOnPage = driver.getPageSource();
    assertTrue(dataOnPage.contains("Daenerys Targaryen"));
    assertTrue(dataOnPage.contains("Dragonstone"));
    assertTrue(dataOnPage.contains("Break Chains"));

    driver.quit();
}

Appium

Appium is an open source tool with the same Selenium DSL, but for automation in native or hybrid mobile device apps for iOS or Android.

It supports execution on emulators, real device or test lab (cloud), and, in conjunction with Selenium Grid, gives the possibility of creating an internal device grid.

@Test
public void addPerson_Successfully() throws MalformedURLException {
    File app = new File("src/main/resources/app/workshop.apk");

    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, MobilePlatform.ANDROID);
    capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
    capabilities.setCapability(AndroidMobileCapabilityType.APP_PACKAGE, "com.eliasnogueira.workshop");
    capabilities.setCapability(AndroidMobileCapabilityType.APP_ACTIVITY, "activities.ListActivity");

    AndroidDriver<MobileElement> driver = new AndroidDriver<>(new URL("http://127.0.0.1:4723/wd/hub"), capabilities);
    WebDriverWait wait = new WebDriverWait(driver, 20);

    wait.until(ExpectedConditions.presenceOfElementLocated(By.id("com.eliasnogueira.workshop:id/fab")));
    driver.findElement(By.id("com.eliasnogueira.workshop:id/fab")).click();

    driver.findElement(By.id("com.eliasnogueira.workshop:id/txt_nome")).sendKeys("Jon Snow");
    driver.findElement(By.id("com.eliasnogueira.workshop:id/txt_endereco")).sendKeys("The wall");
    driver.findElement(By.id("com.eliasnogueira.workshop:id/txt_hobbies")).sendKeys("Know nothing");
    driver.findElement(By.id("com.eliasnogueira.workshop:id/button")).click();

    wait.until(ExpectedConditions.presenceOfElementLocated(By.id("android:id/search_button")));
    driver.findElement(By.id("android:id/search_button")).click();
    driver.findElement(By.id("android:id/search_src_text")).sendKeys("Jon Snow");

    String texto = driver.findElement(By.id("android:id/text1")).getText();
    assertEquals("Jon Snow", texto);

    driver.quit();
}

Applied Pipeline GitHub Repo

The code for all projects can be found in this repository. There are test suites that do the internal sequencing of the pipeline at each test level. For example: for API tests, there are three sequential tests: smoke, functional in mock and functional in production.