Monitoring CI/CD Workflows

Summary

CI/CD is a method to frequently deliver apps to customers by introducing automation into the stages of app development.

It is the brief definition of CI/CD from the Red Hat community. So, CI/CD pipelines can be defined as the steps for developers to deliver a new software version or product. In the CI part, pipelines build code and run tests; in the CD part, pipelines deploy and deliver the new version.

That can be hard to monitor your pipeline progress and test runs in this context. In this article, we will examine how to monitor a CI/CD workflow that uses Github Actions by using the Foresight product of Thundra, which I am currently an employee of.

Open Source Projects

We will examine two open-source projects in this section. These are Payara and dcm4che in Github. I chose them for this article because they have significant build and test run time so we can understand the difficulty of monitoring workflows.

Payara Server is a middleware platform that supports reliable and secure deployments of Java EE and MicroProfile applications in any environment.

dcm4che is a collection of open-source applications and utilities for the healthcare enterprise.

Actually, we don't need to know what these projects do, we are interested in creating workflows for them.

We cannot use this projects' workflows for this demo so, we will create mock workflows. In this case, we will create 2 different workflows for each project. One of them is for building projects and another one is running tests.

Using separate workflows for test and build is not a best practice but we do not have a chance to run the Continuous Development step and deploy this code anywhere.

After forking these projects, create 2 workflows for each. The first one does not run tests and only builds the project.

name: Java Build with Maven
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up JDK 11
      uses: actions/setup-java@v2
      with:
        java-version: '11'
        distribution: 'adopt'
        cache: maven
    - name: Build 
      run: mvn clean install -Dmaven.test.skip

Also, you can check the dcm4che and Payara repositories that were forked by me. 

Another workflow is almost the same as this but we only will remove the 'Build with Maven' step and replace it with below:

name: Test Run Only
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up JDK 1.8
      uses: actions/setup-java@v2
      with:
        distribution: 'adopt'
        java-version: '8'
        cache: maven
    - name: Tests 
      run: mvn test

After creating workflows it's time to monitor them, we will use Foresight for this. Foresight is a tool developed by Thundra and you can monitor and optimize your CI/CD pipelines with this tool. For more information, you can check the docs from here.

Instrumenting Thundra Foresight

You can easily sign-up and login to Thundra, and after signing in, choose Foresight on the application screen.

Monitoring Your Workflows

You will be redirected to the repositories page. Click connect repositories on this page, then create a foresight project and connect with easy set up one of the repositories that we forked. 

Connect Repository screen.

After a short time, our workflow info will appear in our project. We can inspect the projects' workflow success and average duration day by day. Each point in these graphs is the average duration and success rate for only one day. If your workflows have only been working for one day, you will see only one point on graphs.

White Line: Test runs only

Blue Line: Build project without tests

Yellow Line: Thundra Foresight Instrumentation

There are 5 different points in these graphs so I have been running them for 5 days. Each point is the average of that day.

From this graph, we can conclude test runs take almost the build time long, as you can see in this graph.

Observing Multiple Projects

It is possible to connect multiple repositories in a single Foresight repository, you can choose the same project after clicking connect repository.

Completed example at below, you can see the 2 repositories connected to the one Foresight project. Also, dcm4che has been integrated for monitoring the test runs.

Multiple repositories connected to the single Foresight project.

Monitoring Test Runs

Another thing that we can monitor here is test runs on our projects. We need to make some configurations. To do this, go to connect repository screen again and choose Test Monitoring this time, under the GitHub section we will use Java and Maven.

There are 2 variables; THUNDRA_APIKEY and THUNDRA_PROJECT_ID. These are needed for sending test run data to the correct project (your Foresight project).

Now we will update our workflow as shown as, I highly recommend using these variables as a GitHub secret. But If you only making a demo, you can use hard coded as shown in the integration screen on Thundra Foresight.

name: Java Test with Maven
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 1.8
        uses: actions/setup-java@v2
        with:
          distribution: 'adopt'
          java-version: '8'
      - name: Thundra Maven Test Instrumentation
        uses: thundra-io/thundra-maven-test-action@v1
        with:
          apikey: ${{ secrets.THUNDRA_API_KEY }}
          project_id: ${{ secrets.THUNDRA_PROJECT_ID }}
          command: mvn test

Updating Java Test with Maven workflow is enough in our case because it is running test only. After a little while running the updated workflow, you can see the test runs in your project.

As the picture below, multiple repositories connected in a single Foresight project; dcm4che also integrated for test runs, Payara has failed workflow runs:

Multiple repositories that observed for test runs.

Conclusion

Monitoring automated tests is a pretty painful job, there might be some flaky tests that exist. When you schedule workflows to run with Foresight, it is so easy to detect unwanted activities in your tests. On the screen below, we can examine the latest test runs in our dcm4che project. As you can see, there are no flaky tests. All tests are passing in all test runs, there are no changes in failed or skipped test counts: