In a previous post, we talked about how we can check our Docker images for any known vulnerabilities by means of Anchore Engine. This still required a manual action. Wouldn’t it be great if we could incorporate Anchore Engine into our Jenkins CI build job or pipeline? In this post, we will take a look at how we can accomplish this by means of the Anchore Container Image Scanner Jenkins Plugin.

What Is Anchore Engine?

Anchore Engine is an open source tool that scans your Docker images for security vulnerabilities. Definitely a powerful tool when you are running Docker containers in production. With Anchore Engine, your Docker images are automatically checked whether they are up-to-date before you run them into production. For a more elaborate explanation, we refer to our post about Anchore Engine. It is advised to read this post, because we will build upon what we have learned there.

Prerequisites

Before we can experiment with the Anchore Plugin, we need to have some things in place:

• We need a running Jenkins CI server. If you don’t have access to a Jenkins CI server, you can set up a Jenkins instance quite easily.
• We need a running Anchore Engine. Check out our previous post about Anchore Engine and how to install it.
• We will use a Docker image we created for a previous post, which contains a Spring Boot MVC application and which is based on the openjdk:10-jdkDocker image. The openjdk:10-jdk Docker image is not updated anymore because it is not a Java LTS release and currently openjdk:11-jdk should be used. However, this will give Anchore Engine the opportunity to report vulnerabilities which we can try to solve.
• A Jenkins Freestyle job (The Anchore Plugin is only supported for Freestyle or Pipeline jobs) needs to be configured which builds the Spring Boot MVC application with branch feature/anchore and pushes it to a Docker repository as version 0.0.4-SNAPSHOT. We will use DockerHub as our Docker repository. The sources can be found at GitHub.

In our setup, we are running a Jenkins Docker container and an Anchore Engine Docker container. Jenkins will need to have access to the Anchore Engine. By default, this is not possible when both are running in separate Docker containers because of security. Therefore, we need to ensure that both are running in the same network. The Anchore Engine Docker containers (the Anchore Engine and the corresponding database) already run in their own network. The only thing we need to do is to start the Jenkins Docker container also in this network.

$ docker inspect containerId

A JSON output is shown, navigate to the NetworkSettings section:

"NetworkSettings": {
    ...
    "Networks": {
        "anchorevolume_default": {
            ...
        }
    }
}

In our case, the network name is anchorevolume_default. Now, start the Jenkins Docker container with the parameter --network=anchorevolume_default. Your Jenkins Docker container will receive another IP address which you can also retrieve with the docker inspectcommand.

Install and Configure

Go to "Manage Jenkins – Manage Plugins," select the "Anchore Container Image Scanner" plugin, and click the "Install without restart" button:

Go to "Manage Jenkins – Configure System" and navigate to the "Anchore Plugin Mode" section. Fill in the following fields:

• Engine URL: the URL where the Anchore Engine is accessible. You can use the docker inspect command when running Anchore Engine as a Docker container in order to retrieve the IP address (parameter IPAddress in the Networks section).
• Engine Username: the Anchore Engine user name.
• Engine Password: the Anchore Engine password.

Add Anchore to Build Job

The Anchore Plugin will scan the workspace for a file named anchore_images. This file must contain the Docker image name, tag and, since it is a Docker image we create ourselves, also the location to the Dockerfile. After our Maven build step, we, therefore, need to add a build step in our build job for creating this file. Add a Execute shell build step:

The complete command we use:

echo "mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT" ${WORKSPACE}/Dockerfile > anchore_images

After the build step to create the anchore_images file, we add the "Anchore Container Image Scanner" build step. We leave the default settings as is.

Run the build. The Anchore Plugin will instruct Anchore Engine to analyze the Docker image. It will then poll the status of the analysis until the analysis has finished or until the Anchore Engine operation retries count is hit. In the latter case, the build will fail with the following error:

2019-01-01T11:43:06.107 INFO AnchoreWorker Waiting for analysis of mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT, polling status periodically
2019-01-01T11:49:42.151 WARN AnchoreWorker anchore-engine get policy evaluation failed. HTTP method: GET, URL: http://172.18.0.3:8228/v1/images/sha256:b0d228be88502a14d4a30d5f431504f9435e2af369c374f2804ccdabc98d16d9/check?tag=mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT&detail=true, status: 404, error: {
"detail": {},
"httpcode": 404,
"message": "image is not analyzed - analysis_status: analyzing"
}

The solution is to increase the timeout parameter and a faster machine for Anchore Engine also might help.

When the image is successfully analyzed, our build fails because of serious vulnerabilities. This is also what we expected since we are using the openjdk:10-jdk Docker image. This is shown as follows in the Jenkins output build log:

2019-01-01T12:33:16.886 INFO AnchoreWorker Submitting mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT for analysis
2019-01-01T12:33:20.357 INFO AnchoreWorker Analysis request accepted, received image digest sha256:6142342c2f9baa7b2d2ecae8c67c49353690808c215b347d46ad2b22aebd5af1
2019-01-01T12:33:20.358 INFO AnchoreWorker Waiting for analysis of mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT, polling status periodically
2019-01-01T12:40:19.473 INFO AnchoreWorker Completed analysis and processed policy evaluation result
2019-01-01T12:40:19.497 INFO AnchoreWorker Policy evaluation summary for docker.io/mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT - stop: 14 (+0 whitelisted), warn: 27 (+0 whitelisted), go: 0 (+0 whitelisted), final: stop
2019-01-01T12:40:19.497 INFO AnchoreWorker Anchore Container Image Scanner Plugin step result - FAIL
2019-01-01T12:40:19.499 INFO AnchoreWorker Querying vulnerability listing for mydeveloperplanet/mykubernetesplanet:0.0.4-SNAPSHOT
Archiving artifacts
2019-01-01T12:40:21.041 WARN AnchorePlugin Failing Anchore Container Image Scanner Plugin step due to final result FAIL
2019-01-01T12:40:21.138 INFO AnchorePlugin Completed Anchore Container Image Scanner step
ERROR: Failing Anchore Container Image Scanner Plugin step due to final result FAIL
Finished: FAILURE

In the "Build Status" page, a link is added for the "Anchore Report":

Clicking the "Anchore Report" link shows us the "Anchore Policy Evaluation Summary":

The report shows us information about the vulnerabilities which are found, a link to the CVE page and whether the issue caused the analysis to fail because of the used Gate settings.

Solve the Vulnerabilities

The Anchore Analysis failed due to some critical vulnerabilities in our Docker image. But how to solve them? In our case, we are going to try to solve this by using the Java 11 LTS version Docker image as our base image. We fix this in branch feature/anchore_solved. We upgrade our version to 0.0.5-SNAPSHOT and change the following line in our Dockerfile:

FROM openjdk:10-jdk

into

FROM openjdk:11-jdk

Run the build again. Ensure that you changed the tag when creating the anchore_images file. To our surprise, the build still fails due to 5 vulnerabilities.

Let’s investigate this a bit further and check one of these vulnerabilities. First, we check the link referring to the CVE. The screenshot below shows us the information about the libidn package.

We notice that the issue is already fixed in some of the Debian releases. So, which Debian release are we using anyway? Go to the OpenJDK DockerHub repository, search for 11-jdk and click the Dockerfile link at the end of the line. This shows us that the Docker image we created from, is based on Debian Stretch. This explains more or less why the Anchore Engine analysis failed. The issue is fixed, but not yet released to Debian Stretch. This also applies to the 4 other vulnerabilities. So what to do? We are using the latest Docker image available but this will cause our build to fail as long as these fixes are not released. The answer is to whitelist these vulnerabilities by means of a custom policy file.

Anchore Policies

Anchore policies are stored in a Policy Bundle. A Policy Bundle contains the policies for your Anchore gate, whitelists, etc. More information about the concept can be found here. What we need to do, is to change the existing Policy Bundle and whitelist the 5 vulnerabilities. We will use Anchore CLI for this.

First, retrieve the list of policies in Anchore Engine. There should be one default policy file available which is also active:

$ anchore-cli policy list
Policy ID                               Active    Created                 Updated
2c53a13c-1765-11e8-82ef-23527761d060    True      2018-12-27T09:12:30Z    2018-12-27T09:12:30Z

We can also view which gates are configured in Anchore Engine:

$ anchore-cli policy describe
+-----------------+------------------------------------------------------------+
| Gate            | Description                                                |
+-----------------+------------------------------------------------------------+
| always          | Triggers that fire unconditionally if present in policy,   |
|                 | useful for things like testing and blacklisting.           |
+-----------------+------------------------------------------------------------+
| dockerfile      | Checks against the content of a dockerfile if provided, or |
|                 | a guessed dockerfile based on docker layer history if the  |
|                 | dockerfile is not provided.                                |
+-----------------+------------------------------------------------------------+
| files           | Checks against files in the analyzed image including file  |
|                 | content, file names, and filesystem attributes.            |
+-----------------+------------------------------------------------------------+
| licenses        | License checks against found software licenses in the      |
|                 | container image                                            |
+-----------------+------------------------------------------------------------+
| metadata        | Checks against image metadata, such as size, OS, distro,   |
|                 | architecture, etc.                                         |
+-----------------+------------------------------------------------------------+
| npms            | NPM Checks                                                 |
+-----------------+------------------------------------------------------------+
| packages        | Distro package checks                                      |
+-----------------+------------------------------------------------------------+
| passwd_file     | Content checks for /etc/passwd for things like usernames,  |
|                 | group ids, shells, or full entries.                        |
+-----------------+------------------------------------------------------------+
| ruby_gems       | Ruby Gem Checks                                            |
+-----------------+------------------------------------------------------------+
| secret_scans    | Checks for secrets found in the image using configured     |
|                 | regexes found in the "secret_search" section of            |
|                 | analyzer_config.yaml.                                      |
+-----------------+------------------------------------------------------------+
| vulnerabilities | CVE/Vulnerability checks.                                  |
+-----------------+------------------------------------------------------------+

And we can retrieve the details of a configured gate:

$ anchore-cli policy describe --gate=vulnerabilities
+--------------------------------+-----------------------------------------+----------------------+
| Trigger                        | Description                             | Parameters           |
+--------------------------------+-----------------------------------------+----------------------+
| package                        | Triggers if a found vulnerability in an | package_type,        |
|                                | image meets the comparison criteria.    | severity_comparison, |
|                                |                                         | severity,            |
|                                |                                         | fix_available,       |
|                                |                                         | vendor_only          |
+--------------------------------+-----------------------------------------+----------------------+
| stale_feed_data                | Triggers if the CVE data is older than  | max_days_since_sync  |
|                                | the window specified by the parameter   |                      |
|                                | MAXAGE (unit is number of days).        |                      |
+--------------------------------+-----------------------------------------+----------------------+
| vulnerability_data_unavailable | Triggers if vulnerability data is       |                      |
|                                | unavailable for the image's distro.     |                      |
+--------------------------------+-----------------------------------------+----------------------+

A complete list of the parameters which can be used to configure the policy file, can be found here.

Next step, is to download the Policy Bundle as a JSON file:

$ anchore-cli policy get 2c53a13c-1765-11e8-82ef-23527761d060 --detail > policybundle.json

Before we add the vulnerabilities to the whitelist, we need to check the vulnerabilities and verify whether we accept the risk or whether we have to wait for a fix. In our case, we will whitelist all 5 vulnerabilities. Therefore, add the following section to the policy JSON file:

"whitelists": [
  {
    "comment": "Default global whitelist", 
    "id": "37fd763e-1765-11e8-add4-3b16c029ac5c", 
    "items": [
      {
        "gate": "vulnerabilities",
        "trigger_id": "CVE-2017-14062+*",
        "id": "rule1"
      },
      {
        "gate": "vulnerabilities",
        "trigger_id": "CVE-2017-17458+*",
        "id": "rule2"
      },
      {
        "gate": "vulnerabilities",
        "trigger_id": "CVE-2018-13347+*",
        "id": "rule3"
      },
      {
        "gate": "vulnerabilities",
        "trigger_id": "CVE-2018-13347+*",
        "id": "rule4"
      }
    ], 
    "name": "Global Whitelist", 
    "version": "1_0"
  }
]

The gate and trigger_id are taken from the Jenkins Anchore Policy Report, each whitelist item needs to have a unique id which we fill in ourselves. The changed policybundle.json file can be found at GitHub.

Upload the changed policy JSON file to Anchore Engine:

$ anchore-cli policy add policybundle.json 
Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060
Active: False
Source: local
Created: 2018-12-27T09:12:30Z
Updated: 2019-01-03T10:44:34Z

When we list the policies, we notice that the policy is not active yet:

$ anchore-cli policy list
Policy ID                               Active    Created                 Updated 
2c53a13c-1765-11e8-82ef-23527761d060    False     2018-12-27T09:12:30Z    2019-01-03T10:44:34Z

In order to conclude, we activate the policy:

$ anchore-cli policy activate 2c53a13c-1765-11e8-82ef-23527761d060
Success: 2c53a13c-1765-11e8-82ef-23527761d060 activated

Instead of updating the default policy, we could also create a custom policy file and use that in our Jenkins build configuration.

Run the Jenkins build again. The Anchore Engine analysis does not have a STOP gate action anymore and the build is successful.

Conclusion

The Anchore Container Image Scanner Jenkins Plugin is a great plugin which automatically scans your Docker images for known vulnerabilities, a must-have plugin when creating Docker images. The whitelisting of vulnerabilities can be a little bit cumbersome, but this might be an incentive for choosing the Anchore Enterprise version, which comes with a nice GUI instead of CLI configuration.