Software supply chain security has become a hot topic in recent years due to the catastrophic breaches we have seen in the news and their massive financial ramifications. We only need to mention names like Log4Shell to remember how devastating a simple vulnerability can be. To avoid these calamities, we must apply core practices for securing our software supply chains and ensure that we deliver and deploy artifacts that are verifiably secure.

In this Refcard, we will answer the following five questions:

1. What is a software supply chain?
2. How can we ensure that we consume secured dependencies?
3. How can we securely build artifacts?
4. How can we securely deliver and deploy artifacts?
5. What standards and compliance levels exist for software supply chain security?

Before we dive into securing our supply chain, we must first understand what a supply chain is. A supply chain is the network of procured resources needed to produce a product. The chain is formed when our product is used as a resource for another product. For example, if we want to manufacture pencils, we will need to procure the following:

• Wood
• Metal
• Graphite
• Rubber

We can then supply our pencils to an art company, who will package them into their art kits, which are eventually sold in an art store. While this supply chain is simple, there is a lot for us to consider, including:

• Do the materials supplied to us meet our quality standards?
• What if there is a recall on the materials that were supplied to us? 
• What if a vendor that supplies our materials is unable to meet its quota?
• What if there are embargoes that restrict our importation from a vendor?
• How do we respond to a world event — such as a pandemic or war — that disrupts our supply chain?
• What legal restrictions or obligations do our vendors place on our products?

This list only scratches the surface of the concerns we must bear in mind when examining a supply chain, and these concerns also apply to software supply chains. A software supply chain is the supply chain of dependencies — a piece of software or configuration that our product depends on — that are required to create our software product. In addition, we must also consider the build system that consumes these dependencies and creates our software product.

Figure 1

In general, there are three components to a software supply chain:

1. Upstream – the dependencies required to build our product
2. Build system – the infrastructure and processes used to build our product
3. Downstream – the artifacts we produce that make up and accompany our product

Upstream and downstream deliverables may be more than software packages or components; they may also include build scripts, configuration files, checksums, and a host of other files that accompany the software itself. Likewise, build systems are not just the build appliance itself but are the entire stack, including Docker images, deployment configurations, and repositories.

Note: Software supply chain security differs from application security (AppSec). AppSec focuses on the core practices used to secure an application — such as defense in depth, open design, and least privilege — against infiltration, while supply chain security ensures our dependencies and deliverables are verifiably secure.

In the following section, we will explore core practices for securing each component of the software supply chain and understand the main considerations when creating a trusted software product.

Securing our supply chain — from the dependencies we use to the artifacts we build and publish — centers around creating trust. This trust must be verifiable. In this section, we will dive into how to instill this trust by securing the upstream inputs, build systems, and downstream outputs of our supply chain. 

We will use the following key terms throughout this discussion:

• Vulnerability – a software defect that an attacker can exploit to gain undesired access to systems or data
• Mitigation – a measure that does not remove a vulnerability but ensures our product will never enter a state in which the vulnerability can be exploited
• Remediation – a strategy (e.g., upgrading a dependency) to remove a vulnerability

Note: These core practices assume that we are using an automated continuous integration and continuous delivery (CI/CD) pipeline to build and deliver our software. As we will see, automation and logical division of pipeline stages are crucial to establishing verifiable trust in our supply chain

Upstream Security 

Few of our projects are built in isolation. In most modern applications, we use dozens, or even hundreds, of dependencies, all of which can introduce different vulnerabilities and obligations. 

Note: Not all dependencies are direct dependencies. There are also transitive dependencies, or dependencies of dependencies. For example, if our application depends on A, A depends on B, and B depends on C, a vulnerability in B or C could create an attack vector, even though we have not directly included B as a dependency. This structure can be composed into a dependency graph using dependencies as nodes (vertices) and dependence as edges. 

Figure 2

There are numerous types of dependencies, including:

• Software artifacts (e.g., Jars, NPM packages)
• Docker images
• Configuration files
• Infrastructure and IT automation configurations (e.g., Ansible configurations)

Dependencies, including transitive dependencies, can also introduce legal obligations through their licenses. We will not expand on those legal obligations in this Refcard, but it is important to understand the licenses of our dependencies and how they may incur legal obligations on our application (e.g., publishing code in a certain manner, forcing our entire product to be open source).

There are four main practices to secure our dependencies:

1. Enumerating our dependencies
2. Discovering their vulnerabilities
3. Verifying the legitimacy of our dependencies
4. Remediating their vulnerabilities in a timely manner

Software Bill of Materials 

The first step in securing our application against vulnerabilities introduced through our dependencies is understanding what our application depends on. Generating a software bill of materials (SBOM) is the simplest way to enumerate our dependencies. An SBOM is a standardized document that lists all our application's dependencies. This list has the following properties:

• Includes all dependencies in the dependency graph
• Includes each dependency only once

The uniqueness of an entry is determined by its name and version. If two dependencies exist with the same name and same version, only one entry for the dependency is created in the SBOM.

Note: Generating an SBOM does not require a build tool, but utilizing one makes our job significantly easier. Most build tools require us to specify our project dependencies so that they can pull them during compilation. Therefore, most build tools, such as Maven, Gradle, NPM, or PyPi, can easily generate an SBOM from this dependency specification.

Currently, there are three main SBOM formats that we can generate:

1. Software Package Data Exchange (SPDX) by the Linux Foundation
2. CycloneDX by OWASP
3. Syft by Anchor

Abiding by standards is important because it allows for different tools — including the vulnerability scanners we will see in the next section — to understand our SBOM and provide useful information, such as vulnerable or outdated dependencies. It also allows users of our artifacts to easily verify our dependencies.

To automatically generate an SBOM, we can add the following tools to our project:

• SPDX contains plugins for Maven and other languages and build systems.
• CycloneDX contains numerous plugins for different build systems, including Maven, Gradle, and NPM.
• Syft generates an SBOM from a Docker container image or a file system.

The correct SBOM generation tool to use will depend on our specific project and build system, but regardless of the tool we select, it is important to generate an SBOM in a standard format so that vulnerability scanners can check if our product includes any vulnerable dependencies.

Recognizing Vulnerabilities

Using our enumerated dependencies, we can now determine which dependencies have known vulnerabilities. Recognizing these vulnerabilities requires us to cross-reference our SBOM against a list of known vulnerabilities. In most cases, vulnerabilities are published as a Common Vulnerabilities and Exposures (CVE) submission — a security bulletin that contains information about a specific vulnerability and which versions of a product are affected. Vendors will often publish a CVE to the CVE system when they recognize that their artifact is vulnerable to exploitation. 

Note: The CVE system is maintained by Mitre and made available to the public for free. This CVE database is also made available through the National Vulnerability Database, which is maintained by the U.S. National Institute of Standards and Technology.

More sophisticated tools can also cross-reference Vulnerability Exploitability eXchange (VEX) reports to provide information about a vulnerability's scope and possible remediations. CVEs, VEXs, and other vulnerability information can provide context and understanding about vulnerabilities we have inherited from upstream and how we can remediate or mitigate them.

Currently, the most common open-source tools available to perform automatic SBOM verification are Trivy and Grype. There are many other free, open-source, and paid tools available as well. Many of these tools also include vulnerability scanning beyond simple CVE checks, such as automated tools to recognize where a vulnerability exists in our code, automated pull request (PR) generation to remediate the vulnerability, and build system integration.

Note: This process must be part of our automated build process and performed during every build. Every build we launch can contain different dependencies and, therefore, different vulnerabilities. We must check every build to know what vulnerabilities our product has at any given time.

Other tools can directly integrate into our repositories, such as GitHub Security and GitLab Dependency Scanning. The benefits of these tools include:

• Integrating natively with our existing repositories and CI/CD pipelines (one-stop view of our code)
• Automating PR creation for remediation
• Tracking vulnerability reduction over time (graphing the number of vulnerabilities in our code over time)
• Removing the need for a standalone SBOM

While creating an SBOM is an important step, some of these repository-native tools do not require a separate SBOM, and instead, they scan our repositories and generate a dependency graph for us. While this removes the need to generate our own SBOM to find vulnerabilities, we should still consider generating one. As we will see later, a standalone SBOM can be useful for third parties that use our artifacts and can provide a simple, human-readable document detailing what our product includes. We may also have legal or contractual obligations to generate and publish an SBOM

The tools we choose for vulnerability scanning will depend on the specific project and our needs, but the lists above are a good starting place when considering an automated vulnerability scanner.

Note: There are occasions when we recognize a vulnerability or bug in a dependency before others, including the vendor, do. In that case, it is important to file a bug with the vendor or notify the vendor so that a CVE can be filed and a fix can be created as quickly as possible, allowing us to remediate the vulnerability in our product.

Timely Remediation 

Once we recognize that a dependency is vulnerable to exploitation, we must quickly remediate the vulnerability. Remediation usually takes one of three forms:

1. Upgrading – upgrading a dependency to a version that is not vulnerable
2. Mitigation – changing our product to ensure it cannot enter a state where the vulnerability can be exploited 
3. Removal – removing the dependency entirely

Note: There are times when a dependency has reached end of life (EOL) and is no longer supported. This means that no further updates will be made to the dependency. For dependencies approaching EOL, it is important to have a contingency — or alternative solution — in place. If a vulnerability is found with an EOL dependency, we have a few paths to choose from:

• Use a new dependency that replaces the functionality of the existing one.
• Change our code to not use the dependency's functionality, allowing us to remove the dependency.
• Assume ownership of the dependency's source code, remediate the vulnerability in the source code, and build a new artifact that we can use as an updated dependency.

The speed at which we can apply one of these remediation strategies largely depends on two factors:

1. How quickly we can update our source code
2. How quickly we can release an updated version of our product

The first factor is usually dictated by the complexity of our code. For example:

• Do we have numerous, tightly coupled dependencies, or is it decoupled, where changes in one component will have a minimal impact on other components?
• Do we keep our code up to date?
• Is our dependency list organized, where we have a central version list that can be updated easily?

Updating our code quickly will depend on how well we write our code and follow best practices for source development. On the other hand, how quickly we can release our updated product will largely depend on how agile and lean our release cycle is. If we have achieved maximal automation and a commit results in a fully tested, automatically delivered artifact, we can easily release a fix for our product that remediates a known, upstream vulnerability. This quick remediation becomes more difficult the more complex and bloated our release process becomes.

Here are some tips that help ensure our remediated product can be released quickly:

• Streamline the release process and remove as much bloat as possible.
• Create an automated CI/CD pipeline for releases.
• Increase automated test coverage and reduce manual testing.
• Shorten official release cycles (ideally, every commit can result in a release).

Even if release cycles cannot be reduced to one release per commit, we should reduce our cycle time as much as possible — a one-month cycle is better than a one-quarter cycle, and a one-week cycle is better than a one-month cycle.

Verifying Legitimacy of Dependencies 

Apart from knowing which vulnerabilities we introduce through our dependencies, we must also verify that our dependencies are legitimate. For example, how do we know that the dependencies we include are not imposters of the real dependencies, where the imposter contains malicious code? In practice, this means ensuring the authenticity of a dependency's publisher — verifying that it is not a malicious actor — and that the dependency is untampered after publication.

We do this verification through:

1. Verifying the signature of our dependencies and rejecting unsigned or tampered dependencies
2. Verifying the certificate for the repository from which we obtain the dependency

We will see the details of how each of these steps works when we discuss how we make our downstream artifacts trustworthy in the "Downstream Security" section, but it suffices to know that both of these steps utilize verifiable cryptographic signatures. If we successfully verify the signatures, we can trust the publisher's authenticity and ensure the artifacts it provides have not been tampered with after publication.

Build Security 

Securing the inputs to our product and quickly releasing new versions of our product are core tenants in supply chain security, but if we neglect the security of our build systems, those efforts can be quickly nullified. For example, creating an effective vulnerability scanning and remediation process can be quickly quashed if an attacker gains access to our build systems and disrupts those processes. 

Build system security centers around two practices:

1. Securing our build pipeline
2. Securing our build infrastructure

The goal of securing our build systems is to create a trusted build process. In practice, this means that if we effectively secure our build systems, artifacts produced by these systems can be trusted since they come from a trusted source.

Build Pipeline Security 

The first type of build security centers around our CI/CD build pipeline. It may not be obvious at first, but our pipelines have dependencies themselves — compromising these dependencies could cause disclosure of sensitive information or allow vulnerabilities to be injected into the artifacts we build.

In many cases, our code is committed to a public or private repository that can be compromised. Suppose a bad actor gains access to this repository and creates a new branch with an in-repository build configuration that outputs sensitive keys or passwords. In that case, our build system may pick up this commit and execute the compromised build. This attack is called a poisoned pipeline execution (PPE).

Note: Repositories, and the code contained within them, are inputs to our build and should be treated as attack vectors like any other dependency or input. Even if the code is written within our company, or the repository is hosted within a demilitarized zone (DMZ) in our network, it can still be compromised. Therefore, we should treat it with the appropriate level of distrust.

In order to prevent PPE attacks, we should:

• Provide repository access to only those developers who need it.
• Revoke repository access when a developer no longer needs it.
• Require pull requests to be created when a change is made and require reviews of each PR — ideally by two or more developers — before a PR can be merged.
• Minimize the number of secrets and sensitive information exposed to a build by the build system.
• Institute constraints for builds (e.g., manual build triggering) triggered by commits to public repositories.
• Isolate builds that have a higher degree of risk, such as those deriving from public repositories or repositories with large development teams (i.e., higher exposure).

Despite our best efforts, it is possible that our build repositories may become compromised. In that case, it is important that our build system recognizes a PPE attack and halts a build immediately. This can be done by enacting build system monitoring that will recognize when a secret is exposed in an artifact. If such a compromised artifact is generated, the build should immediately fail, and administrators should be alerted to the breach.

Infrastructure Security 

The second portion of our build system that must be secured is the build infrastructure. In general, threats to our build infrastructure usually take on one or more of the following forms:

1. Human error and negligence
2. Unhardened machines or containers
3. Insufficient cryptographic security

Human error is an unavoidable part of software development, but there are measures we can take to reduce its likelihood. One of the simplest steps is to remove as much manual intervention as possible. In practice, this means:

• Scripting and automating as much as possible, including build steps,container deployment, and pipeline as code
• Enabling proper and minimal access control lists for all machines and processes that users can access
• Minimizing the number of accounts that can access the build system, including removing inactive accounts and providing access on a need-to-know basis
• Providing only the exact permissions a user needs and no more

While we cannot completely remove human interactions with the build system, each time a human accesses the system, it should be viewed as an opportunity to automate a process. For example, if an administrator logs onto a build machine to make a manual change, we should consider improving our IT automation so that this manual change does not have to be made again.

To harden our build machines, we should also apply core practices for IT security, including:

• Minimizing the attack surface by closing all unnecessary access points and ports
• Routinely penetration testing our systems
• Routinely auditing each user's access to the system
• Continuously updating packages on the build machine using IT automation
• Maintaining lightweight systems (e.g., containers), so compromised systems can be destroyed quickly and new systems brought up easily
• Backing up data to ensure ransomware and other lock-out attacks can be remediated quickly

Note: This level of security should be applied up and down the entire stack for our build system. For example, if we deploy Docker containers to perform our build, the containers and the host machine must be secured.

Lastly, it is important that we maintain our cryptographic tools on our build systems and treat them as first-class security concerns. Using outdated private keys, invalid certificates, and outdated standards — such as outdated Transport Layer Security (TLS) versions — can result in compromises to our build system. In addition, as we will see shortly, neglecting cryptographic tools can also undermine the trust in our artifacts (through digital signatures) when they are published downstream.

Downstream Security 

Apart from securing the inputs to our build and the build itself, we also have an obligation to secure the artifacts we create. We commonly use these artifacts in two non-exclusive ways:

1. Deployed to a production environment
2. Used as dependencies in further downstream products (inside and outside our team)

In the first case, we deploy the artifacts generated by our build and execute them in a production environment. For example, if we are developing a web service, this could mean deploying our Docker containers into a Kubernetes cluster and ensuring that our service is available to our intranet or the public in general. In this case, we must verify that the artifacts are legitimate and can be trusted before deploying them into our production environment.

In the second case, our artifacts may be used as inputs for another project. For example, we may create Docker images that are used by another project as a parent image in a new image, or we may create Jars that are used by another project as a dependency in a Maven build.

Note: We still have an obligation to secure our artifacts, even if they never leave our team. For example, if we have a build pipeline that creates a Jar, and this Jar is used as an input in another internal pipeline, the Jar may never be used externally. Nonetheless, we are still responsible for diligently providing proof of the Jar's trustworthiness. Failing to do so creates a break in the security of the second project's supply chain, just as it would if any external dependency failed to provide such proof.

In essence, we are obligated to provide the same proof of trustworthiness (called an attestation) to others — even other projects within our own purview — that we expect when using artifacts as dependencies.

In practice, providing downstream security imposes three requirements:

1. Signing our artifacts 
2. Proving the authenticity of our artifact repositories
3. Monitoring our production environment

We will also explore some additional considerations that should be addressed to ensure downstream security.

Artifact Signing 

Signing our artifact means that we cryptographically certify that the artifact is built from trusted sources and built on trusted build nodes.

The process of cryptographic signing is illustrated below:

Figure 3

The output of the signing process is a digital signature, which is created by hashing the artifact and encrypting the hash with a private key. We then publish the signature, the corresponding public key, and the artifact. A verifier can then hash the artifact and decrypt the signature using the public key. If the decrypted hash matches the verifier's hash, then the verifier knows that:

1. The artifact has not been tampered with after release
2. The artifact was built on a trusted build node that has the private key

This process ensures both the integrity and authenticity of the artifact and proves that the artifact can be trusted.

Note: This security rests upon the security of the private key. If the private key is compromised in any way, it must be immediately revoked — and a new public key published — to ensure that an attacker cannot act as an imposter. Provided that we properly secure our build nodes (see the "Build Security" section above), artifact signing provides confidence to the downstream user of our artifact, whether ourselves or a third party, that our artifact can be trusted.

Automated signing can be performed by most build systems. For more information, see the following resources:

• "Apache Maven Jarsigner Plugin," Maven
• "The Signing Plugin," Gradle
• "About ECDSA Registry Signatures," NPM
• "PEP 458 – Secure PyPI Downloads Sith Signed Repository Metadata," PyPI

Artifact Repository Authenticity 

While cryptographic signing is crucial in trusting an artifact, we must also prove the authenticity of the repository in which our artifacts are stored. For example, what if the signature is correct for the artifact, but both the artifact and the signature come from an untrusted source? To ensure that the artifact can be trusted, we must prove that the repository in which it is stored can also be trusted.

Trusting an artifact repository and the artifacts that it stores rests upon proving its authenticity. This proof is provided through certificates and the use of protocols that verify these certificates. In practice, this means:

1. Our artifact repository should always have a valid certificate.
2. The certificate should be signed by a trusted Certificate Authority (CA).
3. All access to our artifact repository, such as obtaining artifacts, should be performed through a secure protocol that validates the certificate via TLS (e.g., HTTPS, FTPS).

The second point above is of special importance. There are valid uses for self-signed certificates, but in this case, we should always use a certificate that has been properly signed by a trusted CA whenever possible. This not only removes any warnings when viewing our repository through a browser or other interface, but it also informs our repository users that we have done our due diligence and that our certificate can be trusted all the way back to the root CA.

Lastly, if possible, we should restrict all access to our artifact repository to sufficiently secure protocols. For example, if we allow access to our repository using an older version of TLS, or through protocols that do not include certificate verification (e.g., HTTP), we break the security chain. Even though it is up to the client to ensure that they are using a secure protocol, removing the possibility of such insecure protocols ensures that we eliminate as many points of distrust as possible.

Production Monitoring 

In some cases, we deploy our artifacts as working applications. When doing so, we must not only secure the delivery and deployment of our artifacts but also monitor them at runtime. Monitoring includes:

• Frequently updating vulnerable infrastructure to remediate or mitigate vulnerabilities
• Auditing production environments to ensure alignment with our infrastructure's declarative state (e.g., Ansible or Puppet configurations)
• Auditing production infrastructure, including Docker containers, to ensure alignment with its deployment configuration (e.g., Kubernetes or OpenShift configurations)
• Securing the entire stack — from the kernel to the application layer

Note: Production environments should be configured using code whenever possible. Ideally, this configuration is declarative, meaning it declares the desired state of the environment rather than an arbitrary set of steps to alter the environment (e.g., with a shell script). This allows us to compare and audit if our environments match the desired state declared in our configuration files.

If any of these criteria are violated, monitoring tools must immediately alert administrators and isolate the vulnerable or exploited pieces in production. In some cases, automated measures can be taken, such as bringing down exploited containers and replacing them with secured replicas.

Other Considerations 

In addition to artifact signing, artifact repository security, and production monitoring, there are other aspects of downstream security that we should consider:

• Legal responsibility – When a vulnerability is exploited, certain parties may be held responsible. Due diligence must be performed and best practices must be adhered to. Failure to do so may result in legal action against the company, its developers, or even its executives.
• Reputational damage – Releasing vulnerable software or failing to fix a known vulnerability may result in damage to a vendor's or brand's reputation, and this damage can last for months, years, or even the life of the vendor.
• Publishing SBOMs – In some cases, we may need to publish our SBOM in a specific format so that other vendors can audit our artifacts and enable traceability downstream. This is often true when publishing artifacts to sensitive markets, such as government, military, medical, or financial industries. It is important to understand our legal and contractual obligations and to ensure that our build system and downstream delivery system abide by them.
• Personal responsibility – Not only are we legally responsible for securing our supply chain, but we are also personally responsible for doing so. Becoming a master of our craft requires that we secure our supply chains because it sets a high personal standard that steers the rest of our work in the right direction.

As software supply chain security has become more of a requirement for software development, standards have been established to ensure that industry-accepted practices are codified. These standards also allow us to judge how secure or compliant a software supply chain is. Currently, the main standards are:

• Supply-chain Level for Software Artifacts (SLSA)
• Secure Software Development Framework
• United States Executive Order 14028
• OWASP Software Component Verification Standard

These standards are not mutually exclusive, and we may be required to adhere to more than one depending on our industry. SLSA has taken the lead as one of the more widely adopted standards and includes four security levels:

1. Build L0 – no guarantees
2. Build L1 – provenance exists
3. Build L2 – hosted build platform
4. Build L3 – hardened builds

For more information on the requirements for each of these levels, see the SLSA Levels and SLSA Requirements pages.

Securing software supply chains has become a first-class consideration — along with coding and CI/CD pipelines — when developing a software product. Far too many vulnerabilities have been subliminally introduced into software products and resulted in catastrophic breaches for us as diligent developers to treat supply chain security as an afterthought. The core practices and principles outlined in this Refcard provide a foundation for creating secure supply chains that produce deliverables and products that others can trust.