DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
  1. DZone
  2. Refcards
  3. Getting Started With CI/CD Pipeline Security
refcard cover
Refcard #387

Getting Started With CI/CD Pipeline Security

The increasingly distributed nature of CI/CD frameworks has made organizations more vulnerable to attacks, which can range from threats to supply chains and servers to exploitations of the application code itself. In this Refcard, you'll learn about the primary focus areas of CI/CD pipeline security, review common pipeline threats and security challenges, as well as walk through seven steps to get started with securing your pipelines.

Free PDF for Easy Reference

Brought to You By

Check Point
refcard cover

Written By

author avatar Sudip Sengupta
Technical Writer, Javelynn
Table of Contents
► Introduction ► Key Aspects of Securing CI/CD Pipelines ► Common CI/CD Pipeline Security Threats ► Administering Comprehensive Security on CI/CD Pipelines ► Conclusion
Section 1

Introduction

A continuous integration/continuous delivery (CI/CD) pipeline is an agile workflow that automates the build, test, and deploy cycles of application delivery. While automated deployment cycles enable developers to release new features and updates rapidly, CI/CD pipelines are commonly targeted by attackers who are looking to exploit vulnerabilities and inject malicious code into application workflows. A compromised pipeline often has severe consequences, such as an attacker gaining access to sensitive data and even controlling the release of new software versions.

In this Refcard, we discuss the key aspects and challenges of securing CI/CD pipelines as well as the fundamental steps to administer security on CI/CD pipelines.

Section 2

Key Aspects of Securing CI/CD Pipelines

A DevOps workflow is typically characterized by its non-traditional approach to security. This is often because the security of a DevOps workflow is not centralized or does not follow the same approach as other workflows. Instead, securing a DevOps workflow is often distributed among various tools and processes.

Figure 1: Key aspects of securing CI/CD pipelines

Securing the CI/CD pipeline at every stage requires a thorough understanding of the core aspects, common threats, and challenges for CI/CD security. Core aspects of CI/CD security include testing, automation, source control, incident management, secrets management, vulnerability scanning, and access control.

Testing

Continuous application testing helps ensure software security and quality without compromising delivery cycles. Besides inspecting application source code, testing also relies on an iterative cycle of identifying security flaws in third-party libraries, resource-level conflicts, and misconfigurations.

It is also important to employ the appropriate testing approaches that inspect flaws across various stages of the CI/CD pipeline. These include:

  • Static tests – These tests can be run against code that isn't yet deployed to production, making them extremely fast and easy to automate. However, this testing mechanism can only test for superficial defects, which lacks offering a comprehensive picture of how the code will actually behave in a production environment.
  • Dynamic tests– These tests, on the other hand, inspect code during application runtime. This makes dynamic tests slower and more difficult to automate, but they are efficient at detecting flaws that static tests would normally miss. Dynamic tests are further categorized into:
    • Load testing to ensure that the system can handle heavy traffic
    • Stress testing to identify performance bottlenecks
    • Security testing to check for vulnerabilities
  • Penetration testing – This is a proactive approach used to simulate real-world attacks and offer valuable insights into an organization's security posture. The testing methodology also helps validate the strength of security controls (e.g., firewalls, antivirus systems).

Automation

By automating the processes of building, testing, and deploying code, you can ensure that only approved code is deployed to production. Automated enforcement of security controls eliminates errors associated with the manual execution of repetitive tasks while making it easier to track changes and roll back if necessary.

To ensure systems operate on the most secure versions of software, enterprises can also leverage automation for faster roll out of security updates and patches. Besides automatically documenting and recording system vulnerabilities, you can also leverage automation platforms to configure notifications and alerts to flag security threats as soon as they arise.

Source Control

One of the most powerful ways of enforcing code integrity is to use source control systems that enable enterprise teams to securely manage code changes, collaborate with cross-functional teams, and resolve conflicts in code before committing changes. This approach also helps prevent accidental or malicious changes from being introduced into the codebase, which could potentially break the build or cause other problems downstream. 

Also commonly referred to as version control, source control involves the configuration of access permissions to the codebase, ensuring only the approved contributors are allowed to make code changes. This guarantees that only authorized users have access to the codebase and that all changes are tracked and audited. Additionally, by using a centralized source control repository, you can more easily automate code reviews and roll back changes if something does go wrong.

Incident Management

Incidents are unplanned events that disrupt normal operations by compromising the integrity of a system. In the context of CI/CD pipelines, incidents can range from simple build failures to more complex security breaches. Consequently, it's essential to formulate an incident management process that encompasses various procedures and tools to manage and respond to security events.

While the primary purpose of an incident management framework is to reduce the impact of an event, it also helps alleviate the future occurrence of similar incidents by helping recognize identical patterns and fine-tuning alerting systems for expedited response. A typical approach is also to hard-code incident response plans into workflow tools, allowing for the automatic remediation of CI/CD security threats.

Secrets Management

Managing secrets involves practices and procedures to securely manage, store, and transmit confidential credentials, including encryption keys, API keys, passwords, session tokens, database connection strings and certificates. Effectively administered secrets management maintains a fine balance between the ease of injecting secrets and limiting data exposure. This essentially implies that sensitive data remain confidential, while services can autonomously use secrets to interconnect with other services or tools.

There are a few key things to keep in mind when managing secrets for CI/CD pipelines:

  • Always use strong encryption for storing and transmitting secrets. This will help ensure that even if a malicious user gains access to your secrets, they will not be able to read or use them.
  • Be sure to rotate your secrets regularly. This will help prevent attackers from using old secrets that they may have discovered.
  • Make sure that only authorized users have access to your secrets. This can be accomplished through role-based access control (RBAC) or other authorization mechanisms.
  • Use environment variables to store secrets as part of your application code. This approach allows you to keep secrets out of your code repository that prevent deeper compromise of the system.

Vulnerability Scanning

Automated vulnerability scanning helps teams enforce a shift-left approach for security by identifying and remediating threats from early stages of a development cycle. Remediating vulnerabilities typically involves detecting a flaw, assessing its impact and severity, deploying a fix, and performing a determinative scan to ensure the flaw no longer exists. Since CI/CD pipelines are composed of numerous components and dependencies, vulnerability scanning for CI/CD is often broken down into:

  • Source code scanning
  • Third-party dependency scanning
  • Container image scanning
  • Infrastructure component scanning

To ensure all misconfigurations are appropriately attributed with their impacts, a common practice is also to leverage databases of known weaknesses. Some popular vulnerability databases include:

  • Common Weakness Enumerations
  • National Vulnerability Database
  • OWASP's Top 10 CI/CD Vulnerabilities List

Access Control

One of the most important aspects of CI/CD security is making sure all cluster endpoints are secured. Access control mechanisms help mitigate the risk of data breaches by determining who has the privileges to access specific data and resources of a pipeline. Administering stricter policies requires users to verify their identity before they are allowed to access sensitive information. Beyond verifying a user's identity, access control policies also determine the allowed actions by defining permissions granted for each user.

Section 3

Common CI/CD Pipeline Security Threats

As per OWASP, although there are emerging practices and tools to avert security incidents, attackers continue to adapt novel techniques that exploit the distributed complexity of a CI/CD framework. Some common security threats of CI/CD pipelines include:

  • Distributed denial-of-service (DDoS) attacks – are orchestrated by compromising the server, network, or service by overwhelming it with a high number of requests/internet traffic in a given time
  • Supply chain attacks – focus on weak links in trusted third-party vendors that offer tools and services to the CI/CD pipeline
  • Dependency confusion attacks – abuse flaws within package managers to replace legitimate private packages with malicious versions in public registries
  • Injection attacks – are exploited over input validation errors to inject unauthorized code into the application, which ends up interpreting it as part of a command or a query
  • Remote code execution attacks – are widely exploited attacks executed through malicious code on remote machines by connecting to them over insecure public and private networks

Table 1: Common CI/CD pipeline security threats

Threat Target CI/CD Pipeline Stage Attack Pattern
DDoS attacks Deployment
Leveraging botnets to target the victim server/network, overwhelming it and resulting in a denial of service
Supply chain attacks Build Injecting malicious code into an open-source component to compromise the entire tech stack
Dependency confusion attacks Source and build stages Registering a package with a similar name to the target application of a public repository, which gets committed to the pipeline every time a new install occurs
Injection attacks
Deployment
Altering request URLs to change the parameters of the resulting database query, consequently enabling unauthorized access of restricted data
Remote code execution attacks
All Tricking the target user to install arbitrary scripts on the host machine, which are subsequently executed to orchestrate deeper, system-level attacks

Challenges With Securing CI/CD Pipelines

Securing CI/CD is a complex practice that encompasses the identification, remediation, and prevention of security risks across each stage of a pipeline. While building a robust security posture is the fundamental objective of the practice, the framework should also continue to maintain the agility and pace of release cycles. As a result, when compared to securing legacy frameworks, there are a number of challenges with administering security on CI/CD pipelines, including:

  • Improper secrets management
  • Inconsistent approaches to microservices
  • Inadequate security automation
  • Conflicts between security and velocity
  • Unauthorized access to code registries
  • Developer and DevOps resistance 
Section 4

Administering Comprehensive Security on CI/CD Pipelines

Apart from protecting data and code from potential breaches that traverse through various endpoints of the pipeline, administering security on CI/CD pipelines also helps maintain compliance and prevent accidental issues such as data loss or corruption. In the following section, we discuss the steps for effective CI/CD security implementation and open-source tools to simplify the process.

Steps to Ensure CI/CD Pipeline Security

While the specifics of a CI/CD pipeline security strategy will differ by use case, the process typically follows a similar workflow.

1. Implement Strong Access Controls

The first step toward securing a pipeline is to control and organize access privileges. This essentially requires policy enforcement that restricts every user of the organization to possess similar privileges for accessing tools and resources within the CI/CD pipeline. Additionally, those with permissions to access the pipeline should not be assigned default permissions to view all resources and data within the pipeline. 

Some approaches to help enforce access controls include:

  • Configure identity and access management (IAM) – helps configure digital identities and enforce access permissions at the entity level
  • Enforce role-based access controls (RBACs) – restricts users to access data and resources based on the functions/tasks associated with their roles
  • Apply the principle of least privilege – limits a user's access rights to strictly what is required to perform their job

2. Secure Access to Code Repositories

Since a code repository acts as the central storage, review, and management system of the code used within a DevOps pipeline, securing repos is the next step that requires key consideration. Public code repos, or those lacking secure controls, are often targets of malicious exploits that lead to code tampering and loss of code integrity.

Approaches to securing code repositories include:

  • Choose a trusted repository by providers with a reputation for secure infrastructure administration and management
  • Enforce the principle of least privilege for repository access
  • Secure access credentials and separate them from source code
  • Revoke access to the repository when it is no longer required
  • Review all code changes before merging to the main branch
  • Conceal personally identifying information when using public repositories
  • Enforce backup and disaster recovery for all code used within the system
  • Perform regular audits against security benchmarks

3. Avoid Hard-Coding Secrets

Hard-coded passwords and secrets are common attack targets that lead to data breaches and malicious access of pipeline resources. Attackers typically target source codes within public repos and identify hard-coded credentials through code scanning, guessing, and learning. As a recommended practice, security admins should implement policies to regulate the usage of hard-coded secrets into application code. If secrets are to be parsed, they should be included as variables in a .gitignore file, which keeps them from being committed into the repository.

For instance, before distributing secrets in a Kubernetes cluster, secrets should first be encrypted at rest and then stored in the ETCD server.

A conventional approach for achieving this is by encoding the secrets in Base64 format as shown:

 
$ username=$(echo -n "default" | base64)
$ password=$(echo -n "a62fjbd37942dcs" | base64)


And then, defining the secrets:

 
echo "apiVersion: v1
> kind: Secret
> metadata:
>   name: darwin-secret
> type: Opaque
> data:
>   username: $username
>   password: $password" >> secret.yaml


Following the above, you can now create the secret using the kubectl create command:

 
$ kubectl create -f secret.yaml


4. Perform Application Security Tests

Once code repositories are secure and secrets are safely managed, developers and security teams should collectively ensure the source code is free of any vulnerabilities. This is accomplished through a combination of tests that are deployed at each layer of CI/CD workflows to automatically notify security teams upon detecting pipeline vulnerabilities.

Figure 2: Security testing of a CI/CD pipeline

Automated tests can also be combined with automated remediation tools that use the findings of security checks to safeguard pull requests from attack vectors. In production-grade pipelines, a common approach is also to engage external penetration testers to provide an unbiased view of the pipeline's security posture and help identify flaws that may have been missed by automated tests.

5. Use Rollbacks to Enforce Security in Production Pipelines

Once policies are framed to secure pipelines, the next stage focuses on minimizing the consequences of a successful attack. This requires the formulation of controls that help revert to earlier, stable versions of an application if the current one is compromised. The ability to quickly roll back an insecure application version also helps reduce application downtimes while expediting patch cycles for faster remediation.

6. Outline an Incident Response Plan

Incident response plans strengthen a continuous testing process by shortening the feedback loop of identifying and addressing CI/CD security threats. Once potential security threats have been mapped with their respective attack vectors, the incident response plan should outline tools and processes to be used to restore normal operations. Besides reducing the response time for a security event, response plans should also tag a summary of related non-critical incidents that may signal potential issues within the application, thereby helping developers to fine-tune their code for security and performance.

7. Leverage a Security Information and Event Management Tool

Security information and event management (SIEM) tools go beyond incident response plans by offering granular indicators of various events. For CI/CD security, SIEM tools perform three critical capabilities:

  1. Threat detection
  2. Event investigation
  3. Response time reduction

These tools aggregate and analyze telemetry data from different resources of the CI/CD pipeline. The composite data is then stored, normalized, and analyzed for threat detection and trend analysis. When configuring an SIEM solution, security testers and developers should also integrate a continuous testing and monitoring framework for faster discovery of security breaches and remediation.

Popular Open-Source Tools for CI/CD Pipeline Security

Securing a CI/CD pipeline is a multi-pronged process that requires an in-depth understanding of the tech stack's core aspects, changing threat patterns, and inherent vulnerabilities. Out of the number of tools available, below is a list of popular open-source tools that are free, simplify the implementation of CI/CD security, and offer comprehensive hardening solutions.

OWASP SonarQube

SonarQube is a static application security testing (SAST) tool that tests applications against the most critical risk categories within application code. The tool performs a static analysis of pull requests to ensure that every piece of code entering the pipeline is free of threats found on the OWASP Top 10 list of vulnerabilities.

To help track unvalidated user inputs from the point of entry to the stage of code execution that enables a compromise, the tool relies on a taint analysis mechanism to detect malicious inputs flowing into the DevOps workflow. Besides this, the tool also offers an issue visualizer that enables closer inspection of how vulnerabilities flow within the pipeline while offering guidance to identify the root cause and enforce stricter controls.

OWASP Threat Dragon

OWASP Threat Dragon is a threat modeling tool that helps you record possible threats within DevOps pipelines and remediate them using threat model diagrams. The tool follows the principles and values of the OWASP's threat modeling manifesto and implements a rule engine that auto-generates threats in the pipeline and their possible mitigations. Installed as a desktop or web application, the tool offers a simple command-line workflow for quick threat modeling.

Threat Dragon uses the STRIDE threat model to group threats into six categories:

  1. Spoofing
  2. Tampering
  3. Repudiation
  4. Information disclosure
  5. Denial of Service (DoS)
  6. Escalation of privileges

Project Calico

Project Calico is a software-defined, open-source secure networking solution for container-native deployments, as containers sit at the core of modern CI/CD pipelines on account of their support for flexible, isolated, and infrastructure-agnostic deployments. Project Calico enforces zero-trust, endpoint-level security through GlobalNetworkPolicies to help secure both containerized hosts and workloads. The tool also helps secure in-cluster pod traffic with on-the-wire encryption, subsequently enforcing data integrity without requiring specialized hardware.

ELK Stack

Also known as Elastic Stack, the ELK Stack is a powerful platform for comprehensive observability of CI/CD pipelines. Elastic Stack is made up of three open-source projects, each specializing in different areas of the observability pipeline:

  • Elasticsearch – a search and analytics engine that aids in the storage and indexing of log data
  • Logstash – a data processing tool that extracts log data from multiple components of the CI/CD pipeline
  • Kibana – the front-end visualization framework that provides security event information in a graphical format for simple analysis

Unlike other tools, the ELK Stack offers a centralized platform that helps inspect issues with other core indicators of the system. With ELK Stack's support of code audits, you can also identify vulnerabilities and fix them proactively before they lead to configuration conflicts or compliance issues.

Section 5

Conclusion

Google's State of DevOps 2022 report suggests that using continuous integration and delivery systems for production releases is one of the most commonly established practices in modern application delivery. With increasing adoption of DevOps practices, the foundational security of CI/CD pipelines has come under greater scrutiny. That's because these pipelines are often the gateway to an organization's codebase and deployments, making them a common target for attackers.

Although the report also offers early evidence that pre-deployment security scanning is effective at finding vulnerable dependencies, traditional security measures to administer security on CI/CD-based workflows are often insufficient. Consequently, a DevOps practice relies on the implementation of granular policies across every stage of pipelines for comprehensive security. 

Additional resources:

  • Continuous Delivery Pipeline Security Essentials – https://dzone.com/refcardz/continuous-delivery-pipeline-security-essentials
  • Advanced Cloud Security: Continuous Security Strategies for Cloud Infrastructure – https://dzone.com/refcardz/advanced-cloud-security
  • Threat Detection for Containers: Essentials to Securing Threats for Containerized Cloud-Native Applications – https://dzone.com/refcardz/threat-detection-for-containers
  • Cloud-Native Application Security Patterns and Anti-Patterns – https://dzone.com/refcardz/cloud-native-application-security-1
  • IaC Security: Core DevOps Practices to Secure Your Infrastructure as Code – https://dzone.com/refcardz/iac-security-1

Like This Refcard? Read More From DZone

related article thumbnail

DZone Article

related refcard thumbnail

Free DZone Refcard

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.linkDescription }}

{{ parent.urlSource.name }}
by
CORE
· {{ parent.articleDate | date:'MMM. dd, yyyy' }} {{ parent.linkDate | date:'MMM. dd, yyyy' }}
Tweet
{{ parent.views }} ViewsClicks
  • Edit
  • Delete
  • {{ parent.isLocked ? 'Enable' : 'Disable' }} comments
  • {{ parent.isLimited ? 'Remove comment limits' : 'Enable moderated comments' }}