Apart from protecting data and code from potential breaches that traverse through various endpoints of the pipeline, administering security on CI/CD pipelines also helps maintain compliance and prevent accidental issues such as data loss or corruption. In the following section, we discuss the steps for effective CI/CD security implementation and open-source tools to simplify the process.
Steps to Ensure CI/CD Pipeline Security
While the specifics of a CI/CD pipeline security strategy will differ by use case, the process typically follows a similar workflow.
1. Implement Strong Access Controls
The first step toward securing a pipeline is to control and organize access privileges. This essentially requires policy enforcement that restricts every user of the organization to possess similar privileges for accessing tools and resources within the CI/CD pipeline. Additionally, those with permissions to access the pipeline should not be assigned default permissions to view all resources and data within the pipeline.
Some approaches to help enforce access controls include:
- Configure identity and access management (IAM) – helps configure digital identities and enforce access permissions at the entity level
- Enforce role-based access controls (RBACs) – restricts users to access data and resources based on the functions/tasks associated with their roles
- Apply the principle of least privilege – limits a user's access rights to strictly what is required to perform their job
2. Secure Access to Code Repositories
Since a code repository acts as the central storage, review, and management system of the code used within a DevOps pipeline, securing repos is the next step that requires key consideration. Public code repos, or those lacking secure controls, are often targets of malicious exploits that lead to code tampering and loss of code integrity.
Approaches to securing code repositories include:
- Choose a trusted repository by providers with a reputation for secure infrastructure administration and management
- Enforce the principle of least privilege for repository access
- Secure access credentials and separate them from source code
- Revoke access to the repository when it is no longer required
- Review all code changes before merging to the main branch
- Conceal personally identifying information when using public repositories
- Enforce backup and disaster recovery for all code used within the system
- Perform regular audits against security benchmarks
3. Avoid Hard-Coding Secrets
Hard-coded passwords and secrets are common attack targets that lead to data breaches and malicious access of pipeline resources. Attackers typically target source codes within public repos and identify hard-coded credentials through code scanning, guessing, and learning. As a recommended practice, security admins should implement policies to regulate the usage of hard-coded secrets into application code. If secrets are to be parsed, they should be included as variables in a .gitignore file, which keeps them from being committed into the repository.
For instance, before distributing secrets in a Kubernetes cluster, secrets should first be encrypted at rest and then stored in the ETCD server.
A conventional approach for achieving this is by encoding the secrets in Base64 format as shown:
$ username=$(echo -n "default" | base64)
$ password=$(echo -n "a62fjbd37942dcs" | base64)
And then, defining the secrets:
echo "apiVersion: v1
> kind: Secret
> metadata:
> name: darwin-secret
> type: Opaque
> data:
> username: $username
> password: $password" >> secret.yaml
Following the above, you can now create the secret using the kubectl create command:
$ kubectl create -f secret.yaml
4. Perform Application Security Tests
Once code repositories are secure and secrets are safely managed, developers and security teams should collectively ensure the source code is free of any vulnerabilities. This is accomplished through a combination of tests that are deployed at each layer of CI/CD workflows to automatically notify security teams upon detecting pipeline vulnerabilities.
Figure 2: Security testing of a CI/CD pipeline
Automated tests can also be combined with automated remediation tools that use the findings of security checks to safeguard pull requests from attack vectors. In production-grade pipelines, a common approach is also to engage external penetration testers to provide an unbiased view of the pipeline's security posture and help identify flaws that may have been missed by automated tests.
5. Use Rollbacks to Enforce Security in Production Pipelines
Once policies are framed to secure pipelines, the next stage focuses on minimizing the consequences of a successful attack. This requires the formulation of controls that help revert to earlier, stable versions of an application if the current one is compromised. The ability to quickly roll back an insecure application version also helps reduce application downtimes while expediting patch cycles for faster remediation.
6. Outline an Incident Response Plan
Incident response plans strengthen a continuous testing process by shortening the feedback loop of identifying and addressing CI/CD security threats. Once potential security threats have been mapped with their respective attack vectors, the incident response plan should outline tools and processes to be used to restore normal operations. Besides reducing the response time for a security event, response plans should also tag a summary of related non-critical incidents that may signal potential issues within the application, thereby helping developers to fine-tune their code for security and performance.
7. Leverage a Security Information and Event Management Tool
Security information and event management (SIEM) tools go beyond incident response plans by offering granular indicators of various events. For CI/CD security, SIEM tools perform three critical capabilities:
- Threat detection
- Event investigation
- Response time reduction
These tools aggregate and analyze telemetry data from different resources of the CI/CD pipeline. The composite data is then stored, normalized, and analyzed for threat detection and trend analysis. When configuring an SIEM solution, security testers and developers should also integrate a continuous testing and monitoring framework for faster discovery of security breaches and remediation.
Popular Open-Source Tools for CI/CD Pipeline Security
Securing a CI/CD pipeline is a multi-pronged process that requires an in-depth understanding of the tech stack's core aspects, changing threat patterns, and inherent vulnerabilities. Out of the number of tools available, below is a list of popular open-source tools that are free, simplify the implementation of CI/CD security, and offer comprehensive hardening solutions.
OWASP SonarQube
SonarQube is a static application security testing (SAST) tool that tests applications against the most critical risk categories within application code. The tool performs a static analysis of pull requests to ensure that every piece of code entering the pipeline is free of threats found on the OWASP Top 10 list of vulnerabilities.
To help track unvalidated user inputs from the point of entry to the stage of code execution that enables a compromise, the tool relies on a taint analysis mechanism to detect malicious inputs flowing into the DevOps workflow. Besides this, the tool also offers an issue visualizer that enables closer inspection of how vulnerabilities flow within the pipeline while offering guidance to identify the root cause and enforce stricter controls.
OWASP Threat Dragon
OWASP Threat Dragon is a threat modeling tool that helps you record possible threats within DevOps pipelines and remediate them using threat model diagrams. The tool follows the principles and values of the OWASP's threat modeling manifesto and implements a rule engine that auto-generates threats in the pipeline and their possible mitigations. Installed as a desktop or web application, the tool offers a simple command-line workflow for quick threat modeling.
Threat Dragon uses the STRIDE threat model to group threats into six categories:
- Spoofing
- Tampering
- Repudiation
- Information disclosure
- Denial of Service (DoS)
- Escalation of privileges
Project Calico
Project Calico is a software-defined, open-source secure networking solution for container-native deployments, as containers sit at the core of modern CI/CD pipelines on account of their support for flexible, isolated, and infrastructure-agnostic deployments. Project Calico enforces zero-trust, endpoint-level security through GlobalNetworkPolicies to help secure both containerized hosts and workloads. The tool also helps secure in-cluster pod traffic with on-the-wire encryption, subsequently enforcing data integrity without requiring specialized hardware.
ELK Stack
Also known as Elastic Stack, the ELK Stack is a powerful platform for comprehensive observability of CI/CD pipelines. Elastic Stack is made up of three open-source projects, each specializing in different areas of the observability pipeline:
- Elasticsearch – a search and analytics engine that aids in the storage and indexing of log data
- Logstash – a data processing tool that extracts log data from multiple components of the CI/CD pipeline
- Kibana – the front-end visualization framework that provides security event information in a graphical format for simple analysis
Unlike other tools, the ELK Stack offers a centralized platform that helps inspect issues with other core indicators of the system. With ELK Stack's support of code audits, you can also identify vulnerabilities and fix them proactively before they lead to configuration conflicts or compliance issues.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}