How to Secure Secrets in CI/CD Pipelines

CI/CD pipelines are the foundation of modern software delivery. Every code change, no matter how small or large, always goes through automated build, test, and deployment workflows prior to production delivery, and then becomes available to end users.

These CI/CD pipelines are connected with several systems. They are connected with different external systems, including image container registries, cloud platforms, artifact repositories, package managers, infrastructure tools, third-party applications, and many other systems. To enable this automation, pipelines depend on credentials including API tokens, cloud keys, service accounts, and passwords.

CI/CD pipelines gather and store a number of secrets over time to perform their required job and perform the automation. However, secret management in CI/CD is often considered a lower priority and generally doesn’t take precedence. It’s one of the most common security threats in CI/CD and DevOps environments.

In this article, we will discuss common errors teams make when handling secrets and specific methods to secure them in CI/CD pipelines.

Why CI/CD Pipelines Are High-Risk Systems

Unlike application runtime environments where software runs, CI/CD pipelines often operate with lifted permissions. These elevated permissions are required for the automation, and pipelines typically have access to perform the following tasks:

• Deploy infrastructure in production. This involves deploying infrastructure as code to production, including spinning up new virtual instances, increasing storage, updating network policies, etc.
• Publish artifacts to artifact storage. The artifacts are generated and published in the artifacts storage. These artifacts typically get generated as output from one of the stages and later used by either another stage in the same pipeline or by another pipeline.
• Push container images to the container registry. This involves building and pushing Docker images. These images are used to containerize the applications.
• Deploy application code in production. Deploying the application with the new changes, bug fixes, and new features in production. This application is used by the end users.
• System access. The pipeline needs access to other systems, such as the database and file server, during the workflow to automate tasks.

If a secret is exposed knowingly or otherwise, the impact can be significant. The pipeline has broad access, and if the secret gets exposed, it can have big security risks and cause severe damage.

Additionally, pipelines are triggered by commits, tags, or pull requests based on the configuration. This means external contributors or automated systems may indirectly influence pipeline execution.

Without proper controls, sensitive data can be exposed through:

• Build logs
• Configuration files
• Environment variables
• Misconfigured access policies

Typical CI/CD Secret Management Errors

1. Keeping Secrets as Plain Text in Repository Files

You might save credentials in repository configuration files. Keeping credentials in the configuration files makes it easier while designing the pipeline at the initial stage, when we are working on the prototype. However, over time, this adds more security risks when repositories become complex in nature, and more users are added to the repository as contributors. Having credentials in the file and having more users accessing the repository increases the risk of secrets being exposed. 

2. Using Long-Lived Credentials

Static tokens, which are kept long-term, really increase the risk of possible leaks. If they are exposed, the attacker can use them for a long time without being noticed.

3. Overusing the Environment Variables

Users might store secrets in environment variables. If these variables are not handled carefully, secrets stored in environment variables can be exposed in logs or debugging output.

4. Lack of Visibility and Auditing

It is very difficult to see how secrets are used across pipelines if not audited regularly. 

Best Practices for Secure Secret Management

1. Use of Dedicated Secret Management Systems

Keep secrets in secure systems such as:

• AWS Secrets Manager
• HashiCorp Vault
• And there are a few other secret management systems

These secrets can be retrieved from the pipeline's secret manager whenever needed. Do not keep secrets in code or pipeline configuration files. 

2. Secure Variables

Few CI/CD systems provide secure environment variables. Secrets in these variables are stored securely and cannot be accessed once they are saved and stored. Printing these secure variables in the pipeline also doesn’t expose these secrets, and they will always be printed as masked values. This practice can be used as an alternative if the secret management system is not a viable option.

3. Use Short-Lived Credentials

Utilize temporary tokens or role-based access (IAM roles or OIDC) to minimize the effects of credential exposure. These tokens can have a lifetime of a few minutes and can be configured to expire after that. Short-lived tokens are sufficient for the pipeline as they just need to be used by the pipeline for a short time to do a specific job. This practice reduces the risk of secrets remaining exposed for a long time and provides better control over them.

4. Apply Least Privilege Access

Access to pipelines shall be provided only within the scope of the work at hand. Example:

• No production credential access should be provided in the build stage
• Limited-scoped credentials access should be provided on the deployment stage

5. Avoid Logs Showing Secret Information

Add secret masking and don’t print or echo sensitive values in the logs. There shouldn’t be any need to print the secrets in the logs, and users should avoid it at any cost.

6. Credentials Should Be Rotated and Audited Regularly

Regularly rotate credentials and monitor them to enhance security and minimize the long-term risk. This is always a best practice to rotate secrets regularly at a set interval. Certain monitoring systems are available that can be configured to alert as and when any anomaly is found in the usage of those credentials. 

Example: Using Secrets in CI/CD Pipelines

Here is a simple example of using secrets securely in a CI/CD pipeline:

env:

 DB_PASSWORD: ${{ secrets.DB_PASSWORD }}

Key points:

• Secrets are stored securely in the platform
• They are injected only during pipeline execution
• They are not hardcoded in the repository

CI/CD Secret Management Checklist

Apply this list to help make your pipeline always remain secure: 

• Do not store secrets in source code or configuration files.
• Create centralized secret management systems and store secrets there.
• Prefer short-term credentials over long-term.
• Limit access by least privilege by proving least access.
• Mask secrets in the logs. Avoid it at any cost, and it should never be required. 
• Rotate credentials periodically and make it an automated rotation.
• Monitor and audit secret usage continuously and get alerted if something is unusual.

Secrets Management in the DevOps Supply Chain

CI/CD pipelines are the backbone of the software supply chain. They build, validate, and deploy software to production. As a result, security in pipelines is closely related to system security. 

Good secret management practices not only mitigate risk but also enhance reliability, visibility, and operational confidence.

Conclusion

CI/CD pipelines are not just automation tools; they are essential to infrastructure. They must be security-centric from the beginning, as systems become more complex later. 

Adopting best practices such as centralized secret storage, least-privileged access, and credential rotation can help organizations reduce risk while also benefiting from the speed and agility of CI/CD systems.