Secrets Management

Today's digital businesses are expected to innovate, execute, and release products at a lightning-fast pace. The widespread adoption of automation tools, when coupled with DevOps and DevSecOps tools, is instrumental to these businesses achieving increased developer velocity and faster feedback loops. This eventually helps in shortening release cycles and improving the product quality in an iterative manner.

Though the shift to microservices and containerized applications and the adoption of open source are helping developers ship faster, they also pose challenges related to compliance and security. As per the Hidden In Plain Sight report from 1Password, DevOps and IT teams in enterprises continually face challenges posed by leakage of secrets, insecure sharing of secrets, and manual secrets management, amongst others.

There are significant complexities involved in managing secrets like API keys, passwords, encryption keys, and so on for large-scale projects. Let’s take a deep dive into the integral aspects of secrets management in this article.

What Is Secrets Management?

In simple terms, secrets are non-human privileged credentials that give developers the provision to access resources in applications, containers, and so on. Akin to passwords management, secrets management is a practice whereby secrets (e.g., access tokens, passwords, API keys, and so on) are stored in a secure environment with tighter access controls.

Managing secrets can become mayhem as the complexity and scale of an application grows over time. Additionally, there could be situations where secrets are being shared across different blocks across the technology stack. This could pose severe security threats, as it opens up the back doors for malicious actors to access your application.

Secrets management ensures that sensitive information is never hard-coded and available only in an encrypted format. Secure access to sensitive data in conjunction with RBAC (role-based access controls) is the secret sauce of secrets management.

Challenges of Secret Management

There might be numerous cases where developers could have accidentally used hard-coded plain-text format credentials in their code or configuration files. The repercussions to the business could be huge if the respective files housing the secrets are pushed to the designated public repository on GitHub (or any other popular code hosting platforms).

The benefits offered by multi-cloud infrastructures, containerized applications, IoT/IIoT, CI/CD, and similar advancements can be leveraged to the maximum extent by also focusing on efficient management of secrets. Educating development and DevOps teams about application security is the foremost step to build a security-first culture within the team.

Here are the major challenges DevOps and DevSecOps teams face when managing secrets:

Secrets Sprawl

This scenario normally arises when the team’s (and/or organization’s) secrets are distributed across the organization. Digital-first organizations are increasingly using containers and cloud-based tools to increase developer velocity, save costs, and expedite releases. The same principle also applies for the development and testing of IoT-based applications.

Depending on the scale and complexity of the applications, there is a high probability that the secrets are spread across:

• Containerized microservices-based applications (e.g., Kubernetes, OpenShift, Nomad)
• Automated E2E testing/tracing platforms (e.g., Prometheus, Graphite)
• Internally developed tools/processes
• Application servers and databases
• DevOps toolchain 

The items in the above list vary depending on the scale, size, and complexity of the application. Providing RBAC, using strong rotating passwords, and avoiding password sharing are some of the simple practices that must be followed at every level within the team/organization.

Proliferation of Cloud Developer and Testing Tools

Irrespective of the size and scale of the project, development teams look to maximize the usage of cloud development tools like GCP (Google Cloud Platform), Microsoft Azure, AWS (Amazon Web Services), Kubernetes, and more.

Cloud tools definitely expedite processes related to development and testing, but they must be used while keeping security practices at the forefront. Any compromise of keys used for accessing the respective cloud platform (e.g., AWS keys) could lead to financial losses.

AWS Credentials publicly exposed in a repository

With so much at stake, DevOps and development teams must ensure that any sort of keys are not available in human-readable format in public domains (e.g., GitHub repositories). Organizations focusing on community-led growth (CLG) for evangelizing their product or developer tool need to ensure that their users do not leave any keys out in the open! If keys are left publicly accessible, hackers could exploit your platform for malicious reasons.

Manual processes for managing secrets, data security when using third-party resources (e.g., APIs), and end-to-end visibility from the security lens are other challenges that organizations face with secrets management. 

Best Practices of Secret Management

There is no one-size-fits-all approach in securely managing secrets, since a lot depends on the infrastructure, product requirements, and other such varying factors.

Keeping variables aside, here are some of the best practices when it comes to efficient and scalable management of secrets:

Use RBAC (Role-Based Access Control)

Every project and organization has sensitive data and resources that must be accessible only by trusted users and applications. Any new user in the system must be assigned default privilege (i.e., minimum access control). Elevated privileges must be available only to a few members in the project or organization.

The admin (or super-admin) must have the rights to add or revoke privileges of other members on a need basis. Escalation of privileges must also be done on a need basis, and only for a limited time. Proper notes must be added when giving/revoking privileges so that all the relevant project stakeholders have complete visibility.

Use Secure Vaults

In simple terms, a vault is a tool that is primarily used for securing any sensitive information (e.g., passwords, API keys, certificates, and so on). Local storage of secrets in a human readable form is one of the worst ways to manage secrets.

This is where secure vaults can be extremely useful, as they provide a unified interface to any secret, along with providing a detailed audit log. Secure vaults can also be used for instrumenting role-based access control (RBAC) by specifying access privileges (authorization). Hashicorp Vault Helm chart and Vault for Docker are two of the popular vault managers that can be used for running vault services, accessing and storing secrets, and more.

Since most applications leverage the potential of the cloud, it is important to focus on data security when it is in transit or at rest. This is where EaaS (Encryption as a Service) can be used for offloading encryption needs of applications to the vault before the data is stored at rest.

Rotate Keys Regularly

It is a good security practice to reset keys after a few weeks or months. One practice is to manually regenerate the keys, since there is a probability that applications using the secrets might be leaving behind traces in the log files or centralized logging systems. Attackers can get back-door access to the logs and use it to exfiltrate secrets.

Additionally, co-workers might unintentionally leak secrets outside the organization. To avoid such situations, it is recommended to enable rotation of secrets in the respective secrets management tool. For instance, Secrets Manager rotation in AWS Secrets Manager uses an AWS Lambda function to update the secret and the database.

Above all, teams should have practices in place to detect unauthorized access to the system. This will help in taking appropriate actions before significant damage can be done to the business.

Why Implement Secrets Management in a DevSecOps Pipeline?

Accelerated release cycles and faster developer feedback can only be achieved if the code is subjected to automated tests in a CI/CD pipeline. The tests being run in the CI pipeline might require access to critical protected resources like databases, HTTP servers, and so on.

Even running unit tests inside Docker containers is also a common practice, but developers and QAs need to ensure that secrets are not stored inside a Dockerfile. Secret management tools can be used in conjunction with popular CI/CD tools (e.g., Jenkins) whereby keys and other secrets are managed in a centralized location. Secrets are also stored with encryption and tokenization.