In modern software systems, secrets are the keys to the kingdom. Time and time again, we've observed security breaches caused by the lack or improper safeguarding of secrets, leading to severe financial and reputational damage for organizations. The 2024 Data Breach Investigations Report by Verizon identified the use of stolen credentials as one of the top five root causes of attackers gaining unauthorized access.

This recurring issue underscores the critical need for an effective secrets management strategy to protect sensitive data. As organizations increasingly rely on cloud services, microservices architectures, and DevOps practices, the need for isolating secrets from code and safeguarding them becomes critical.

This Refcard delves into the core practices of secrets management, common challenges, and its pivotal role in modern security environments, providing a comprehensive guide for organizations aiming to enhance the security posture of their secrets management strategy.

In secrets management, "secrets" refer to the sensitive information used to secure access to systems and data. Secrets management solutions securely store and manage secrets at scale, ensuring that only authorized users and systems can access the specific secrets they need. This approach reduces the risk of exposure and protects against unauthorized access via leaked secrets.

Secrets typically include:

• Passwords required for user accounts or system access (typically stored in separate password managers)
• API keys used for accessing APIs
• Encryption keys used for encrypting and decrypting data
• Database credentials, including credentials such as connection strings
• Tokens, including access tokens (e.g., OAuth tokens), used for authentication
• Certificates used for securing communications (e.g., X.509, TLS certificates)
• SSH keys required to secure Shell (SSH) access to servers

Each of these secrets requires careful handling, storage, and access control to ensure they remain confidential and secure. This is where secrets management comes into play. At its core, secrets management is about maintaining the security and integrity of sensitive information within organizations. Properly managing secrets is pivotal in preventing unauthorized access and data breaches.

Keep in mind: User and/or customer data is usually not categorized as a "secret" in this context but rather as sensitive or personal data that needs to be protected through data protection security controls.

Key Benefits of Secrets Management

To ensure that secrets are stored securely and accessed only by authorized entities, centralized secrets management solutions provide security benefits such as encryption, fine-grained access controls, monitoring, and auditing.

Encryption

Encryption is a key aspect of secrets management, safeguarding secrets both at rest and in transit. It not only protects sensitive data from accidental exposure but also ensures security in the catastrophic event where the secrets management solution itself is compromised. Moreover, secrets management solutions protect data against tampering by employing cryptography and digital signatures.

Regulations and Compliance

Regulations such as GDPR and PCI-DSS mandate strict controls over the handling of sensitive data, including secrets. Secrets management solutions ensure that access to secrets is strictly maintained and monitored, helping organizations improve their compliance with standards and regulations. Additionally, their ability to generate comprehensive audit trails facilitates easier reporting and verification processes during compliance audits.

Common Challenges

When managing secrets, organizations often face the following common challenges.

Single Source of Truth

Lack of a single source of truth for secrets creates inconsistent practices and duplicated efforts, increasing the risk of exposure and security breaches. Not having a single source of truth for secrets leads to operational inefficiencies, often resulting in outdated or compromised secrets remaining in use. It also complicates tracking and reporting on secret usage, making audit and compliance efforts more difficult.

Consistent Secrets Management Policy

Enforcing a uniform policy for managing secrets across an entire organization becomes particularly challenging without dedicated secrets management tools. This often leads to a scattered approach: inconsistent practices with unmanaged secrets and a lack of centralized monitoring and control. Unenforced or inconsistent policies expose organizations to serious security risks. A centralized secrets management solution standardizes secrets management policies, ensuring consistency and security. 

Secrets Misuse

Further enhancing data protection, centralized secrets management solutions reduce the risk of secrets being misused. By serving as a single source of truth for secrets, consistency and control over secrets distribution can be ensured, which minimizes the likelihood of secrets being mishandled or leaked.

Secrets Rotation and Expiry

Secrets must be rotated periodically and revoked when compromised; however, manual secrets rotation and revocation can be challenging, or even unmanageable, in complex environments. Secrets management solutions include automated secrets rotation, revocation, and expiration features, further reducing the risk of long-term exposure and unauthorized access.

Secrets Sprawl

Without a secrets management solution, organizations often struggle to manage secrets effectively due to the complexities and scale of modern IT environments. The proliferation of applications, services, and infrastructure increases the risk of secrets sprawl, inconsistent handling, and security vulnerabilities. Addressing these challenges is crucial to maintaining a secure and compliant operational posture.

Centralized secrets management helps address secrets sprawl by providing a single source of truth for secrets, ensuring that all secrets are stored and accessed securely. This approach fosters a more secure and compliant environment for handling secrets across various services, teams, and projects, which reduces the risk of their uncontrolled distribution and mismanagement.

Scalability

Scalability in secrets management is challenging due to the exponential increase in secrets and identities as well as the need for immediate access and secure distribution in rapidly changing and dynamic infrastructures. Managing and monitoring the lifecycle of secrets becomes harder as the number of secrets and identities grows. Integrating with diverse technologies and ensuring secrets consistency across multiple locations and environments add to the complexity. Using a centralized secrets management solution is key for addressing these challenges as they offer scalability, automation, and seamless integration across various environments.

Adopting secrets management solutions and the following core practices will ensure an easier and more effective secrets management strategy across your organization.

Core Practices for Centralized Secrets Management

This section explores the core practices of centralized secrets management that address the common challenges described previously.

Organize Secrets Across Projects and Teams

Core practice: Store secrets in a centralized and secure repository

Typically called vaults, these repositories are designed to protect secrets against unauthorized access. Organizations can centrally control their secrets, ensuring consistent policy enforcement and auditing. This approach offers the visibility required to simplify the secrets lifecycle, enforce policies, achieve compliance, monitor secrets access, identify threats, and revoke privileges when necessary. It reduces the risk of secrets being mishandled or leaked and ensures consistent practices for handling sensitive data across all teams and projects.

Secrets Sprawl Elimination

Core practice: Avoid secrets sprawl through centralized secrets management

A common risk in handling secrets is secrets sprawl, which occurs when secrets are dispersed across multiple locations and systems. This dispersion often happens when different teams or individuals use ad hoc methods to manage secrets, leading to various insecure practices like embedding secrets directly into source code or configuration files. The scattered nature of these secrets makes it difficult to maintain consistent security policies, increasing the risk of unauthorized access and data breaches. With the consistent use of a centralized secrets management strategy, the uncontrolled distribution of secrets across various locations is mitigated.

Avoid Duplicate Secrets

Core practice: Avoid duplicating secrets across teams and/or projects

Duplicate secrets can occur when multiple teams and/or projects create and use their own copies of the same secret across services, clusters, or regions without proper coordination. This problem increases the attack surface and the risk of unauthorized access. Importantly, it introduces operational inefficiencies, making secrets management, rotation, and revocation time consuming and challenging.

Centralized secrets management solutions provide tools for detecting, reporting, and eliminating duplicate secrets (i.e., deduplication). Centralization ensures that each secret is unique across environments. Individual secrets are stored only once, centrally managed, and accessed through a unified interface, reducing the risks and challenges introduced by duplicated secrets.

Secrets Synchronization

Core practice: Sync newly created and updated secrets across your estate

A core function of secrets management is the ability to automatically propagate, or sync, newly created and updated secrets across all dependent systems and environments. When a secret is updated, centralized secrets management systems ensure that the updated secret is seamlessly distributed across all applications and services that rely on it. This automated change propagation is vital for reducing the risk of discrepancies and unauthorized access, ensuring that all systems consistently use the most up-to-date secrets.

Secrets synchronization is particularly crucial in dynamic environments, such as microservices architectures, where operational efficiency is key to success. By eliminating manual secrets updates, not only do we save time and minimize the potential for human error, but we also ensure compliance with the defined security, availability, and reliability SLAs.

Figure 1: High-level diagram of secrets synchronization

Secrets Version Tracking and Rollback

Core practice: Maintain version history for each secret

Beyond the need for propagating changes, maintaining a ledger of historical changes for each secret is crucial when a secret is updated. Version tracking enables teams to keep a comprehensive record of changes and updates over time, which is particularly important for auditing and compliance. This detailed history provides a clear account of secret changes and usage.

Centralized secrets management provides version tracking and roll-back features, ensuring that secrets can be restored to a known good state if necessary. For instance, if an update to a secret unexpectedly disrupts services or applications that rely on it, version tracking allows teams to identify the exact change that caused the issue and take corrective action, such as rolling back to a previous state. Roll-back features add a layer of resilience, making recovery straightforward and efficient, even if mistakes occur during secret updates.

Core Practices for Injecting Secrets

To minimize the risk of exposure and avoid storing secrets in plaintext within source code or configuration files, secrets management offers secure methods for injecting secrets directly into apps and services at runtime. Depending on the application's type and environment, secrets are injected via various mechanisms into the application memory in a transient manner, guaranteeing that sensitive data remains protected and ephemeral.

Table 1: How secrets are typically injected per environment

Secrets at Runtime

Core practice: Adopt SDKs to securely inject secrets at runtime

In most cases, leveraging the software development kits (SDKs) provided by secrets management solutions is the preferred way to securely inject secrets into applications at runtime. By integrating these SDKs, developers can programmatically interact with the secrets management service, thus avoiding the risks associated with hardcoding or storing secrets in plaintext within the application's codebase. The SDK handles authentication and secrets retrieval, ensuring that secrets remain protected and transient.

Additionally, SDKs come with features like automatic secrets rotation, renewal, and in-memory client-side caching. Developers can configure the SDK using environment variables or role-based access settings to specify authentication credentials, secret paths, and access policies. Once set up, the SDK seamlessly integrates with the application, injecting secrets only when needed and promptly removing them from memory after use.

Note: One drawback is that developers need to bundle the SDK as a library dependency within their applications.

Secrets in CI/CD Pipelines

Core practice: Create secrets that expire once the CI/CD pipeline job is complete

Secrets are essential not only during an application's runtime but also within continuous integration and continuous deployment (CI/CD) pipelines. Build, test, and deployment workflows often require secrets to access external services or resources, such as cloud providers, databases, and APIs. Unfortunately, the security of these secrets is frequently overlooked.

Different CI/CD platforms offer various methods for injecting and handling secrets. For instance, Jenkins and CircleCI typically store secrets as environment variables, whereas GitHub Actions uses secret variables. While GitLab allows you to store secrets as CI/CD variables, it is generally advisable to utilize an external secrets management provider to keep these secrets outside the GitLab instance for better security.

A standard practice in CI/CD is to use dynamic, short-lived secrets that expire once the pipeline job is complete. This approach helps prevent accidental exposure and minimizes the risk of secrets being inadvertently passed down during the CI/CD process.

Secrets in Kubernetes

Core practice: Do not rely on the default Kubernetes Secrets

In Kubernetes environments, Secrets are, by default, not as secure as one might expect. They are stored unencrypted in the API server's underlying data store (etcd), which poses a significant security risk. Anyone with API access or access to etcd can retrieve or modify these secrets. Therefore, it is crucial to follow security best practices to protect these secrets and to consider integrating with external secrets management solutions.

Figure 2: Kubernetes Secrets are not encrypted by default

Core practice: Use External Secrets Operator and Sealed Secrets in Kubernetes

To elevate the security of Secrets in Kubernetes, open-source options like External Secrets Operator and Sealed Secrets can be considered:

• The External Secrets Operator allows Kubernetes to integrate with external secrets management tools, enabling secure access to external secrets within clusters.
• Sealed Secrets utilizes asymmetric cryptography to generate encrypted Kubernetes Secret templates. These templates can be safely shared and stored in source code repositories without exposing sensitive data.

Secrets in Service Meshes

Core practice: Integrate service meshes with external secrets management tools

Service meshes enhance the security of communication between microservices by utilizing features like mutual TLS (mTLS) to encrypt traffic. This process inherently requires the management of certificates and keys, commonly referred to as secrets. These secrets must be securely stored, distributed, and periodically rotated. For example, Istio, a widely used open-source service mesh, manages its secrets — such as TLS certificates and private keys — using Kubernetes Secrets and dynamically delivers these Secrets to Istio proxies via the Secret Discovery Service (SDS).

Storing secrets directly as Kubernetes Secrets isn't always the best approach. Instead, integrating service meshes with external secrets management tools can greatly enhance both the security and operational efficiency of secrets management. These integrations allow external tools to inject secrets, such as certificates and private keys, into the service mesh.

In practice, this process can look like the following:

• Injection involves delivering secrets directly into the service mesh control plane pods, minimizing reliance on Kubernetes Secrets during the bootstrapping phase.
• Alternatively, secrets can be injected via a sidecar container that runs alongside the application container within the same pod. In this setup, the application container authenticates with the secrets management container to securely retrieve the necessary secrets.

Core Practices for Automated Secrets Management

The manual management of secrets is often error-prone, inefficient, and not scalable, potentially leading to significant security vulnerabilities. Secrets management provides a more robust and scalable approach by automating the entire secrets lifecycle, thereby enhancing both reliability and operational efficiency.

Automated Secrets Rotation

Core practice: Proactively and regularly rotate secrets

Secrets rotation involves changing credentials periodically to mitigate the risk of long-term exposure and unauthorized access due to compromised secrets. Manual rotation of secrets can be cumbersome, primarily due to concerns about potential downtime and the complexities of scaling the process across multiple environments, clouds, and providers. Consequently, manual secrets rotation is frequently neglected or is poorly implemented. 

Automating secrets rotation can greatly enhance efficiency and ensure consistent enforcement of secrets management policies. Secrets management solutions offer rotation policies that specify how frequently secrets should be automatically rotated. This automation not only simplifies the rotation process but also helps eliminate the risk of downtime by implementing secrets rotation strategies such as the two secrets strategy, staged rollout, and grace periods.

Automated Secrets Deletion

Core practice: Define secrets deletion policies based on conditions

Automated secrets deletion is a key convention of a comprehensive secrets management strategy that enhances security by ensuring that obsolete or unused secrets do not linger, thus minimizing potential vulnerabilities. To implement this, secrets management solutions integrate with the organization's infrastructure to monitor and manage the deletion policies of secrets.

In practice, define policies to automate secret deletion based on specific conditions:

• Time-based deletion – Secrets can be deleted after a predefined period of inactivity.
• Resource-linked deletion – Secrets can be deleted upon termination of a related resource they were used by.
• Secret rotation events – Policies can trigger deletion of the old secret version when a new rotation occurs.

Core practice: Include a recovery window during which a secret can be restored

Recovery windows ensure operational continuity and reduce the risk of accidental data loss. Event-driven architectures can further enhance automated secrets deletion. For instance, triggering deletion workflows in response to specific events — such as the decommissioning of a virtual machine or the expiration of a secret — ensures that secrets do not persist longer than necessary. This proactive approach ensures that secrets are deleted promptly, consistently, and securely, aligning with standard practices in secrets management.

Automated Certificate Expiration Alerts

Core practice: Create automated certificate expiration alerts

Expired certificates can cause service disruptions and vulnerabilities. Proactive certificate renewal in large environments can be challenging if there is lack of automation, visibility, and real-time monitoring. Effective certificate management ensures validity and timely renewal, and automation streamlines certificate renewal and deployment, minimizing the risk of human error. Automated expiration alerts notify administrators, helping to prevent disruptions and security incidents.

In practice, automated certificate expiration alerts can be delivered through:

• Enabling notifications within the user interface or via email
• Adding webhooks to incident management platforms
• Creating triggering actions within automation workflows

Core Practices for Enterprise Security and Compliance

This section explores the key practices of centralized secrets management that help organizations achieve enterprise-level security.

Environment Segregation

Core practice: Segregate secrets based on their environment and scope

A common mistake that can lead to security incidents is the accidental leakage of secrets that are intended for pre-production environments into production releases. To mitigate this risk, environment segregation is used to ensure that secrets are only accessible by authorized identities within their appropriate environment, reducing the risk of cross-environment leakage and maintaining data confidentiality.

In practice, environment segregation for secrets can include the following:

• Separate secrets by environment (e.g., development, staging, production) to guarantee that secrets are only accessible by authorized identities in the appropriate context.
• Isolate pre-production secrets by storing them in a dedicated secure vault, separate from the production vault. This can be achieved by creating distinct vaults accessible only by authorized identities.
• If available, create separate environments for each secrets vault, with different policies and protection rules tailored for each vault.
• Implement a policy for shorter secrets lifespans in development and testing environments, ensuring they do not outlive the development/test lifecycle.

Dynamic Secrets

Core practice: Use on-demand, time-restricted secrets

Dynamic secrets are on-demand, time-bound, single-use secret leases that are valid for a defined period. Each lease has a mandatory time to live (TTL) and is automatically deleted upon expiration. These secrets are designed to be used in a scoped manner, with each service requesting a new lease rather than reusing the same one across multiple services.

The primary advantage of dynamic secrets is their ephemeral nature, which enhances security. Since they are only valid for a specific duration, the risk of long-term exposure is limited. Additionally, dynamic secrets improve auditability as each lease is only available once in its original context.

Figure 3: High-level diagram of dynamic secrets

A dynamic secret lease is a temporary instance that is revealed only once. Dynamic secrets integration facilitates the provisioning and revocation of these leases through connections with third-party services. For instance, when configuring an identity management secrets integration, a policy can be attached to the user during lease provisioning. Upon TTL expiration, the secret lease is revoked and the secret gets deleted.

Encrypting Secrets

Core practice: Encrypt your secrets at rest and in transit

Encrypting secrets at rest is crucial for protecting them from unauthorized access and breaches, even if the storage infrastructure is compromised, and server-side encryption also ensures that secrets are securely stored and protected from exposure.

Centralized secrets management solutions encrypt secrets at rest using advanced techniques such as envelope encryption and algorithms (e.g., AES-256), transmitting them securely over TLS. Additionally, they often provide features like bring your own keys (BYOK) or customer-managed encryption keys (CMEK), giving users full control over their encryption keys. To ensure their security, at no point are the secrets stored in clear text.

Using a managed secrets management solution reduces the burden on developers to handle encryption themselves, allowing them to focus more on development tasks.

Secrets Re-Encryption

Core practice: Proactively re-encrypt secrets to adapt to evolving encryption standards

A standard method for maintaining the security of secrets at rest is to re-encrypt them proactively when encryption standards change. Advanced secrets management solutions automate this re-encryption process for consistent protection without manual intervention. These solutions continuously monitor encryption standards and automatically initiate re-encryption whenever updates are required.

Core practice: Immediately re-encrypt secrets if the encryption keys are compromised

Another trigger for secrets re-encryption is the detection of key compromise. When a key compromise is identified, the security of all secrets encrypted with that key is at risk. Reactive re-encryption involves generating new, secure encryption keys and re-encrypting all affected secrets with the new keys. This minimizes the damage from a security breach by rendering compromised keys useless. This practice is especially important when you use BYOK or CMEK features.

Fine-Grained Access Control

Core practice: Limit secrets exposure through restricted access

Fine-grained role-based access control is a core security feature in secrets management that ensures only authorized identities can access specific secrets, minimizing the risk of unauthorized access. This approach enables organizations to manage and restrict secrets access based on precise criteria such as identities, roles, permissions, groups, teams, and environments.

At the center of this practice is a robust policy engine. Administrators can create access control list (ACL) policies to enforce the principle of least privilege. These policies specify exactly which identities can access particular secrets under certain conditions. Misconfigurations of role-based access control policies are common issues — often stemming from overly broad policies or reliance on default configurations — and can inadvertently expose secrets to unauthorized users.

Core Practices for Monitoring and Resiliency

This section explores the essential practices for monitoring and resilience in secrets management.

Access and Audit Logs

Core practice: Conduct regular secrets audits and review access logs

The real-time monitoring and logging of secrets access provides imperative visibility into who is accessing secrets and when, aiding in audit and compliance efforts. These access and audit logs create a detailed record of secrets access and usage, allowing organizations to detect and respond to unauthorized access. Knowing when changes, especially high-risk ones, are made and by whom is vital for maintaining a highly performant, secure environment.

Understanding what identity has accessed a specific secret, when they accessed it, and through which medium the secret was accessed is essential. Secrets access logs enable administrators to see detailed information about each access event, including the actor, access method, and access times. Importantly, these logs do not record the plaintext value of the secrets; instead, some solutions log the secret's value hash, such as a salted hash using a cryptographic function (e.g., HMAC-SHA256).

Advanced secrets management solutions also offer alerting on key operational conditions, which ensures that any suspicious activity is promptly identified and addressed.

Resilient Secrets

Core practice: Ensure secrets remain resilient in case of accidents or disasters

Emergencies and accidents can put organizations without a comprehensive secrets management resiliency plan at risk. Organizations can guarantee secrets resiliency either via offline backups or by implementing disaster recovery solutions. Creating offline backups of your secrets provides a safety net so that access can be restored if access to the secrets vault is lost. However, backups often introduce operational inefficiencies that may not always be desirable.

Disaster recovery options to avoid these operational challenges in practice:

• If backups are needed, create encrypted snapshots: Generate a read-only file containing an encrypted snapshot of current secrets. These backups should only be decrypted using the secrets management solution during restoration.
• Automate back-up processes: Set up automated back-up processes to simplify the task and reduce the risk of human error compared to manual backups.
• Implement disaster recovery: Use secrets management solutions that offer disaster recovery and high-availability capabilities. Enable secrets replication to separate instances, either on-premises or in the cloud, with automated syncing across instances to ensure continuity.

Centralized secrets management significantly enhances visibility and control, allowing organizations to track the entire lifecycle of each secret from creation to retirement. This comprehensive visibility facilitates better auditing and compliance while swiftly identifying outdated or unused secrets that could pose security risks. With a central system serving as the source of truth for secrets, organizations can:

• Standardize policies for secrets rotation and deletion
• Set expiration alerts
• Enforce strong authentication mechanisms and fine-grained access controls
• Segregate secrets by environment
• Utilize dynamic secrets
• Eliminate secrets sprawl
• Revoke access promptly and when necessary
• Monitor secrets usage
• Simplify compliance audits

However, even the most robust centralized secrets management solutions can be undermined by human error. Common mistakes such as sharing secrets between processes via the command line, storing secrets within Dockerfiles, or sending secrets through emails can significantly undermine these systems' effectiveness.

For further reading and to engage with community-driven projects, consider resources like the OWASP Secrets Management Cheat Sheet and the OWASP WrongSecrets project. These resources offer valuable insights and guidelines to help strengthen your secrets management strategy and avoid common pitfalls.