Adopting secrets management solutions and the following core practices will ensure an easier and more effective secrets management strategy across your organization.
Core Practices for Centralized Secrets Management
This section explores the core practices of centralized secrets management that address the common challenges described previously.
Organize Secrets Across Projects and Teams
Core practice: Store secrets in a centralized and secure repository
Typically called vaults, these repositories are designed to protect secrets against unauthorized access. Organizations can centrally control their secrets, ensuring consistent policy enforcement and auditing. This approach offers the visibility required to simplify the secrets lifecycle, enforce policies, achieve compliance, monitor secrets access, identify threats, and revoke privileges when necessary. It reduces the risk of secrets being mishandled or leaked and ensures consistent practices for handling sensitive data across all teams and projects.
Secrets Sprawl Elimination
Core practice: Avoid secrets sprawl through centralized secrets management
A common risk in handling secrets is secrets sprawl, which occurs when secrets are dispersed across multiple locations and systems. This dispersion often happens when different teams or individuals use ad hoc methods to manage secrets, leading to various insecure practices like embedding secrets directly into source code or configuration files. The scattered nature of these secrets makes it difficult to maintain consistent security policies, increasing the risk of unauthorized access and data breaches. With the consistent use of a centralized secrets management strategy, the uncontrolled distribution of secrets across various locations is mitigated.
Avoid Duplicate Secrets
Core practice: Avoid duplicating secrets across teams and/or projects
Duplicate secrets can occur when multiple teams and/or projects create and use their own copies of the same secret across services, clusters, or regions without proper coordination. This problem increases the attack surface and the risk of unauthorized access. Importantly, it introduces operational inefficiencies, making secrets management, rotation, and revocation time consuming and challenging.
Centralized secrets management solutions provide tools for detecting, reporting, and eliminating duplicate secrets (i.e., deduplication). Centralization ensures that each secret is unique across environments. Individual secrets are stored only once, centrally managed, and accessed through a unified interface, reducing the risks and challenges introduced by duplicated secrets.
Secrets Synchronization
Core practice: Sync newly created and updated secrets across your estate
A core function of secrets management is the ability to automatically propagate, or sync, newly created and updated secrets across all dependent systems and environments. When a secret is updated, centralized secrets management systems ensure that the updated secret is seamlessly distributed across all applications and services that rely on it. This automated change propagation is vital for reducing the risk of discrepancies and unauthorized access, ensuring that all systems consistently use the most up-to-date secrets.
Secrets synchronization is particularly crucial in dynamic environments, such as microservices architectures, where operational efficiency is key to success. By eliminating manual secrets updates, not only do we save time and minimize the potential for human error, but we also ensure compliance with the defined security, availability, and reliability SLAs.
Figure 1: High-level diagram of secrets synchronization
Secrets Version Tracking and Rollback
Core practice: Maintain version history for each secret
Beyond the need for propagating changes, maintaining a ledger of historical changes for each secret is crucial when a secret is updated. Version tracking enables teams to keep a comprehensive record of changes and updates over time, which is particularly important for auditing and compliance. This detailed history provides a clear account of secret changes and usage.
Centralized secrets management provides version tracking and roll-back features, ensuring that secrets can be restored to a known good state if necessary. For instance, if an update to a secret unexpectedly disrupts services or applications that rely on it, version tracking allows teams to identify the exact change that caused the issue and take corrective action, such as rolling back to a previous state. Roll-back features add a layer of resilience, making recovery straightforward and efficient, even if mistakes occur during secret updates.
Core Practices for Injecting Secrets
To minimize the risk of exposure and avoid storing secrets in plaintext within source code or configuration files, secrets management offers secure methods for injecting secrets directly into apps and services at runtime. Depending on the application's type and environment, secrets are injected via various mechanisms into the application memory in a transient manner, guaranteeing that sensitive data remains protected and ephemeral.
Table 1: How secrets are typically injected per environment
| Environment |
Secrets Injection Techniques |
| Runtime |
Environment variables, configuration files (e.g., JSON, YAML), secrets management SDKs |
| CI/CD pipeline |
Environment variables, secret variables, secrets management integrations |
| Kubernetes |
Kubernetes Secrets, External Secrets Operator, Sealed Secrets, data volume mounts, environment variables |
| Service mesh |
Injecting secrets into service mesh control plane pods, sidecar injection, secret discovery service |
Secrets at Runtime
Core practice: Adopt SDKs to securely inject secrets at runtime
In most cases, leveraging the software development kits (SDKs) provided by secrets management solutions is the preferred way to securely inject secrets into applications at runtime. By integrating these SDKs, developers can programmatically interact with the secrets management service, thus avoiding the risks associated with hardcoding or storing secrets in plaintext within the application's codebase. The SDK handles authentication and secrets retrieval, ensuring that secrets remain protected and transient.
Additionally, SDKs come with features like automatic secrets rotation, renewal, and in-memory client-side caching. Developers can configure the SDK using environment variables or role-based access settings to specify authentication credentials, secret paths, and access policies. Once set up, the SDK seamlessly integrates with the application, injecting secrets only when needed and promptly removing them from memory after use.
Note: One drawback is that developers need to bundle the SDK as a library dependency within their applications.
Secrets in CI/CD Pipelines
Core practice: Create secrets that expire once the CI/CD pipeline job is complete
Secrets are essential not only during an application's runtime but also within continuous integration and continuous deployment (CI/CD) pipelines. Build, test, and deployment workflows often require secrets to access external services or resources, such as cloud providers, databases, and APIs. Unfortunately, the security of these secrets is frequently overlooked.
Different CI/CD platforms offer various methods for injecting and handling secrets. For instance, Jenkins and CircleCI typically store secrets as environment variables, whereas GitHub Actions uses secret variables. While GitLab allows you to store secrets as CI/CD variables, it is generally advisable to utilize an external secrets management provider to keep these secrets outside the GitLab instance for better security.
A standard practice in CI/CD is to use dynamic, short-lived secrets that expire once the pipeline job is complete. This approach helps prevent accidental exposure and minimizes the risk of secrets being inadvertently passed down during the CI/CD process.
Secrets in Kubernetes
Core practice: Do not rely on the default Kubernetes Secrets
In Kubernetes environments, Secrets are, by default, not as secure as one might expect. They are stored unencrypted in the API server's underlying data store (etcd), which poses a significant security risk. Anyone with API access or access to etcd can retrieve or modify these secrets. Therefore, it is crucial to follow security best practices to protect these secrets and to consider integrating with external secrets management solutions.
Figure 2: Kubernetes Secrets are not encrypted by default
Core practice: Use External Secrets Operator and Sealed Secrets in Kubernetes
To elevate the security of Secrets in Kubernetes, open-source options like External Secrets Operator and Sealed Secrets can be considered:
- The External Secrets Operator allows Kubernetes to integrate with external secrets management tools, enabling secure access to external secrets within clusters.
- Sealed Secrets utilizes asymmetric cryptography to generate encrypted Kubernetes Secret templates. These templates can be safely shared and stored in source code repositories without exposing sensitive data.
Secrets in Service Meshes
Core practice: Integrate service meshes with external secrets management tools
Service meshes enhance the security of communication between microservices by utilizing features like mutual TLS (mTLS) to encrypt traffic. This process inherently requires the management of certificates and keys, commonly referred to as secrets. These secrets must be securely stored, distributed, and periodically rotated. For example, Istio, a widely used open-source service mesh, manages its secrets — such as TLS certificates and private keys — using Kubernetes Secrets and dynamically delivers these Secrets to Istio proxies via the Secret Discovery Service (SDS).
Storing secrets directly as Kubernetes Secrets isn't always the best approach. Instead, integrating service meshes with external secrets management tools can greatly enhance both the security and operational efficiency of secrets management. These integrations allow external tools to inject secrets, such as certificates and private keys, into the service mesh.
In practice, this process can look like the following:
- Injection involves delivering secrets directly into the service mesh control plane pods, minimizing reliance on Kubernetes Secrets during the bootstrapping phase.
- Alternatively, secrets can be injected via a sidecar container that runs alongside the application container within the same pod. In this setup, the application container authenticates with the secrets management container to securely retrieve the necessary secrets.
Core Practices for Automated Secrets Management
The manual management of secrets is often error-prone, inefficient, and not scalable, potentially leading to significant security vulnerabilities. Secrets management provides a more robust and scalable approach by automating the entire secrets lifecycle, thereby enhancing both reliability and operational efficiency.
Automated Secrets Rotation
Core practice: Proactively and regularly rotate secrets
Secrets rotation involves changing credentials periodically to mitigate the risk of long-term exposure and unauthorized access due to compromised secrets. Manual rotation of secrets can be cumbersome, primarily due to concerns about potential downtime and the complexities of scaling the process across multiple environments, clouds, and providers. Consequently, manual secrets rotation is frequently neglected or is poorly implemented.
Automating secrets rotation can greatly enhance efficiency and ensure consistent enforcement of secrets management policies. Secrets management solutions offer rotation policies that specify how frequently secrets should be automatically rotated. This automation not only simplifies the rotation process but also helps eliminate the risk of downtime by implementing secrets rotation strategies such as the two secrets strategy, staged rollout, and grace periods.
Automated Secrets Deletion
Core practice: Define secrets deletion policies based on conditions
Automated secrets deletion is a key convention of a comprehensive secrets management strategy that enhances security by ensuring that obsolete or unused secrets do not linger, thus minimizing potential vulnerabilities. To implement this, secrets management solutions integrate with the organization's infrastructure to monitor and manage the deletion policies of secrets.
In practice, define policies to automate secret deletion based on specific conditions:
- Time-based deletion – Secrets can be deleted after a predefined period of inactivity.
- Resource-linked deletion – Secrets can be deleted upon termination of a related resource they were used by.
- Secret rotation events – Policies can trigger deletion of the old secret version when a new rotation occurs.
Core practice: Include a recovery window during which a secret can be restored
Recovery windows ensure operational continuity and reduce the risk of accidental data loss. Event-driven architectures can further enhance automated secrets deletion. For instance, triggering deletion workflows in response to specific events — such as the decommissioning of a virtual machine or the expiration of a secret — ensures that secrets do not persist longer than necessary. This proactive approach ensures that secrets are deleted promptly, consistently, and securely, aligning with standard practices in secrets management.
Automated Certificate Expiration Alerts
Core practice: Create automated certificate expiration alerts
Expired certificates can cause service disruptions and vulnerabilities. Proactive certificate renewal in large environments can be challenging if there is lack of automation, visibility, and real-time monitoring. Effective certificate management ensures validity and timely renewal, and automation streamlines certificate renewal and deployment, minimizing the risk of human error. Automated expiration alerts notify administrators, helping to prevent disruptions and security incidents.
In practice, automated certificate expiration alerts can be delivered through:
- Enabling notifications within the user interface or via email
- Adding webhooks to incident management platforms
- Creating triggering actions within automation workflows
Core Practices for Enterprise Security and Compliance
This section explores the key practices of centralized secrets management that help organizations achieve enterprise-level security.
Environment Segregation
Core practice: Segregate secrets based on their environment and scope
A common mistake that can lead to security incidents is the accidental leakage of secrets that are intended for pre-production environments into production releases. To mitigate this risk, environment segregation is used to ensure that secrets are only accessible by authorized identities within their appropriate environment, reducing the risk of cross-environment leakage and maintaining data confidentiality.
In practice, environment segregation for secrets can include the following:
- Separate secrets by environment (e.g., development, staging, production) to guarantee that secrets are only accessible by authorized identities in the appropriate context.
- Isolate pre-production secrets by storing them in a dedicated secure vault, separate from the production vault. This can be achieved by creating distinct vaults accessible only by authorized identities.
- If available, create separate environments for each secrets vault, with different policies and protection rules tailored for each vault.
- Implement a policy for shorter secrets lifespans in development and testing environments, ensuring they do not outlive the development/test lifecycle.
Dynamic Secrets
Core practice: Use on-demand, time-restricted secrets
Dynamic secrets are on-demand, time-bound, single-use secret leases that are valid for a defined period. Each lease has a mandatory time to live (TTL) and is automatically deleted upon expiration. These secrets are designed to be used in a scoped manner, with each service requesting a new lease rather than reusing the same one across multiple services.
The primary advantage of dynamic secrets is their ephemeral nature, which enhances security. Since they are only valid for a specific duration, the risk of long-term exposure is limited. Additionally, dynamic secrets improve auditability as each lease is only available once in its original context.
Figure 3: High-level diagram of dynamic secrets
A dynamic secret lease is a temporary instance that is revealed only once. Dynamic secrets integration facilitates the provisioning and revocation of these leases through connections with third-party services. For instance, when configuring an identity management secrets integration, a policy can be attached to the user during lease provisioning. Upon TTL expiration, the secret lease is revoked and the secret gets deleted.
Encrypting Secrets
Core practice: Encrypt your secrets at rest and in transit
Encrypting secrets at rest is crucial for protecting them from unauthorized access and breaches, even if the storage infrastructure is compromised, and server-side encryption also ensures that secrets are securely stored and protected from exposure.
Centralized secrets management solutions encrypt secrets at rest using advanced techniques such as envelope encryption and algorithms (e.g., AES-256), transmitting them securely over TLS. Additionally, they often provide features like bring your own keys (BYOK) or customer-managed encryption keys (CMEK), giving users full control over their encryption keys. To ensure their security, at no point are the secrets stored in clear text.
Using a managed secrets management solution reduces the burden on developers to handle encryption themselves, allowing them to focus more on development tasks.
Secrets Re-Encryption
Core practice: Proactively re-encrypt secrets to adapt to evolving encryption standards
A standard method for maintaining the security of secrets at rest is to re-encrypt them proactively when encryption standards change. Advanced secrets management solutions automate this re-encryption process for consistent protection without manual intervention. These solutions continuously monitor encryption standards and automatically initiate re-encryption whenever updates are required.
Core practice: Immediately re-encrypt secrets if the encryption keys are compromised
Another trigger for secrets re-encryption is the detection of key compromise. When a key compromise is identified, the security of all secrets encrypted with that key is at risk. Reactive re-encryption involves generating new, secure encryption keys and re-encrypting all affected secrets with the new keys. This minimizes the damage from a security breach by rendering compromised keys useless. This practice is especially important when you use BYOK or CMEK features.
Fine-Grained Access Control
Core practice: Limit secrets exposure through restricted access
Fine-grained role-based access control is a core security feature in secrets management that ensures only authorized identities can access specific secrets, minimizing the risk of unauthorized access. This approach enables organizations to manage and restrict secrets access based on precise criteria such as identities, roles, permissions, groups, teams, and environments.
At the center of this practice is a robust policy engine. Administrators can create access control list (ACL) policies to enforce the principle of least privilege. These policies specify exactly which identities can access particular secrets under certain conditions. Misconfigurations of role-based access control policies are common issues — often stemming from overly broad policies or reliance on default configurations — and can inadvertently expose secrets to unauthorized users.
Core Practices for Monitoring and Resiliency
This section explores the essential practices for monitoring and resilience in secrets management.
Access and Audit Logs
Core practice: Conduct regular secrets audits and review access logs
The real-time monitoring and logging of secrets access provides imperative visibility into who is accessing secrets and when, aiding in audit and compliance efforts. These access and audit logs create a detailed record of secrets access and usage, allowing organizations to detect and respond to unauthorized access. Knowing when changes, especially high-risk ones, are made and by whom is vital for maintaining a highly performant, secure environment.
Understanding what identity has accessed a specific secret, when they accessed it, and through which medium the secret was accessed is essential. Secrets access logs enable administrators to see detailed information about each access event, including the actor, access method, and access times. Importantly, these logs do not record the plaintext value of the secrets; instead, some solutions log the secret's value hash, such as a salted hash using a cryptographic function (e.g., HMAC-SHA256).
Advanced secrets management solutions also offer alerting on key operational conditions, which ensures that any suspicious activity is promptly identified and addressed.
Resilient Secrets
Core practice: Ensure secrets remain resilient in case of accidents or disasters
Emergencies and accidents can put organizations without a comprehensive secrets management resiliency plan at risk. Organizations can guarantee secrets resiliency either via offline backups or by implementing disaster recovery solutions. Creating offline backups of your secrets provides a safety net so that access can be restored if access to the secrets vault is lost. However, backups often introduce operational inefficiencies that may not always be desirable.
Disaster recovery options to avoid these operational challenges in practice:
- If backups are needed, create encrypted snapshots: Generate a read-only file containing an encrypted snapshot of current secrets. These backups should only be decrypted using the secrets management solution during restoration.
- Automate back-up processes: Set up automated back-up processes to simplify the task and reduce the risk of human error compared to manual backups.
- Implement disaster recovery: Use secrets management solutions that offer disaster recovery and high-availability capabilities. Enable secrets replication to separate instances, either on-premises or in the cloud, with automated syncing across instances to ensure continuity.
