Externalizing Your Configurations With Vault for Scalable Deployments

Table of Contents:

• Introduction
  1. The Solution
• Setting Up Vault
  1. Creating API Admin Policy
  2. Creating Read-Only user policy
  3. Creating Token attached with API read-only policy
• 1. Linux Shell Integration
• 2. Java Integration
• 3. Python Integration
• 4. Nodejs Integration
• 5. Ansible Integration
• Conclusion

Introduction:

To implement automation for microservices or applications deployed to a large number of systems, it becomes essential to externalize the configurations to a secure, scalable, centralized configuration store. This is necessary to be able to deploy the application in multiple environments with environment-specific parameters without requiring human intervention and without requiring modification to the core application during automated deployments, scaling, and failover recoveries.

Besides the fact that manually managed configurations involve the risk of human error, they are also not scalable for 24x7 large-scale deployments, particularly when we are deploying several instances of microservices across various infrastructure platforms.

It is a standard security best practice to periodically rotate the application credentials to ensure housekeeping and to avoid misuse over time. Having credentials externalized ensures the reliability of the system and reduces human error/dependency. If the whole accountability of maintaining the credentials lies with few superheroes it is considered a threat to business continuity. There are various tools available in the security domain to handle the federated access to privileged credentials.

The Solution

In this article, we look at one of the solutions to this problem using the tool called Vault. Here are some of the reasons why we chose Vault:

• Vault is an open-source project with a large and actively contributing community. Enterprise support is available from HashiCorp with different licensing models to suit the organization’s SLA.
• There is a wide range of programming language libraries available for vault integration.
• Vault plays nicely with a large number of authentication backends. Many organizations primarily enable the LDAP backend to integrate to central Active Directory; however, Vault has the ability to federate various authentication methods.
• Vault can easily be set up into a High Availability configuration with two simple variables in config. In the backend, Vault data can be stored in various High Availability storage backends.
• There is a wide range of configuration data types that we can persist and consume easily in available secret engines of the Vault. In this example, we are going to look at the most simple: Key-Value storage.

In this article, I tried to consolidate some of the simplest methods to integrate with Vault with 5 primary programming languages/tools we mostly work with. If I have missed one of the mainstream languages, it is likely API library exists for the same and conceptually can be implemented in a similar fashion.

We will look at the following examples:

1. Shell: Simple shell script integration with environment variables from Vault; we also take a look at some Python-based scripts for some advanced use cases
2. Python: Flask API integration using the hvac Client library
3. Java: Spring boot application using the VaultConfig API to directly use Vault secrets in the application.properties
4. Node.js: Integration with Node.js using node-vault library
5. Ansible: Integration to Ansible to directly use variables from HashiCorp Vault (Not Ansible Vault) using the hashi_vault plugin.

Although a lot of documentation exists on this topic, the question keeps coming back to me, as our team found the examples and documentation a bit difficult to get started. So I decided to write these simple kickstarters to get us started with the least amount of steps.

The code for this article is hosted in GitHub. 

Ok then, let us get started with setting up Vault first.

Setting Up Vault

Running Vault locally is pretty simple. We can download the binary https://www.vaultproject.io/downloads and run vault in dev mode vault server -dev.

We can run Vault from the image available in docker hub as well: docker run -d -p 8200:8200 --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' vault

In the production environment, you would run Vault in server mode: https://learn.hashicorp.com/tutorials/vault/getting-started-deploy. The high availability backends and HA configurations for Vault are available at Vault-HA.

Once Vault is up and running, we can login to Vault with either the token provided above or obtain it from the console. We would enable the Key Value engine at location kv if it is not already enabled. For this use case, we enabled userpass authentication, although in enterprise environments LDAP authentication would be pretty standard. The user experience does not change. Vault will be able to provide a seamless API experience regardless of the authentication method. We only need to change from /auth/userpass to /auth/ldap in API call.

Now, to manage or read secrets from Vault, we would need to set up the policies to manage/access the secrets. You would want to set up this structure according to our project/applications hierarchy. In this example, we are going to create two simple policies. The Application admin policy can be assigned to Application admin who would be able to login to Vault and generate either a token capable of managing the configurations of the app himself, or a read-only token that allows only read-only capabilities to the configs.

Please note that in all the below examples we are using pretty standard variables which need to be provided according to the location of your vault during the launch of respective samples.

VAULT_HOST=localhost
VAULT_PORT=8200
VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=<<vault_token>>
VAULT_KEYS_PATH=<<Location of your config values. In our case it is kv/data/amitthk/vault-demo/dev>>

Creating API Admin Policy

To create the policy, we logged in to Vault UI with a root token and created the following policy:

vaultdemo_api_adm

This policy allows the user to, first of all, be able to list the contents in key value store (kv); and then, allows our safe location kv/amitthk/vault-demo/dev* to be editable.

path "kv/*" {
  capabilities = ["list"]
}path "kv/data/application*" {
  capabilities = ["create", "read", "update", "list"]
}path "kv/metadata/application*" {
  capabilities = ["create", "read", "update", "list"]
}path "kv/amitthk/vault-demo/dev*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}path "kv/data/amitthk/vault-demo/dev*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "kv/metadata/amitthk/vault-demo/dev*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Creating readonly User Policy

vaultdemo_api_readonly

path "kv/application*" {
  capabilities = ["read","list"]
}path "kv/data/application*" {
  capabilities = ["read","list"]
}path "kv/amitthk/vault-demo/dev*" {
  capabilities = ["read", "list"]
}path "kv/data/amitthk/vault-demo/dev*" {
  capabilities = ["read"]
}

We created a user named vaultdemoadm under userpass authentication method:

vault write auth/userpass/users/vaultdemoadm password=password123 policies=default,vaultdemo_api_adm,vaultdemo_api_readonly

More details: https://www.vaultproject.io/docs/concepts/policies

Our user vaultdemoadm will then be able to login to the Vault via UI and save the key value pairs as in the below image.

Users can also login from CLI or API interface and update the above values.

curl --request POST --data '{"password":"password123", "ttl": "1h"}' http://localhost:8200/v1/auth/userpass/login/vaultdemoadm

This gives us a response as below. Here, our token is client_token:

{"request_id":"a4be2704-5079-e719-7c07-51ea105250a2","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.HvmpPVjnGEK1KpSJPZDhhEfi","accessor":"kodwoxRXSgzVTnhKv2nLfdNT","policies":["default","vaultdemo_api_adm","vaultdemo_api_readonly"],"token_policies":["default","vaultdemo_api_adm","vaultdemo_api_readonly"],"metadata":{"username":"vaultdemoadm"},"lease_duration":2764800,"renewable":true,"entity_id":"41df50e9-98e1-d252-b630-10ae12952397","token_type":"service","orphan":true}}

Creating Token Attached With API readonly Policy

To create the token, we create the file payload.json as below:

{
  "policies": ["vaultdemo_api_readonly"],
  "ttl": "1h",
  "renewable": false
}

The following command will be used to generate the token. (Remember to replace <<api_admin_user_token>> below with token from your login above.)

curl --header "X-Vault-Token: <<api_admin_user_token>>" --request POST --data @payload.json http://localhost:8200/v1/auth/token/create

We can also use the above token to generate an API admin token for a longer lease time.

{
   "policies": ["vaultdemo_api_adm", "default", "vaultdemo_api_readonly"],
   "ttl": "1h",
   "renewable": false
}

Now let us use the config values stored in Vault above with our apps. We will be using our readonly token to read these values in the below examples.

1. Linux Shell Integration: Environment Variables From Vault

The most basic way of interacting with Vault here is to interact with the API directly with cURL and then use jq to parse the response. The following code snippet from /shell/vault_draw_kv.sh in the project directory does the same.

Note: You either need to install jq yum install -y jq or use Python and pip install requests.

#!/bin/bashif [[ $# -eq 0 ]] ; then
    echo 'usage ./vault_draw_kv.sh <<VAULT_ADDR>> <<VAULT_TOKEN>>  <<VAULT_KEYS_PATH>>'
    exit 1
fiVAULT_ADDR=$1
VAULT_TOKEN=$2
VAULT_KEYS_PATH=$3rm -f .envuser=$(curl  -H "X-Vault-Token: $VAULT_TOKEN" \
        -X GET $VAULT_ADDR/v1/$VAULT_KEYS_PATH)echo DB_ENDPOINT=$(echo $user | jq -r .data.data.dbendpoint) > .env
echo DB_USER=$(echo $user | jq -r .data.data.dbuser) >> .env
echo DB_PASSWORD=$(echo $user | jq -r .data.data.dbpass) >> .env

Using Simple Python Script To Fetch Config From Vault

Take a look at following code snippet from /python/vault_withdraw_secrets.py file in project directory:

response = requests.get(request_url,headers=headers)
while retry_count >= 0:
    time.sleep(3) # wait 3 seconds then try again
    try:
        #print('response: '+str(response.json()))
        foutput = ''
        for field_name in response.json()['data']['data']:
            value = str(response.json()['data']['data'][field_name])
            if should_decode is True:
                value = value.decode('base64')
            foutput = foutput + field_name + "=" + value + "\n"            write_to_file(file_name=VALUE_FILE,f_output=foutput)

For another implementation that we basically use with our LDAP authentication (for the api_readonly user), please check https://github.com/amitthk/vault-intgs/blob/master/python/vault_withdraw_secrets_ldap-auth.py in the project directory.

2. Java Spring Boot App Integration: application.properties Environment Variables From Vault

In the second scenario, we are going to take a look with a Java application. To integrate Vault to our Java application, first let us add following dependencies to our pom/gradle:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-bootstrap</artifactId>
    <version>3.0.4</version>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-vault-config</artifactId>
    <version>3.0.4</version>
</dependency>

Bootstrap is needed to integrate the Vault values during bootstrap of our app using following bootstrap file:

spring:
  cloud:
    vault:
      host: ${VAULT_HOST}
      port: ${VAULT_PORT}
      scheme: http
      token: ${VAULT_TOKEN}
      kv:
        enabled: true
        backend: kv
      application-name: amitthk/vault-demo/dev

That is all. Now we can use our variables directly in our application.properties as below:

server.port=8089app.db.user=${dbuser}
app.db.pass=${dbpass}

3. Python: Flask App Config From Vault

For Python, we have some of the scripts above which interact with API interface. For our Flask API in the example directory https://github.com/amitthk/vault-intgs/blob/master/python/webapp.py, we use hvac client library (remember to pip install hvac ). The below code does the integration:

try:
    client = hvac.Client(url=app.config['VAULT_ADDR'])
    client.token=token=app.config['VAULT_TOKEN']
    keys_path=app.config['VAULT_KEYS_PATH']
    keys_data=client.read(keys_path)
    for appkey in keys_data['data']['data']:
        app.config[appkey]=keys_data['data']['data'][appkey]
except Exception as exc:
    print("Couldn't fetch config from vault: " + str(exc))
    raise exc

4. Node.JS: Environment Variables From Vault

For Node.js integration, we used the node-vault library npm install --save node-vault . The code in example directory https://github.com/amitthk/vault-intgs/blob/master/nodejs/index.js is a simple HTTP web server which uses following code to fetch config from Vault:

const vaultAddr = process.env.VAULT_ADDR;
const vaultToken = process.env.VAULT_TOKEN;
const vaultKeysPath = process.env.VAULT_KEYS_PATH;
const vault = require("node-vault")({
  apiVersion: "v1",
  endpoint: vaultAddr,
});vault.token = vaultToken
const { data } = await vault.read(vaultKeysPath);
let respStr = '<html><head></head><body><ul>'
for(var key in data['data']){
respStr = respStr + '<li>' + key + ': ' + data['data'][key] + '</li>';
}

5. Ansible Automation: hashi_vault Plugin

One of the important use cases for our automation is Ansible, which we pretty heavily use across all Linux based systems. Ansible comes with lookup plugin which can directly integrate with vault. In our example, we are setting variables from vault as below:

vars:
  vault_keys_data: "{{ lookup('hashi_vault', 'url=\"{{VAULT_ADDR}}\" token=\"{{VAULT_TOKEN}}\" secret=\"{{VAULT_KEYS_PATH}}\"')}}"
tasks:
  - set_fact:
      db_username: "{{vault_keys_data.dbuser}}"
      db_password: "{{vault_keys_data.dbpass}}"
      db_endpoint: "{{vault_keys_data.dbendpoint}}"

Conclusion:

In this article, we understood why we should externalize our configurations to make our automated deployments more consistent, scalable, and reliable. We looked at how we can use HashiCorp Vault to solve this problem and various integration options available with HashiCorp Vault.