How To Handle Secrets in Python

We live in a world where applications are used to do everything, be it stock trading or booking a salon, but behind the scenes, the connectivity is done using secrets. Secrets, such as database passwords, API keys, tokens, etc., must be managed appropriately to avoid any breach.

The need for managing secrets is critical for any organization. Secrets can be leaked in many ways, including through version control systems (never hardcode any secrets in your code), private messages, email, and other communication channels. If secrets are leaked, it can lead to a loss of trust, credibility, and even business. In some cases, leaked secrets can also lead to legal action. That’s why it’s so important to have a plan for managing secrets.

In this article, we will discuss four different ways to manage secrets in Python efficiently.

Prerequisites

Before getting started, below are a few things to keep in mind to avoid any issues later:

• Python and pip installed on your machine.
• An understanding of Python and CLI.
• Python IDE, such as PyCharm or VS Code.
• A basic understanding of the cloud.

We will be using MacOS for this device. Please use the commands as per your OS.

Four Ways To Manage Secrets in Python

In this section, we will discuss four different ways to manage your secrets in Python:

1. From a File

Using a .env File

The .env file is a file used to store environment variables in Python. Environment variables are variables set outside of the Python code and are used to configure the Python code. The .env file is typically used to store secret keys and passwords.

We will use the python-dotenv package for accessing the content of the .env file. To get started, first install the package using the following command:

$ pip install python-dotenv

Create a .env file for testing purposes and paste the following secrets:

API_KEY=test-key
API_SECRET=test-secret

Of course, this file should not be committed to your git repo. Otherwise, it would be versioned and readable even after you delete it.

Add this line to your .gitignore file:

.env

Once done, create a main.py file and paste the below-mentioned code snippet. In this code, we are using the load_dotenv() function to load content from the .env file:

from dotenv import load_dotenv
import os

load_dotenv()

api_key = os.getenv("API_KEY")
api_secret = os.getenv("API_SECRET")

print("API_KEY: ", api_key)
print("API_SECRET: ", api_secret)

We can also use the dotenv_values() function, which converts the secrets into a dictionary. Secrets can be accessed by using the following code snippet:

from dotenv import dotenv_values

secrets = dotenv_values(".env")

def main():
   print(secrets["API_KEY"])
   print(secrets["API_SECRET"])

if __name__ == "__main__":
   main()

When working on a large project, you may find that you need multiple .env files. For example, you may have a .env file for your local development environment and a .env.dev file for your cloud development production environment. The following code snippet can be helpful if you have multiple .env files:

from dotenv import dotenv_values

secrets = dotenv_values(".env")
local_secrets = dotenv_values(".env.dev")

def main():
   print(secrets["API_KEY"])
   print(local_secrets["API_SECRET"])

if __name__ == "__main__":
   main()

Using a JSON File

To keep the secrets more organized, you can also use a JSON file. Let’s create a secrets.json file and paste the following secrets into it:

{
   "db": {
       "host": "localhost",
       "port": 27017,
       "name": "test"
   },
   "server": {
       "host": "localhost",
       "port": 3000
   }
}

The same as above applies, do not commit this file.

Now that we have the JSON file ready, let’s write a function to access secrets from the file:

# main.py
import json

def get_value_from_json(json_file, key, sub_key):
   try:
       with open(json_file) as f:
           data = json.load(f)
           return data[key][sub_key]
   except Exception as e:
       print("Error: ", e)
       
print(get_value_from_json("secrets.json", "db", "host")) # prints localhost

2. Using Environment Variables

Environment variables are variables that are set by the operating system or a specific user and are used by programs to determine various settings. We can use these variables to store our secrets and then access them in our program. You can use the following syntax to create an environment variable in macOS or a Linux machine:

$ export variable_name=value
$ export API_KEY_TEST=dummykey

On a Windows machine, you use GUI to add environment variables or use the following command to add a variable:

$ setx [variable_name] “[value]”

You can use the OS package to access the OS environment variable. Mentioned below is the sample code:

import os

# Get the secret key from the environment
secret_key = os.environ.get('api_key_test')
print(secret_key) // prints dummykey

Secrets at the command line should be handled with special care too.

3. Use a Cloud Secrets Manager

Most cloud service providers offer an inbuilt secrets manager that can be used to create and use secrets across cloud infrastructure. Following the secret managers offered by cloud providers:

• AWS Secrets Manager
• Secret Manager (By Google Cloud)
• Azure Key Vault

AWS Secrets Manager is widely used across the industry. Let’s write a function to create and access a secret in AWS using Boto3:

import boto3

def fetch_secret_from_aws(secret_name):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       get_secret_value_response = client.get_secret_value(SecretId=secret_name)
       return get_secret_value_response['SecretString']
   except Exception as e:
       print(e)
       return None
       
def create_secret_in_aws(secret_name, secret_value):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       client.create_secret(Name=secret_name, SecretString=secret_value)
       return True
   except Exception as e:
       print(e)
       return False

4. Using a KMS

A KMS is a key management system used to manage cryptographic keys. It is typically used in organizations to centrally manage and secure keys. A KMS can be used to generate, store, and distribute keys. It can also be used to revoke keys and monitor key usage.

KMS is a convenient way to centrally manage the keys used by your applications and services and helps ensure that only authorized users have access to them.

Hashicorp Vault is one of the best open-source KMS available in the market that offers many features and benefits, including the ability to manage secrets and keys across multiple environments, strong security controls, and good scalability.

Let’s write a function to read and write secrets to a specific path in the vault.

Note: Please ensure you have hvac (Python client for Vault) installed and have a Hashicorp Vault setup:

import hvac

def read_secret_from_vault(secret_path, token, secret_name):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       read_response = client.secrets.kv.read_secret_version(path=secret_path)
       return read_response['data']['data'][secret_name]
   except Exception as e:
       print(e)
       return None

def write_secret_to_vault(secret_path, token, secret_name, secret_value):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       create_response = client.secrets.kv.v2.create_or_update_secret(
           path=secret_path,
           secret={secret_name: secret_value},
       )
       return create_response
   except Exception as e:
       print(e)
       return None

Wrapping Up

Managing secrets is an essential part of application development. When developers hardcode secrets in cleartext into their applications, it creates a potential security breach. An attacker can use those secrets to access sensitive data if those secrets are discovered.

Another alternative to the proposed methods here is to check secrets into source code and share them with your team encrypted. This can be a very flexible solution if you learn how to leverage a tool such as Mozilla SOPS.