How to Protect Azure Key Vault Resources

Azure Key Vault is an excellent solution for storing secrets, for example, simple passwords or certificates, and allowing applications to access them securely. However, this means that Key Vault data becomes critical for your application, and you need to make sure that it is protected and readily available. Key Vault already includes some protections, including version history for secrets and geo-redundancy for disaster recovery. However, these do not protect you against accidental deletion of secrets or the entire vault. In this article, we'll look at ways you can protect your Key Vault from accidental deletion.

Resource Locks

A quick and easy protection for your Key Vault is to enable a resource lock on the vault. The resource lock prevents anyone from deleting the vault itself without having rights to remove the lock. It will not, however, stop you from deleting the contents of the vault.

If you're not familiar with resource locks, take a look at this article on using these to protect any resource.

Soft Delete

Key Vault has an option to enable soft delete. When enabled, this means that if either an object is deleted from the vault or the entire vault is deleted, then, the item is retained for 90 days. During this 90 day period, you have the option to restore the deleted item.

Soft Delete is not enabled on vaults by default; you need to enable it. You can do this either at deploy time or later. There is no UI in the portal for soft delete, so it needs to be configured either through ARM template or PowerShell/CLI.

Enable Soft Delete

Enabling soft delete at creation time with PowerShell is done with an extra "EnableSoftDelete" flag:

New-AzureRmKeyVault -VaultName "keyvault1" -ResourceGroupName "keyvaultRG" -Location "westeurope" -EnableSoftDelete

Enabling soft delete on an existing Key Vault is a bit more complicated and requires you to directly manipulate the ARM net property:

($resource = Get-AzureRmResource -ResourceId (Get-AzureRmKeyVault -VaultName "keyvault1").ResourceId).Properties | Add-Member -MemberType "NoteProperty" -Name "enableSoftDelete" -Value "true"

Set-AzureRmResource -resourceid $resource.ResourceId -Properties $resource.Properties

You can then use the get-azurermKeyVault command to get the Key Vault and check that soft delete is enabled.

Restore Deleted Vault

If we need to restore a whole vault that has been deleted the first thing to do is check that is still available in a soft-deleted state by running the get-AzureRMKeyVault command with the InRemovedStateVault flag:

Get-AzureRmKeyVault -InRemovedStateVault 

Then, we can use the Undo-AzureRmKeyVaultRemoval command to restore the vault:

Undo-AzureRmKeyVaultRemoval -VaultName "keyvault1" -ResourceGroupName "keyvaultRG" -Location westeurope

Restore the Deleted Item

If we need to restore individual items, we can again use the InRemovedState flag to query for the removed item. Depending on the type of item we want to recover (secret, key, certificate etc.), we would use the appropriate get command with this flag. So, to look for a secret, we would use:

Get-AzureKeyVaultSecret -VaultName "keyvault1" -InRemovedState

Once verified, we could recover the item, again, using the appropriate command for the type of object. For example, for a secret, we would use the following:

Undo-AzureKeyVaultSecretRemoval -VaultName "keyvault1" -Name "secret1"

Backup Key Vault

Soft deletes protect you against accidental deletion for up to 90 days, but if you need to be able to go back longer than 90 days, or you need the option to store your data outside of Key Vault, then we need to look at a backup process.

There is no regular backup process built into Key Vault that you can leverage. However, there are some backup PowerShell commands (one for each item type) that can be used to backup your Key Vault objects to file (1 file per object), which you can store where you like and use to restore later. There is not one command to back up the entire vault; these commands are designed to backup individual items. Therefore, we need to put together a script to look at backing up the whole vault. We could, then, look at running this on a regular schedule, using something like Azure Automation.

One important caveat to bear in mind with Key Vault backup is that you can only restore this data to a Key Vault in the same subscription as the one you backed up.

Backup Script

To facilitate an easy backup, we are going to create a script that does the following:

• Connects to Key Vault
• Backs up each item to a file in a temporary location
• Upload the files to Azure Files

You'll note that we are using Azure files as our backup location. The reason for using this rather than blob storage is because Azure Files have a backup built in. By using Azure Files, we can upload the files, overwriting the existing files, and then use the Azure Files' backup to back up the folder and keep the Key Vault backup history in the files backed up. By doing this, we avoid having to keep multiple copies of the backup files for version history and can store them securely in a backup vault

The script is relatively simple. Its present form assumes that it is being run using an account that has already logged into Azure (using login-azurermaccount) and selected the appropriate subscription. I will shortly update the script to work with Azure Automation credentials.

The script expects to receive parameters detailing the Key Vault to backup, the File Share to backup to, and a local folder to store the downloaded backup files temporarily.

[CmdletBinding()]
Param(

    [Parameter(Mandatory = $True)]
    [string]$keyvaultName,

    [Parameter(Mandatory = $True)]
    [string]$keyVaultResourceGroup,


    [Parameter(Mandatory = $True)]
    [string]$storageAccountName,


    [Parameter(Mandatory = $True)]
    [string]$storageResourceGroup,


    [Parameter(Mandatory = $True)]
    [string]$fileshareName,


    [string]$backupFolder = "$env:Temp\KeyVaultBackup"  
)

Then, we go ahead and connect to the KeyVault and loop through all the items in Key Vault. Next, we back them up to a local file, prefixed with the type of resource we are backing up. We back up secrets, keys, certificates, and managed storage account keys.

#######Setup backup directory


If ((test-path $backupFolder)) {
    Remove-Item $backupFolder -Recurse -Force

}

####### Backup items

New-Item -ItemType Directory -Force -Path $backupFolder | Out-Null

Write-Output "Starting backup of KeyVault to a local directory."
###Certificates

$certificates = Get-AzureKeyVaultCertificate -VaultName $keyvaultName 

foreach ($cert in $certificates) {
    Backup-AzureKeyVaultCertificate -Name $cert.name -VaultName $keyvaultName -OutputFile "$backupFolder\certificate-$($cert.name)" | Out-Null
}

###Secrets
$secrets = Get-AzureKeyVaultSecret -VaultName $keyvaultName

foreach ($secret in $secrets) {
    #Exclude any secerets automatically generated when creating a cert, as these cannot be backed up   
    if (! ($certificates.Name -contains $secret.name)) {
        Backup-AzureKeyVaultSecret -Name $secret.name -VaultName $keyvaultName -OutputFile "$backupFolder\secret-$($secret.name)" | Out-Null
    }
}

#keys
$keys = Get-AzureKeyVaultKey -VaultName $keyvaultName
foreach ($kvkey in $keys) {
    #Exclude any keys automatically generated when creating a cert, as these cannot be backed up   
    if (! ($certificates.Name -contains $kvkey.name)) {
        Backup-AzureKeyVaultKey -Name $kvkey.name -VaultName $keyvaultName -OutputFile "$backupFolder\key-$($kvkey.name)" | Out-Null
    }
}

Finally, we upload these files to the Azure Files share, overwriting any files that already exist and, then, clean up the local temp files.

Write-Output "Local file backup complete"    
####### Copy files to Azure Files
Write-Output "Starting upload of backup to Azure Files"
$storageAccount = Get-AzureRmStorageAccount -ResourceGroupName $storageResourceGroup -Name $storageAccountName 
$files = Get-ChildItem $backupFolder
$backupFolderName = Split-Path $backupFolder -Leaf

#Create backup folder if it does not exist
$backupFolderTest = Get-AzureStorageFile -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
if (! $backupFolderTest) {
    New-AzureStorageDirectory -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
}
#upload files, overwriting existing
foreach ($file in $files) {
    Set-AzureStorageFileContent -Context $storageAccount.Context -ShareName $fileshareName -Source $file.FullName -Path "$backupFolderName\$($file.name)" -Force

}

Remove-Item $backupFolder -Recurse -Force

Write-Output "Upload complete."
Write-Output "Backup Complete"

You can find the full script for this on GitHub.

Restore Script

We also need to be able to restore the files that we have backed up. This script assumes that the correct version of the backup files is in the folder in the Azure File share. If you want to restore older versions of the files, you can use the Azure Files restore process to restore these files to a folder, then point this script at this folder.

To restore, we are doing the opposite of the backup script, downloading the files locally from the Azure file share and then publishing them to KV. There are a couple of caveats with this restore script to be aware of:

1. It assumes that the  KeyVault  that you are restoring to is either empty or, at the very least, does not contain the secrets that you want to restore. The restore-Key Vault commands don't seem to support overwriting. If the secrets exist, you should either delete them or change the names of the ones that you want to restore.
2. At the moment, the restore script doesn't support restoring managed storage keys. There seems to be an issue with the command to do this that I need to work out.

In the restore script, the parameters are, again, going to expect details of the KV and Storage, along with a local temp location.

[CmdletBinding()]
Param(

    [Parameter(Mandatory = $True)]
    [string]$keyvaultName,

    [Parameter(Mandatory = $True)]
    [string]$keyVaultResourceGroup,


    [Parameter(Mandatory = $True)]
    [string]$storageAccountName,


    [Parameter(Mandatory = $True)]
    [string]$storageResourceGroup,


    [Parameter(Mandatory = $True)]
    [string]$fileshareName,


    [Parameter(Mandatory = $True)]
    [string]$backupFolder,

    [string]$tempRestoreFolder = "$env:Temp\KeyVaultRestore"

)

Then, we loop through all the files in the Azure files folder and download them locally:

#Create a temporary folder to download files
If ((test-path $tempRestoreFolder)) {
    Remove-Item $tempRestoreFolder -Recurse -Force

}
New-Item -ItemType Directory -Force -Path $tempRestoreFolder | Out-Null

Write-Output "Starting download of backup to Azure Files"
$storageAccount = Get-AzureRmStorageAccount -ResourceGroupName $storageResourceGroup -Name $storageAccountName 


#Download files from Azure File Share
$backupFolderTest = Get-AzureStorageFile -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
if (! $backupFolderTest) {
    Write-Error "Backup folder in Azure File Share Not Found."
    exit
}

$backupFiles = Get-AzureStorageFile -ShareName $fileshareName -Path $backupFolder  -Context $storageAccount.Context | Get-AzureStoragefile

foreach ($backupFile in $backupFiles ) {
    Write-Output "downloading $backupFolder\$($backupFile.name)"
    Get-AzureStorageFileContent -ShareName $fileshareName -Path "$backupFolder\$($backupFile.name)" -Destination "$tempRestoreFolder\$($backupFile.name)"  -Context $storageAccount.Context 
}

We need to determine the type of KV item for each file, which is done based on the first word in the file name. So, we use the  match command in PowerShell to retrieve this:

$secrets = get-childitem $tempRestoreFolder | where-object {$_ -match "^(secret-)"}
$certificates = get-childitem $tempRestoreFolder | where-object {$_ -match "^(certificate-)"}
$keys = get-childitem $tempRestoreFolder | where-object {$_ -match "^(key-)"}

Then, we go ahead and restore the secrets into KV and clean up the temporary files.

foreach ($secret in $secrets) {
    write-output "restoring $($secret.FullName)"
    Restore-AzureKeyVaultSecret -VaultName $keyvaultName -InputFile $secret.FullName 
}

foreach ($certificate in $certificates) {
    write-output "restoring $($certificate.FullName) "
    Restore-AzureKeyVaultCertificate -VaultName $keyvaultName -InputFile $certificate.FullName 
}

foreach ($key in $keys) {
    write-output "restoring $($key.FullName)  "
    Restore-AzureKeyVaultKey -VaultName $keyvaultName -InputFile $key.FullName 
}

Remove-Item $tempRestoreFolder -Recurse -Force

The full script can be found on GitHub.