PKI secrets engine with managed keys

11min
|
Enterprise
Vault

Challenge

While using Vault's PKI secrets engine to generate dynamic X.509 certificates, an organization may require their private keys to be created or stored within PKCS#11 hardware security modules (HSMs) to meet regulatory requirements.

Solution

Vault version 1.10 introduced the managed keys system to allow operators to configure access to a key stored in an external KMS. This feature also adds support to the PKI secrets engine for using a configured managed key as the backing private key for an intermediate or root certificate authority (CA).

Managed keys offload some PKI operations to the HSM or cloud KMS.

Key generation: Generate new PKI key pairs and certificates from external HSM or cloud KMS.
Certificate signing: Verify and sign some certificate workflows within external HSM or cloud KMS.

When Vault is configured with managed keys, all operations related to the private key, including generation, happen within the secure boundary of the HSM or cloud KMS external to Vault.

Prerequisites

To perform the tasks described in this tutorial, you need:

Vault Enterprise version 1.10 or later
HSM or AWS KMS environment

Note

PKCS#11 HSM-backed managed keys requires Vault Enterprise license.

Policy requirements

Note

For the purpose of this tutorial, you can use a root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following capabilities:

# Work with transform secrets engine
path "sys/managed-keys/*" {
  capabilities = [ "create", "read", "update", "list" ]
}

# Enable secrets engine
path "pki/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# List enabled secrets engine
path "sys/mounts" {
  capabilities = [ "read", "list" ]
}

# Tune mounts
path "sys/mounts/pki/tune" {
  capabilities = ["create", "update"]
}

If you are not familiar with policies, complete the policies tutorial.

In your organization, the configuration of a managed key and the configuration of a secrets engine may be performed by different people, in which case you might choose to segment the Vault policies accordingly.

Start Vault

For convenience and simplicity in the tutorial, you will operate a single Vault server with a minimal configuration using the default Shamir's Secret Sharing seal with one key share.

Open a terminal and create the initial Vault server configuration.

Note

Be sure to set the license_path to where your Vault license key is located.

$ cat > /tmp/vault-server.hcl << EOF
disable_mlock = true
license_path  = "/etc/vault.d/license.hclic"
api_addr      = "http://127.0.0.1:8200"
cluster_addr  = "http://127.0.0.1:8201"

storage "raft" {
   path    = "/tmp/vault-data"
   node_id = "node_1"
}

listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = true
} 
EOF

Create the Vault server data directory.
```
$ mkdir /tmp/vault-data
```
Start the Vault server.
```
$ vault server -config /tmp/vault-server.hcl
```
The Vault dev server defaults to running at 127.0.0.1:8200. It is uninitialized and not yet ready for use.
Open a new terminal session and export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Initialize the Vault server and capture the output to the file /tmp/.vault-init
```
$ vault operator init -key-shares=1 -key-threshold=1 > /tmp/.vault-init
```

Export the VAULT_UNSEAL_KEY environment variable with the unseal key from the file as its value.

$ export VAULT_UNSEAL_KEY="$(grep 'Unseal Key 1' /tmp/.vault-init \
    | awk '{print $NF}')"

Export the VAULT_ROOT_TOKEN environment variable with the initial root token from the file as its value.

$ export VAULT_TOKEN="$(grep 'Initial Root Token' /tmp/.vault-init \
    | awk '{print $NF}')"

Unseal Vault.

$ vault operator unseal $VAULT_UNSEAL_KEY

The Vault server is ready.

Configure a Managed Key

Choose either PKCS#11 HSM or AWS KMS, and follow the steps for your KMS choice.

This tutorial uses the SoftHSM2 software (refer to its documentation for setup instructions). The path to the SoftHSM library varies by platform. These instructions apply to an installation of Ubuntu Linux, but other platforms will also work.

Initialize a new PKCS#11 token.

$ softhsm2-util --init-token --free --so-pin=supersecret --pin=prettysecret \
   --label="test-kms-root"

SoftHSM example response:

Slot 24 has a free/uninitialized token. The token has been initialized and is
reassigned to slot 1601102058

Note

The slot number is returned. In this example, the slot number is 1601102058.

A PKCS#11 HSM requires a shared library to configure Vault to communication with the HSM. To declare this library, edit the Vault server configuration at /tmp/vault-server.hcl/ to add a kms_library stanza.

config-vault.hcl

disable_mlock = true
license_path  = "/etc/vault.d/license.hclic"
api_addr      = "http://127.0.0.1:8200"
cluster_addr  = "http://127.0.0.1:8201"
   
storage "raft" {
   path    = "/tmp/vault-data"
   node_id = "node_1"
}

listener "tcp" {
   address = "0.0.0.0:8200"
   tls_disable = true
}

kms_library "pkcs11" {
    name = "myhsm"
    library = "/usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so"
}

Restart or reload Vault's configuration with SIGHUP.
```
$ kill -HUP $(pidof vault)
```
Each managed key requires type-specific configuration. You must identify the location of a key and PIN to access the HSM. This is very similar to Vault's PKCS#11 auto-unseal mechanism. Configure the managed key.
```
$ vault write sys/managed-keys/pkcs11/learn-managed-key \
      library=myhsm slot=1601102058 pin=prettysecret \
      key_label=my-hsm-pki-key \
      allow_store_key=true \
      allow_generate_key=true \
      mechanism=0x0001 key_bits=4096 \
      any_mount=false
```
The PKCS#11 specific parameters are library, referring to the previously configured kms_library stanza, slot, pin, key_label, and mechanism, which identifies the object in the HSM which will hold the key, and that it's mechanism is CKM_RSA_PKCS (RSA with PKCS#1 v1.5 signatures), and that we will require a 4096 bit key. The allow_generate_key flag indicates that Vault is allowed to request that the HSM generate a key, and allow_store_key indicates that a new key may be stored in the backend. Without this flag, you may configure the key yourself in the HSM and just point Vault at the result. Finally, any_mount means any mount in the namespace may access the managed key. For this example, it is set to false so that you can demonstrate how to lock managed key access down to a specific mount.
Vault output:
```
Success! Data written to: sys/managed-keys/pkcs11/learn-managed-key
```

Read the key back.

$ vault read /sys/managed-keys/pkcs11/learn-managed-key

Key                  Value
---                  -----
UUID                 f1a88f60-3e10-924f-7071-f862250ca4e9
allow_generate_key   true
allow_replace_key    false
allow_store_key      true
any_mount            false
key_bits             4096
key_label            my-hsm-pki-key
library              myhsm
mechanism            1
name                 learn-managed-key
pin                  redacted
slot                 1042332116
type                 pkcs11

Test endpoint to validate that we can access the managed key.

$ vault write -force /sys/managed-keys/pkcs11/learn-managed-key/test/sign

Success! Data written to: sys/managed-keys/pkcs11/learn-managed-key/test/sign

The key has been generated in the HSM and used to sign and verify a dummy value.

AWS Requirements

To complete this tutorial, you must have an access key and secret key for an AWS account. The AWS account must have permissions to create AWS KMS keys and aliases. To set the configuration through AWS, set the environment variables AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY and AWS_REGION with the appropriate values or use AWS CLI. For instructions on how to use the AWS CLI, see the AWS documentation. You can also add these credentials to the managed key configuration itself.

Create the managed key in Vault.
If the AWS credentials have been set in environment variables accessible to the Vault server or via AWS command line tools, run the following command.
$ vault write sys/managed-keys/awskms/learn-managed-key-pki \ kms_key=alias/hashicorp/learn-managed-key-pki \ key_type=rsa \ key_bits=2048 \ allow_generate_key=true
To store the AWS credentials within the managed key configuration, use the optional access_key, secret_key, and region parameters.
$ vault write sys/managed-keys/awskms/learn-managed-key-pki \ kms_key=alias/hashicorp/learn-managed-key-pki \ key_type=rsa \ key_bits=2048 \ allow_generate_key=true \ access_key=<access_key> \ region=<region> \ secret_key=<secret_key>
For AWS KMS, managed keys require an identifier for the key (either an AWS alias or the key ID if using an existing key), the type of key (RSA in this example), and the size of the key (2048 bits in this example). The allow_generate_key parameter indicates that Vault should generate a new KMS key if one doesn't already exist with the provided alias.
Successful output:
```
Success! Data written to: sys/managed-keys/awskms/learn-managed-key-pki
```
This step only creates the managed key configuration in Vault. The AWS KMS key has not been created yet.

Read the key back.

$ vault read sys/managed-keys/awskms/learn-managed-key-pki

Vault returns the managed key configuration and you can verify your configuration.

Example output:

Key                 Value
---                 -----
UUID                3e57cc66-7551-8c3b-2f91-bd8b36a56da2
allow_generate_key  true
allow_replace_key   false
allow_store_key     false
any_mount           false
key_bits            2048
key_type            rsa
kms_key             alias/hashicorp/learn-managed-key-pki
name                learn-managed-key-pki
type                awskms

If you included the AWS credentials in the managed key configuration, you will also see keys access_key and secret_key with their values redacted.

Key                   Value
---                   -----
UUID                  da5c28cd-d98d-14b8-ab64-e21c020a7ffd
access_key            redacted
allow_generate_key    true
allow_replace_key     false
allow_store_key       false
any_mount             false
key_bits              2048
key_type              rsa
kms_key               alias/hashicorp/learn-managed-key-pki
name                  learn-managed-key-pki
region                us-east-1
secret_key            redacted
type                  awskms

Validate that we can access the managed key.
```
$ vault write -f sys/managed-keys/awskms/learn-managed-key-pki/test/sign
```
This may take a couple of seconds.
Vault output:
```
Success! Data written to: sys/managed-keys/awskms/learn-managed-key-pki/test/sign
```
Troubleshooting
If this operation fails, check that your AWS credentials are not expired and that the account has permissions for both AWS KMS and AWS aliases.
Vault will generate an AWS KMS key as part of this test. After this step, you should be able to log into the AWS console and find an AWS KMS key with the alias hashicorp/learn-managed-key-pki.

You may now move on to configuring the PKI secrets engine to use this key.

Configure PKI Secrets Engine

Enable the pki secrets engine at the pki path.

$ vault secrets enable pki
Success! Enabled the pki secrets engine at: pki/

Tune the secrets engine to use managed keys.
```
$ vault secrets tune -allowed-managed-keys=learn-managed-key pki
Success! Tuned the secrets engine at: pki/
```
You configured the managed key with any_mount=false in the previous step. This command grants access to the learn-managed-key to the PKI secrets engine's mount.
Generate the root certificate.
```
$ vault write -field=certificate pki/root/generate/kms \
    managed_key_name=learn-managed-key \
    common_name=root.myco.com \
    ttl=8760h
```
Note
Notice that the path is pki/root/generate/kms (instead of pki/root/generate/internal or pki/root/generate/exported). This indicates that a managed key will be used. The name of the managed key is specified (learn-managed-key).
Configure the CA and CRL URLs.
```
$ vault write pki/config/urls \
    issuing_certificates="$VAULT_ADDR/v1/pki/ca" \
    crl_distribution_points="$VAULT_ADDR/v1/pki/crl"
```
Successful output:
```
Success! Data written to: pki/config/urls
```
In a production environment, you may configure an additional mount or mounts for intermediate CAs, and have those intermediates signed by the root. Refer to the Build Your Own Certificate Authority (CA) tutorial.

Test that you can issue a certificate with your managed key-backed root by configuring and using a role.

$ vault write pki/roles/example-dot-com \
    allowed_domains="example.com" \
    allow_subdomains=true \
    max_ttl="720h"

Successful output:

Success! Data written to: pki/roles/example-dot-com

Request a new certificate for the test.example.com domain based on the example-dot-com role.

$ vault write -format=json pki/issue/example-dot-com \
common_name="test.example.com" ttl="24h"

Example output:

Key                 Value
---                 -----
certificate         -----BEGIN CERTIFICATE-----
MIIDwzCCAqugAwIBAgIUTQABMCAsXjG6ExFTX8201xKVH4IwDQYJKoZIhvcNAQEL
BQAwGjEYMBYGA1UEAxMPd3d3LmV4YW1wbGUuY29tMB4XDTE4MDcyNDIxMTMxOVoX
             ...

-----END CERTIFICATE-----
issuing_ca          -----BEGIN CERTIFICATE-----
MIIDQTCCAimgAwIBAgIUbMYp39mdj7dKX033ZjK18rx05x8wDQYJKoZIhvcNAQEL
             ...

-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAte1fqy2Ekj+EFqKV6N5QJlBgMo/U4IIxwLZI6a87yAC/rDhm
W58liadXrwjzRgWeqVOoCRr/B5JnRLbyIKBVp6MMFwZVkynEPzDmy0ynuomSfJkM
             ...

-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       4d:00:01:30:20:2c:5e:31:ba:13:11:53:5f:cd:b4:d7:12:95:1f:82

Next steps

Leveraging managed keys, operators can run a highly secure CA backed by an HSM or cloud KMS, while preserving the simplicity and flexibility of Vault's PKI secrets engine.

To learn more about the PKI secrets engine, see the following tutorials:

Help and reference

Management of Vault Enterprise with Terraform

Sentinel policies

This tutorial also appears in:

19 tutorials

Secrets management
Centrally store, access, and deploy secrets across applications, systems, and infrastructure.
- Vault