Cubbyhole response wrapping

14min
|
Vault
Interactive

The term cubbyhole comes from an Americanism where you get a "locker" or "safe place" to store your belongings or valuables. In Vault, the cubbyhole is your "locker". All secrets are namespaced under your token. If that token expires or is revoked, all the secrets in its cubbyhole are revoked as well.

It is not possible to reach into another token's cubbyhole even as the root user. This is an important difference between the cubbyhole and the key/value secrets engine. The secrets in the key/value secrets engine are accessible to any token for as long as its policy allows it.

Personas

The end-to-end scenario described in this tutorial involves two personas:

admin with privileged permissions to create tokens
apps trusted entity retrieving secrets from Vault

Challenge

In order to tightly manage the secrets, you set the scope of who can do what using the Vault policy and attach that to tokens, roles, entities, etc.

Think of a case where you have a trusted entity (Chef, Jenkins, etc.) which reads secrets from Vault. This trusted entity must obtain a token. If the trusted entity or its host machine was rebooted, it must re-authenticate with Vault using a valid token.

How can you securely distribute the initial token to the trusted entity?

Solution

Use Vault's cubbyhole response wrapping where the initial token is stored in the cubbyhole secrets engine. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won't see the original value. The wrapping token is short-lived and can be revoked just like any other tokens so that the risk of unauthorized access can be minimized.

What is cubbyhole response wrapping?

When response wrapping is requested, Vault creates a temporary single-use token (wrapping token) and insert the response into the token's cubbyhole with a short TTL
Only the expecting client who has the wrapping token can unwrap this secret
Any Vault response can be distributed using the response wrapping

Benefits of using the response wrapping:

It provides cover by ensuring that the value being transmitted across the wire is not the actual secret. It's a reference to the secret.
It provides malfeasance detection by ensuring that only a single party can ever unwrap the token and see what's inside
It limits the lifetime of the secret exposure
- The TTL of the response-wrapping token is separate from the wrapped secret's lease TTL

Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Policy requirements

Note

For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Manage tokens
path "auth/token/*" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage secret/dev secrets engine - for Verification test
path "secret/dev" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

If you are not familiar with policies, complete the policies tutorial.

Lab Setup

Open a terminal and start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.

Insecure operation

Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

Note

For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

Enable KV v2 secrets engine

Open a web browser and launch the Vault UI and then login using the admin token.
Click Enable new engine.
Select KV from the list, and then click Next.
Enter secret in the Path field. Under the Version, be sure to select 2.
Click Enable Engine to complete.

Scenario Introduction

An app needs to retrieve secrets from Vault at the secret/kv path. However, the app does not have a valid client token to read secrets at secret/kv. Vault admin wraps the secret values using the cubbyhole response wrapping, and sends the wrapping token to the app. The app unwraps the secrets before the wrapping token expires.

Note

This tutorial demonstrates how the response wrapping works. To learn more about K/V secrets engine, refer to the Versioned Key/Value Secrets Engine tutorial.

Step 1: Create and wrap a token

(Persona: admin)

When a newly created token is wrapped, Vault inserts the generated token into the cubbyhole of a single-use token, returning that single-use wrapping token. Retrieving the secret requires an unwrap operation against this wrapping token.

Write some test data at secret/dev.

$ vault kv put secret/dev username="webapp" password="my-long-password"

Example output:

= Secret Path =
secret/data/dev

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-30T18:50:59.153742Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Read the secrets at secret/dev and wrap the output using the -wrap-ttl flag to specify that the response should be wrapped and the life of the wrapping token to be 120 seconds (2 minutes).

$ vault kv get -wrap-ttl=120 secret/dev

Example output:

Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA
wrapping_accessor:               3dyFk8GHlmLNvKEjxcL9TDz2
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2022-04-05 21:09:08.2289 -0700 PDT
wrapping_token_creation_path:    secret/data/dev

Instead of displaying the secrets at secret/dev, the response includes the wrapping token.

Store the returned wrapping token value in a WRAPPING_TOKEN environment variable.

Example:

$ export WRAPPING_TOKEN="hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA"

Create an API request payload containing some test data to write at secret/dev.

$ tee payload.json <<EOF
{
  "data": {
    "username": "webapp",
    "password": "my-long-password"
  }
}
EOF

Create some test data at secret/dev.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/data/dev | jq ".data"

Example output:

{
  "created_time": "2022-04-06T04:24:24.391513Z",
  "custom_metadata": null,
  "deletion_time": "",
  "destroyed": false,
  "version": 1
}

Read the secrets at secret/dev and wrap the output using the -wrap-ttl flag to specify that the response should be wrapped and the life of the wrapping token to be 120 seconds (2 minutes).

$ curl --header "X-Vault-Wrap-TTL: 120" \
   --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/secret/data/dev | jq ".wrap_info"

Example output:

{
  "token": "hvs.CAESIBGbPD-i2IQAGIQMuWTAR_fKldnCC2bGOrTzQwVebsgmGh4KHGh2cy5jQlZpa29TbFBKb3lFNTd6QUk1YzFvQkM",
  "accessor": "dHakTCTkPHc7WHpPs5nCAYD3",
  "ttl": 120,
  "creation_time": "2022-04-05T21:25:46.539073-07:00",
  "creation_path": "secret/data/dev"
}

Instead of displaying the secrets at secret/dev, the response includes the wrapping token.

Store the returned wrapping token value in a WRAPPING_TOKEN environment variable.

Example:

$ export WRAPPING_TOKEN="hvs.CAESIBGbPD-i2IQAGIQMuWTAR_fKldnCC2bGOrTzQwVebsgmGh4KHGh2cy5jQlZpa29TbFBKb3lFNTd6QUk1YzFvQkM"

Create an API request payload containing some test data to write at secret/dev.

$ tee payload.json <<EOF
{
  "data": {
    "username": "webapp",
    "password": "my-long-password"
  }
}
EOF

Create some test data at secret/dev.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/data/dev | jq ".data"

Example output:

{
  "created_time": "2022-04-06T04:24:24.391513Z",
  "custom_metadata": null,
  "deletion_time": "",
  "destroyed": false,
  "version": 1
}

Read the secrets at secret/dev and wrap the output using the -wrap-ttl flag to specify that the response should be wrapped and the life of the wrapping token to be 120 seconds (2 minutes).

$ curl --header "X-Vault-Wrap-TTL: 120" \
   --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/dev | jq ".wrap_info"

Example output:

{
  "token": "s.pdZqqfvowYARIRVmGfIUGe51.GG0sH",
  "accessor": "p7xK5d1W6NLGgfKVf2NJB9LW.GG0sH",
  "ttl": 120,
  "creation_time": "2022-04-06T06:08:09.703238218Z",
  "creation_path": "secret/data/dev"
}

Instead of displaying the secrets at secret/dev, the response includes the wrapping token.

Store the returned wrapping token value in a WRAPPING_TOKEN environment variable.
Example:
```
$ export WRAPPING_TOKEN="s.pdZqqfvowYARIRVmGfIUGe51.GG0sH"
```

Under Secrets, select secret/ and then click Create secret. Enter dev in the Path for this secret field.
Under Secret data, enter username in the key field, and webapp in its value field.
Click Add.
Enter password in the second key field, and my-long-password as its value.
Click Save.
Select Copy > Wrap secret.
Click the copy icon to copy the wrapping token value. NOTE: You can click on the sensitive information toggle to show the wrapping token value.
Save the wrapping token to use later.

Step 2: Unwrap the secret

(Persona: apps)

The apps persona receives a wrapping token from the admin. In order for the apps to acquire a valid token to read secrets from secret/dev path, it must run the unwrap operation using this token.

Note

If a client has been expecting delivery of a response-wrapping token and none arrives, this may be due to an attacker intercepting the token and then preventing it from traveling further. This should cause an alert to trigger an immediate investigation.

Unwrap the secret by passing the wrapping token.

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

To unwrap the secrets using valid wrapping token, the app does not have to have a client token.

Example output:

Key         Value
---         -----
data        map[password:my-long-password username:webapp]
metadata    map[created_time:2022-04-06T04:24:24.391513Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Notice that the data displays the password and username stored at secret/dev.

Try unwrap the secrets again.

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

This returns an error, "wrapping token is not valid or does not exist" because wrapping tokens are limited to single-use.

Unwrap the secret by passing the wrapping token.

$ curl --header "X-Vault-Token: $WRAPPING_TOKEN" \
   --request POST \
   $VAULT_ADDR/v1/sys/wrapping/unwrap | jq ".data.data"

Note

Notice that you can pass the wrapping token in the request header (X-Vault-Token) to unwrap the secrets. In order to unwrap secrets, the client does not authenticate with Vault.

Output:

{
  "password": "my-long-password",
  "username": "webapp"
}

Try unwrap the secrets again.

$ curl --header "X-Vault-Token: $WRAPPING_TOKEN" \
   --request POST \
   $VAULT_ADDR/v1/sys/wrapping/unwrap | jq

This returns an error, "wrapping token is not valid or does not exist" because wrapping tokens are limited to single-use.

$ curl --header "X-Vault-Token: $WRAPPING_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   $VAULT_ADDR/v1/sys/wrapping/unwrap | jq ".data.data"

Note

Notice that you can pass the wrapping token in the request header (X-Vault-Token) to unwrap the secrets. In order to unwrap secrets, the client does not authenticate with Vault.

Output:

{
  "password": "my-long-password",
  "username": "webapp"
}

Try unwrap the secrets again.

$ curl --header "X-Vault-Token: $WRAPPING_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   $VAULT_ADDR/v1/sys/wrapping/unwrap | jq

This returns an error, "wrapping token is not valid or does not exist" because wrapping tokens are limited to single-use.

Default policy

The cubbyhole secrets engine is mounted at the cubbyhole/ prefix by default. The secrets you store in the cubbyhole/ path are tied to your token and all tokens are permitted to read and write to the cubbyhole secrets engine by the default policy. Every token has the default policy attached unless you set the -no-default-policy flag.

$ vault policy read default

The default policy grants permission to perform all operations against the cubbyhole/ path. Also notice that the default policy permits update operation on the sys/unwrapping/ paths.

default policy

...snip...

# Allow a token to manage its own cubbyhole
path "cubbyhole/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

# Allow a token to wrap arbitrary values in a response-wrapping token
path "sys/wrapping/wrap" {
    capabilities = ["update"]
}

# Allow a token to look up the creation time and TTL of a given
# response-wrapping token
path "sys/wrapping/lookup" {
    capabilities = ["update"]
}

# Allow a token to unwrap a response-wrapping token. This is a convenience to
# avoid client token swapping since this is also part of the response wrapping
# policy.
path "sys/wrapping/unwrap" {
    capabilities = ["update"]
}

...snip...

Test the default policy

To better demonstrate the cubbyhole secrets engine, create a token with only default policy attached.

$ vault token create -policy=default \
    -format=json | jq -r ".auth.client_token" > test_token.txt

Write some test data in the token's cubbyhole using the test token.

$ VAULT_TOKEN=$(cat test_token.txt) vault write cubbyhole/private mobile="123-456-7890"
Success! Data written to: cubbyhole/private

Read back the secret you just wrote. It should return the secret.

$ VAULT_TOKEN=$(cat test_token.txt) vault read cubbyhole/private

Key       Value
---       -----
mobile    123-456-7890

Now, try to read the cubbyhole/private path.
```
$ vault read cubbyhole/private
```
Note
The command uses the token stored in the VAULT_TOKEN environment variable instead of test_token.txt. When in doubt, you can use the vault token lookup command to see which token you are using.
Output:
```
No value found at cubbyhole/private
```
Cubbyhole secret backend provide an isolated secrete storage area for an individual token where no other token can violate.

Help and Reference

Versioned Key/value secrets engine

AD service account check-out