Move secrets engines and auth methods across namespaces

16min
|
Enterprise
Vault

Vault Namespaces

This tutorial demonstrates Vault CLI commands and API endpoint to move mounts across Vault namespaces. The namespaces feature requires a Vault Enterprise Standard license.

Vault Enterprise first introduced namespaces in version 0.11 to allow each organization or team (tenant) to manage their own secrets and policies by creating a Vault as a Service (VaaS) environment. Vault Namespace and Mount Structuring Guide provided guidance around the namespace structure.

Challenge

Vault namespace is an ideal solution to isolate secrets for each tenant (organizations, teams, sub-teams, etc.) and let the tenant-level administrator manage the secrets, policies, and tokens; however, it has a challenge.

Think of the following situations.

Migrating from Vault Community Edition to Vault Enterprise: Currently, once you move data from a Community Edition cluster into an Enterprise cluster, you need to manually move the mounts from the root namespace into the desired target namespaces.
Structural reorganizations: Organizations change team structures and areas of responsibility or naming conventions, and as part of that would like to redistribute mounts from the current set of namespaces, into a new set of namespaces

Solution

Vault 1.10 extended the sys/remount API endpoint to support moving secrets engines and auth methods mounts from one location to another, within a namespace or across namespaces.

This tutorial focuses on the vault secrets move and vault auth move command demonstration. If you are new to the namespace feature, read the following tutorials first:

Note

The sys/remount API endpoint as well as vault secrets move and vault auth move CLI commands are available in both Vault Community Edition and Enterprise. While this tutorial demonstrates moving mounts from one namespace to another, you can remount secrets engines and auth methods from their current mount path to another without the context of namespaces.

Targeted audience

Vault Enterprise cluster administrator with privileged policies to manage namespaces, secrets engines, and auth methods.

Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault Enterprise environment.

Note

The creation of namespaces should be performed by a user with a highly privileged token such as root to set up isolated environments for each organization, team, or application.

Scenario introduction

To demonstrate the migration of mount paths from one location to another, this tutorial sets up the following namespace hierarchy.

Initially, your organization had two namespaces: education/ and education/training/. As the organization grew, the Vault admin created additional child namespaces to reflect the team structure so that each team can manage their own secrets.

The following diagram shows the new namespace hierarchy.

Scenario

Now that child namespaces were added, the Vault admin wishes to move the secrets engines and auth methods to the appropriate child namespace along with existing data.

Scenario tasks

In this tutorial, you are going to perform the Vault admin tasks to move secrets engines and auth method mounts.

Move the kv-v2 mount team-vault from education/training to education/training/security-edu/vault-edu namespace. Also, move the kv-v2 mount team-consul from education/training to education/training/consul-edu/ namespace.

Scenario 1

Warning

Each namespace provides an isolated Vault environment; therefore, the policies are tied to the namespace. When you move a mount from one path to another, you need to update the policies as well to ensure that Vault clients can access secrets from the new location. Read the Post-migration considerations section at the end.

Lab setup

Set the VAULT_ADDR environment variable to your Vault address (e.g., http://127.0.0.1:8200).
```
$ export VAULT_ADDR=<vault_ent_address>
```
Where <vault_ent_address> is your Vault address.
Set the VAULT_TOKEN environment variable to store your Vault client token.
```
$ export VAULT_TOKEN=<client_token>
```
Where <client_token> is a valid client token.

Clone the hashicorp/learn-vault-mount-move GitHub repository.

$ git clone https://github.com/hashicorp/learn-vault-mount-move.git

Be sure to set your working directory to where the learn-vault-mount-move folder is located.
```
$ cd learn-vault-mount-move
```
Make the ns-setup.sh script executable.
```
$ chmod +x ns-setup.sh
```

Run the script. This creates the namespaces, and enables secrets engines and auth methods as described in the scenario introduction.

$ ./ns-setup.sh

Script output for validation

Create 'education' namespace
Key     Value
---     -----
id      Dunpn
path    education/

Create 'education/training' namespace
Key     Value
---     -----
id      ykh3j
path    education/training/

Create 'education/certification' namespace
Key     Value
---     -----
id      1XmGJ
path    education/certification/

Create 'education/training/security-edu' namespace
Key     Value
---     -----
id      5tBqc
path    education/training/security-edu/

Create 'education/training/security-edu/vault-edu' namespace
Key     Value
---     -----
id      hXVyj
path    education/training/security-edu/vault-edu/

Create 'education/training/security-edu/boundary' namespace
Key     Value
---     -----
id      OBw8J
path    education/training/security-edu/boundary/

Create 'education/training/consul-edu' namespace
Key     Value
---     -----
id      bMO98
path    education/training/consul-edu/

Enable kv-v2 secrets engine at 'team-consul' under education/training namespace
Success! Enabled the kv-v2 secrets engine at: team-consul/

Enable kv-v2 secrets engine at 'team-vault' under education/training namespace
Success! Enabled the kv-v2 secrets engine at: team-vault/

Create some test data at team-consul/cert
Key                Value
---                -----
created_time       2022-03-03T23:57:28.322165Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create some test data at team-vault/blogs
Key                Value
---                -----
created_time       2022-03-03T23:57:28.380845Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Enable userpass auth method at 'userpass-cert' in the 'education' namespace
Success! Enabled userpass auth method at: userpass-cert/

Create a user, 'bob' in the userpass-cert auth
Success! Data written to: auth/userpass-cert/users/bob

Migrate a secrets engine

Currently, key/value v2 secrets engine is enabled at team-vault under /education/training namespace. Since vault-edu/ child namespace is created, you want to migrate the secrets engine there.

Scenario

List the enabled mounts in the education/training namespace.

$ VAULT_NAMESPACE="education/training" vault secrets list

Output:

Path            Type            Accessor                 Description
----            ----            --------                 -----------
cubbyhole/      ns_cubbyhole    ns_cubbyhole_14ed203c    per-token private secret storage
identity/       ns_identity     ns_identity_9472c7b0     identity store
sys/            ns_system       ns_system_01783e83       system endpoints used for control, policy and debugging
team-consul/    kv              kv_2b18aba4              key/value data for Consul Edu
team-vault/     kv              kv_e636e6b6              key/value data for Vault Edu

Read the pre-populated secrets.

$ VAULT_NAMESPACE="education/training" vault kv get team-vault/blogs

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-03T05:47:58.20188Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======== Data ========
Key             Value
---             -----
publish_date    March 23, 2022
title           What's New in Vault 1.10

Migrate the data at education/training/team-vault under the education/training/security-edu/vault-edu/ namespace.

$ vault secrets move education/training/team-vault \
    education/training/security-edu/vault-edu/team-vault

Current path is education/training/team-vault and the destination is education/training/security-edu/vault-edu/team-vault.

Example output:

Started moving secrets engine education/training/team-vault/ to education/training/security-edu/vault-edu/team-vault/, with migration ID 59fdb70c-ae23-910b-8781-64e9a1e9c271
Waiting for terminal status in migration of secrets engine education/training/team-vault/ to education/training/security-edu/vault-edu/team-vault/, with migration ID 59fdb70c-ae23-910b-8781-64e9a1e9c271
Success! Finished moving secrets engine education/training/team-vault/ to education/training/security-edu/vault-edu/team-vault/, with migration ID 59fdb70c-ae23-910b-8781-64e9a1e9c271

Validation

Confirm that the data was successfully migrated.

$ VAULT_NAMESPACE="education/training/security-edu/vault-edu" vault kv get team-vault/blogs

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-03T05:47:58.20188Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======== Data ========
Key             Value
---             -----
publish_date    March 23, 2022
title           What's New in Vault 1.10

The team-vault mount path no longer exists under the education/training/ namespace.

$ VAULT_NAMESPACE="education/training" vault secrets list

Path            Type            Accessor                 Description
----            ----            --------                 -----------
cubbyhole/      ns_cubbyhole    ns_cubbyhole_14ed203c    per-token private secret storage
identity/       ns_identity     ns_identity_9472c7b0     identity store
sys/            ns_system       ns_system_01783e83       system endpoints used for control, policy and debugging
team-consul/    kv              kv_2b18aba4              key/value data for Consul Edu

Namespace hierarchy in the command

What does the following command do?

$ VAULT_NAMESPACE="education/training" vault secrets move team-consul consul-edu/team-data

In this case, the VAULT_NAMESPACE environment variable specifies the current working namespace. Therefore, the target destination namespace must be relative to the current namespace (education/training), so you can omit education/training from the target namespace. Since the command specifies the destination to consul-edu/team-data, the mount gets moved to education/training/consul-edu namespace, and the path now become team-data instead of team-consul.

Example output:

Started moving secrets engine team-consul/ to consul-edu/team-data/, with migration ID c36073cd-e50b-a969-0ee4-bafa5bc04ec0
Waiting for terminal status in migration of secrets engine team-consul/ to consul-edu/team-data/, with migration ID c36073cd-e50b-a969-0ee4-bafa5bc04ec0
Success! Finished moving secrets engine team-consul/ to consul-edu/team-data/, with migration ID c36073cd-e50b-a969-0ee4-bafa5bc04ec0

Read the data at team-data/cert under the education/training/consul-edu/ namespace to verify.

$ VAULT_NAMESPACE="education/training/consul-edu" vault kv get team-data/cert

Example output:

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-04T02:24:35.481076Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key           Value
---           -----
domain        hashicorp.com
expiration    April 1, 2023

List the enabled mounts in the education/training namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: education/training" \
   $VAULT_ADDR/v1/sys/mounts | jq -r ".data"

Note

This example uses jq to process the JSON output for readability.

Example output:

{
...snip...

"team-consul/": {
   "accessor": "kv_42a40ff4",
   "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0
   },
   "description": "key/value data for Consul Edu",
   "external_entropy_access": false,
   "local": false,
   "options": {
      "version": "2"
   },
   "seal_wrap": false,
   "type": "kv",
   "uuid": "cd5e315d-171c-9410-2ecd-106e2f454f3d"
},
"team-vault/": {
   "accessor": "kv_f46da453",
   "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0
   },
   "description": "key/value data for Vault Edu",
   "external_entropy_access": false,
   "local": false,
   "options": {
      "version": "2"
   },
   "seal_wrap": false,
   "type": "kv",
   "uuid": "84a68fb8-9956-b412-f290-666ffe31bf15"
}
}

You find team-consul and team-vault listed.

Read the pre-populated secrets in the team-vault path.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/education/training/team-vault/data/blogs | jq -r ".data.data"

Output:

{
  "publish_date": "March 23, 2022",
  "title": "What's New in Vault 1.10"
}

Create the request payload to specify the current mount path (education/training/team-vault) and the target destination (education/training/security-edu/vault-edu/team-vault/).
```
$ cat > payload-kv.json <<EOF
{
   "from": "education/training/team-vault/",
   "to": "education/training/security-edu/vault-edu/team-vault/"
}
EOF
```

Migrate the secrets engine using the /sys/remount endpoint, and store the generated migration_id in a file, migration_id.txt which you can use to check the migration status.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST --silent \
   --data @payload-kv.json \
   $VAULT_ADDR/v1/sys/remount | jq -r ".migration_id" > migration_id.txt

Check the mount migration status using the sys/remount/status endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/remount/status/$(cat migration_id.txt) | jq -r ".data"

Example output:

{
  "migration_id": "28e9c3f7-a343-8917-fd34-7b89a7a0eba4",
  "migration_info": {
    "source_mount": "education/training/team-vault/",
    "target_mount": "education/training/security-edu/vault-edu/team-vault/",
    "status": "success"
  }
}

Validation

Confirm that the data was successfully migrated under the education/training/security-edu/vault-edu namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/education/training/security-edu/vault-edu/team-vault/data/blogs \
   | jq -r ".data.data"

Output:

{
  "publish_date": "March 23, 2022",
  "title": "What's New in Vault 1.10"
}

Namespace hierarchy in the command

Now, migrate the data at education/training/team-consul to education/training/consul-edu/ namespace.

First create the request payload. Notice that the from parameter is set to team-consul/ and the destination path is set to consul-edu/team-data/.
```
$ cat > payload-consul.json <<EOF
{
   "from": "team-consul/",
   "to": "consul-edu/team-data/"
}
EOF
```

Migrate the secrets engine using the /sys/remount endpoint. In this case, use the X-Vault-Namespace header to set the working namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: education/training" \
   --request POST \
   --data @payload-consul.json \
   $VAULT_ADDR/v1/sys/remount | jq -r ".data.migration_id" > migration_id_consul.txt

Check the mount migration status using the sys/remount/status endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/remount/status/$(cat migration_id_consul.txt) | jq -r ".data"

Example output:

{
  "migration_id": "14b747c8-6ae3-d02e-8c86-027ec97cc7cb",
  "migration_info": {
    "source_mount": "education/training/team-consul/",
    "target_mount": "education/training/consul-edu/team-data/",
    "status": "success"
  }
}

Read the data at team-data/cert under the education/training/consul-edu/ namespace to verify.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/education/training/consul-edu/team-data/data/cert | jq -r ".data.data"

Example output:

{
  "domain": "hashicorp.com",
  "expiration": "April 1, 2023"
}

Migrate an auth method

Similar to the secrets engine migration, you want to migrate the education/userpass-cert auth mount to the new education/certification/ namespace. Since the auth method will be in the dedicated namespace, you want to change the path name to userpass instead of userpass-cert.

Scenario

Warning

Migrating an auth method to another namespace will not move your existing entities and groups configurations. Read the Post-migration considerations section at the end.

Before migrating the auth method, login with Vault using the userpass auth method to test.

$ VAULT_NAMESPACE=education vault login -method=userpass -path=userpass-cert \
   username=bob password="change@me"

Example output:

WARNING! The VAULT_TOKEN environment variable is set! This takes precedence
over the value set by this command. To use the value set by this command,
unset the VAULT_TOKEN environment variable or set it to the token displayed
below.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIEOGwrvtkKO9LrLj63LHX9xOFyTzqkYcafZDleoy8T1NGiQKImh2cy55V1hqT0s4T2tNaE50OVJTbE1aZEd4c2YuRHVucG4
token_accessor         rXdsZNJb7jKWUBrm0F1Y3flY.Dunpn
token_duration         1h
token_renewable        true
token_policies         ["default"]
identity_policies      []
policies               ["default"]
token_meta_username    bob

Note

Tokens are tied to the namespace they were created in. Therefore, the generated token will be revoked after the auth mount migration.

Move the education/approle-cert auth mount to education/certification/approle.

$ vault auth move education/auth/userpass-cert education/certification/auth/userpass

Example output:

Started moving auth method education/auth/userpass-cert/ to education/certification/auth/userpass/, with migration ID 7d766b3d-b2c5-6fdf-23ae-b50383c3ddac
Waiting for terminal status in migration of auth method education/auth/userpass-cert/ to education/certification/auth/userpass/, with migration ID 7d766b3d-b2c5-6fdf-23ae-b50383c3ddac
Success! Finished moving auth method education/auth/userpass-cert/ to education/certification/auth/userpass/, with migration ID 7d766b3d-b2c5-6fdf-23ae-b50383c3ddac

Validation

$ VAULT_NAMESPACE=education/certification vault login -method=userpass -path=userpass \
   username=bob password="change@me"

Notice that the path is set to userpass instead of userpass-cert.

Example output:

WARNING! The VAULT_TOKEN environment variable is set! This takes precedence
over the value set by this command. To use the value set by this command,
unset the VAULT_TOKEN environment variable or set it to the token displayed
below.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.CAESIOikbPO-oki-1oMej75-Bo6JFMhasUxFmsakeW0f_VMJGiQKImh2cy5LYTNOVXFUbVVjY0VsZVh1emV3T3hPcTQuMVhtR0o
token_accessor         iROq4HU4lQfIYMlNslxv2icu.1XmGJ
token_duration         1h
token_renewable        true
token_policies         ["default"]
identity_policies      []
policies               ["default"]
token_meta_username    bob

Before migrate the auth method, login with Vault using the userpass auth method to test.

$ curl --request POST \
   --data '{ "password": "change@me" }' \
   $VAULT_ADDR/v1/education/auth/userpass-cert/login/bob | jq -r ".auth"

Example output:

{
  "client_token": "hvs.CAESIMeVKhSXftkjEh7MEf4_EM-W4SUyuwqSLM6D4HbQb7PmGiQKImh2cy4wT0ZMS1NBUnhCY1R6Uzk4YzRrWUQ4RGsubjVGcmo",
  "accessor": "S7UU9v8s7QKluykfWlsVDbWQ.n5Frj",
  "policies": [
    "default"
  ],
  "token_policies": [
    "default"
  ],
  "metadata": {
    "username": "bob"
  },
  ...snip...
}

Note

Tokens are tied to the namespace they were created in. Therefore, the generated token will be revoked after the auth mount migration.

Create the request payload to specify the current mount path (education/auth/userpass-cert/) and the target destination (education/certification/auth/userpass/).
```
$ cat > payload-auth.json <<EOF
{
   "from": "education/auth/userpass-cert/",
   "to": "education/certification/auth/userpass/"
}
EOF
```

Migrate the auth method using the /sys/remount endpoint, and store the generated migration_id in a file, migration_id_auth.txt which you can use to check the migration status.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload-auth.json \
   $VAULT_ADDR/v1/sys/remount | jq -r ".migration_id" > migration_id_auth.txt

Check the mount migration status using the sys/remount/status endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/remount/status/$(cat migration_id_auth.txt) | jq -r ".data"

Example output:

{
  "migration_id": "73688adc-7824-7a0a-84a2-c1ca3e3cdbd9",
  "migration_info": {
    "source_mount": "education/auth/userpass-cert/",
    "target_mount": "education/certification/auth/userpass/",
    "status": "success"
  }
}

Validation

$ curl --request POST \
  --data '{ "password": "change@me" }' \
  $VAULT_ADDR/v1/education/certification/auth/userpass/login/bob | jq -r ".auth"

Notice that the path is set to auth/userpass instead of auth/userpass-cert.

Example output:

{
  "client_token": "hvs.CAESIGaZS6ItRLlDkUHZKHDgS_2ILkJbHPjwQqv2t1gFqfPYGiQKImh2cy5Cd3pGWnVYMVpVcnNjaE5udVBnOWNNRG4uSnplVEc",
  "accessor": "sG1Hn4wVFAbHN9XBAJHHfMrF.JzeTG",
  "policies": [
    "default"
  ],
  "token_policies": [
    "default"
  ],
  "metadata": {
    "username": "bob"
  },
  ...snip...
}

Post-migration Considerations

Each namespace has its own policies, auth methods, secrets engines, tokens, identity entities and groups. Therefore, after the mount migration, you must consider the following:

Necessary policies exist in the target namespace
Entities and groups may need to be updated after an auth mount migration

Secrets engines

The following policy allows Vault clients to access key/value v2 secrets at team-vault as well as its metadata.

vault-edu.hcl

# Permission on the 'team-vault' secrets
path "team-vault/data/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Permission on the metadata associated with 'team-vault' secrets
path "team-vault/metadata/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}

After you successfully migrated the secrets engine from education/training/ namespace to education/training/security-edu/vault-edu, you need to make sure that the policy is deployed on the destination namespace.

$ vault policy write -namespace="education/training/security-edu/vault-edu" \
    vault-edu vault-edu.hcl

As for the team-consul, not only do you have to deploy a policy on the education/training/consul-edu namespace, but you need to ensure the policy path matches the new mount point (team-data/).

$ vault policy write -namespace="education/training/consul-edu" consul-edu -<<EOF
path "team-data/data/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}

path "team-data/metadata/*" {
   capabilities = [ "create", "read", "update", "delete", "list" ]
}
EOF

Auth methods

Similarly, you have to ensure that policies exist in the namespace so that auth configuration continues to work for Vault clients.

For example, if you created a user, bob-smith as follow:

$ VAULT_NAMESPACE="education/certification" vault write auth/userpass/users/bob-smith \
    password="change@me" token_policies="certification-edu" ttl=8h

The certification-edu policy must exist in the education/certification namespace after the migration.

Entities and groups

Vault identities (entities and groups) are also tied to a namespace. The auth remount operation will not move the existing entities and groups as a part of the migration. Therefore, you need to recreate entities in the destination namespace to maintain the entity-alias configuration; otherwise, Vault will treat the post-migration login as a new entity.

Example: Before the migration

Key                    Value
---                    -----
token                  hvs.CAESIDnCw-7i2x6bYTRwLINW4VCIiek1cJoacFEt_ubI5LZCGiQKImh2cy5KTlJYa0VtRVNHUW93VU1sSDB6d3pTdWIudmdKNG4
token_accessor         BiuIBPLu8dvzFOJkTTun9RZE.vgJ4n
token_duration         768h
token_renewable        true
token_policies         ["certification-edu" "default"]
identity_policies      ["edu-admin"]
policies               ["certification-edu" "default" "edu-admin"]
token_meta_username    bob-smith

Example: Post-migration

Key                    Value
---                    -----
token                  hvs.CAESIJWH8DZDUPARGbtkCaWpIJ-D5G9uncrj37YLUQetxuLcGiQKImh2cy5FUGlKV3g5YmRaUGljZTZLb3lzdEtIQXMuNEhJOGs
token_accessor         zMURFnO5BSEUhmWkAz6mD2A8.4HI8k
token_duration         768h
token_renewable        true
token_policies         ["certification-edu" "default"]
identity_policies      []
policies               ["certification-edu" "default"]
token_meta_username    bob-smith

This indicates that the entity-level policies and metadata were lost.

Note

If you are not familiar with Vault Identities, read the Identity: Entities and Groups tutorial.

Clean up

Make the ns-cleanup.sh script executable.
```
$ chmod +x ns-cleanup.sh
```
Run the ns-cleanup.sh script.
```
$ ./ns-cleanup.sh
```
Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```

Help and reference

Secrets management across namespaces

Disaster recovery replication setup