Secrets management across namespaces without hierarchical relationship | Vault

10min
|
Enterprise
Vault

Enterprise Feature

Vault namespaces feature is a part of Vault Enterprise.

This tutorial demonstrates the capability introduced in Vault Enterprise 1.13.0. If you are not familiar with Vault Namespaces, complete the Secure Multi-Tenancy with Namespaces tutorial first. In addition, the Vault Namespace and Mount Structuring Guide provides some guidance on designing the namespace hierarchy.

Challenge

The purpose of Vault namespaces is to create an isolated Vault environment within a cluster so that each organization, team, or application can manage secrets independently. Therefore, Vault clients must authenticate into a specific target namespace where the secrets live.

In some use cases, this imposes a burden on the Vault clients especially if the client must read secrets from dozens of independent namespaces (namespaces with no hierarchical relationship). This operational overhead becomes more prominent when you are using Vault Agent to automate secrets management.

Tip

When namespaces have hierarchical relationship, you can write namespace-aware policies to allow access to secrets in the child namespaces.

Solution

Vault Enterprise 1.13.0 introduced the group_policy_application_mode flag which enables secrets sharing across multiple independent namespaces. With this change, a single instance of the Vault Agent can fetch secrets across multiple namespaces. (See Cross namespace secret sharing for an Agent injector example.)

Tutorial example use case

This tutorial demonstrates the secrets management across namespaces funcationality using the userpass auth method, however, all auth methods are supported. Refer to the configure cross namespace access documentation for more information.

Prerequisites

Familiarity with Vault namespaces or completed the Secure Multi-Tenancy with Namespaces tutorial
Vault Enterprise binary v1.13.0 or later

Lab setup

Open a terminal and set your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.
```
$ export VAULT_LICENSE=02MV4UU43BK5....
```
Start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
The Vault server is ready.

Scenario setup

To demonstrate the secrets sharing across namespaces, you need the following:

New namespaces: us-west-org and us-east-org
In the us-west-org namespace:
- Enabled userpass auth method and created a "tam-user" with customer-info-read-only policy attached. The password is "my-long-password"
- Enabled KV v2 secrets engine at kv-customer-info with some test data
- Create a customer-info-read-only policy allowing to read secrets at kv-customer-info/data/*
- Created an identity entity, "TAM" and added "tam-user" as a member (an entity alias)
In the us-east-org namespace:
- Enabled KV v2 secrets engine at kv-marketing with some test data
- Created a marketing-read-only policy allowing read permission on the kv-marketing/data/campaign path
- Create an entity group named "campaign-admin" with the marketing-read-only policy attached

Tip

If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.

Use your preferred text editor, paste in the following to create a script to setup the Vault elements needed for demonstration.

lab-setup.sh

# Create new namespaces - they are peer
vault namespace create us-west-org
vault namespace create us-east-org

#--------------------------
# us-west-org namespace
#--------------------------
VAULT_NAMESPACE=us-west-org vault secrets enable -path="kv-customer-info" kv-v2
VAULT_NAMESPACE=us-west-org vault kv put kv-customer-info/customer-001 name="Example LLC" contact_email="admin@example.com"
# Create a policy to allow read access to kv-marketing
VAULT_NAMESPACE=us-west-org vault policy write customer-info-read-only -<<EOF
path "kv-customer-info/data/*" {
   capabilities = ["read"]
}
EOF
VAULT_NAMESPACE=us-west-org vault auth enable userpass
VAULT_NAMESPACE=us-west-org vault write auth/userpass/users/tam-user password="my-long-password" policies=customer-info-read-only

# Create an entity
VAULT_NAMESPACE=us-west-org vault auth list -format=json | jq -r '.["userpass/"].accessor' > accessor.txt
VAULT_NAMESPACE=us-west-org vault write -format=json identity/entity name="TAM" | jq -r ".data.id" > entity_id.txt
VAULT_NAMESPACE=us-west-org vault write identity/entity-alias name="tam-user" canonical_id=$(cat entity_id.txt) mount_accessor=$(cat accessor.txt)

#--------------------------
# us-east-org namespace
#--------------------------
VAULT_NAMESPACE=us-east-org vault secrets enable -path="kv-marketing" kv-v2
VAULT_NAMESPACE=us-east-org vault kv put kv-marketing/campaign start_date="March 1, 2023" end_date="March 31, 2023" prise="Certification voucher" quantity="100"

# Create a policy to allow read access to kv-marketing
VAULT_NAMESPACE=us-east-org vault policy write marketing-read-only -<<EOF
path "kv-marketing/data/campaign" {
   capabilities = ["read"]
}
EOF

# Create a group
VAULT_NAMESPACE=us-east-org vault write -format=json identity/group name="campaign-admin" policies="marketing-read-only" member_entity_ids=$(cat entity_id.txt)

Change the file permission to make it executable.
```
$ chmod +x lab-setup.sh
```
Run the script.
```
$ ./lab-setup.sh
```

Create a terraform folder, and change into the directory.
```
$ mkdir terraform && cd terraform
```

Create a Terraform configuration file named main.tf and paste in the following content.

main.tf

provider vault {}

# Create namespaces
resource "vault_namespace" "us-west-org" {
   path = "us-west-org"
}

resource "vault_namespace" "us-east-org" {
   path = "us-east-org"
}

# Create 'customer-info-read-only' policy
resource "vault_policy" "customer-info-read-only" {
   name       = "customer-info-read-only"
   namespace  = vault_namespace.us-west-org.path
   policy     = <<EOF
path "kv-customer-info/data/*" {
   capabilities = ["read"]
}
EOF
}

# Create 'marketing-read-only' policy
resource "vault_policy" "marketing-read-only" {
   name       = "marketing-read-only"
   namespace  = vault_namespace.us-east-org.path
   policy = <<EOF
path "kv-marketing/data/campaign" {
   capabilities = ["read"]
}
EOF
}

# Mount a new KV v2 secrets engine at kv-customer-info/ in the us-west-org namespace
resource "vault_mount" "kv-customer-info" {
   depends_on  = [vault_namespace.us-west-org]
   namespace   = vault_namespace.us-west-org.path
   path        = "kv-customer-info"
   type        = "kv-v2"
}

# Create test data at kv-customer-info/customer-001
resource "vault_kv_secret_v2" "customer" {
   depends_on  = [vault_mount.kv-customer-info]
   namespace   = vault_namespace.us-west-org.path
   mount                      = vault_mount.kv-customer-info.path
   name                       = "customer-001"
   delete_all_versions        = true
   data_json                  = jsonencode(
   {
      name          = "Example LLC",
      contact_email = "admin@example.com"
   }
   )
}

# Mount a new KV v2 secrets engine at kv-marketing/ in the us-east-org namespace
resource "vault_mount" "kv-marketing" {
   depends_on  = [vault_namespace.us-east-org]
   namespace   = vault_namespace.us-east-org.path
   path        = "kv-marketing"
   type        = "kv-v2"
}

# Create some test data at kv-marketing/campaign
resource "vault_kv_secret_v2" "campaign" {
   depends_on  = [vault_mount.kv-marketing]
   namespace   = vault_namespace.us-east-org.path
   mount                      = vault_mount.kv-marketing.path
   name                       = "campaign"
   delete_all_versions        = true
   data_json                  = jsonencode(
   {
      start_date = "March 1, 2023",
      end_date   = "March 31, 2023",
      prise      = "Certification, voucher",
      quantity   = "100"
   }
   )
}

#------------------------------------------------------------
# Enable userpass in the us-west-org namespace
#------------------------------------------------------------
resource "vault_auth_backend" "userpass" {
   type = "userpass"
   namespace  = vault_namespace.us-west-org.path
   path = "userpass"
}

# Create a user named, "tam-user"
resource "vault_generic_endpoint" "tam-user" {
   depends_on           = [vault_auth_backend.userpass]
   namespace            = vault_namespace.us-west-org.path
   path                 = "auth/userpass/users/tam-user"
   ignore_absent_fields = true

   data_json = <<EOT
{
"policies": ["customer-info-read-only"],
"password": "my-long-password"
}
EOT
}

#------------------------------------------------------------
# Create an entity, 'TAM' in "us-west-org/" namespace
#------------------------------------------------------------
resource "vault_identity_entity" "tam" {
   namespace   = vault_namespace.us-west-org.path
   name        = "TAM"
}

resource "vault_identity_entity_alias" "tam" {
   depends_on      = [vault_identity_entity.tam]
   namespace       = vault_namespace.us-west-org.path
   name            = "tam-user"
   mount_accessor  = vault_auth_backend.userpass.accessor
   canonical_id    = vault_identity_entity.tam.id
}

#------------------------------------------------------------
# Create Identity Groups in "us-east-org/" namespace
#------------------------------------------------------------
resource "vault_identity_group" "campaign-admin" {
   namespace  = vault_namespace.us-east-org.path
   name       = "campaign-admin"
   policies   = ["marketing-read-only"]
   type       = "internal"
}

resource "vault_identity_group_member_entity_ids" "campaign-admin" {
   namespace  = vault_namespace.us-east-org.path
   member_entity_ids = [vault_identity_entity.tam.id]
   group_id = vault_identity_group.campaign-admin.id
   exclusive = false
}

Initialize Terraform to pull Vault provider plugin.

$ terraform init

Initializing the backend...
...snip...
Terraform has been successfully initialized!

This will download the Vault plugin.

Execute the apply command to configure Vault.

$ terraform apply

This will display the actions to be performed by Terraform.

When prompted, enter yes to accept the plan and proceed with Vault configuration.

Plan: 14 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Once completed, the output similar to the following displays.

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Vault 1.13.0 introduced a new API endpoint, sys/config/group-policy-application which sets the group_policy_application_mode parameter to allow Vault clients to manage secrets across multiple namespaces.

Look up the current setting.

$ vault read sys/config/group-policy-application
Key                              Value
---                              -----
group_policy_application_mode    within_namespace_hierarchy

The default setting is within_namespace_hierarchy.

Set the parameter value to any.

$ vault write sys/config/group-policy-application \
   group_policy_application_mode="any"

Output:

Success! Data written to: sys/config/group-policy-application

Read it back to verify.

$ vault read sys/config/group-policy-application
Key                              Value
---                              -----
group_policy_application_mode    any

Verification: Read secrets

Log in as "tam-user" and store the returned token in token.txt. Remember that auth methods are tied to the namespace they are configured. The "tam-user" is a user created in the userpass auth method in the us-west-org namespace.
```
$ VAULT_NAMESPACE=us-west-org vault login -field=token -method=userpass \
   username=tam-user password="my-long-password" > token.txt
```
Tip
You can ignore the returned warning about the VAULT_TOKEN environment variable to take precedence.

Check to see if you can read the kv-marketing/data/campaign in the us-east-org namespace using the generated token.

$ VAULT_NAMESPACE=us-east-org VAULT_TOKEN=$(cat token.txt) \
   vault kv get kv-marketing/campaign

Output:

======= Secret Path =======
kv-marketing/data/campaign

======= Metadata =======
Key                Value
---                -----
created_time       2023-02-17T04:46:03.22039Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======= Data =======
Key           Value
---           -----
end_date      March 31, 2023
prise         Certification voucher
quantity      100
start_date    March 1, 2023

You can read the secrets in the us-east-org namespace.

(Optional) The "tam-user" token has the customer-info-read-only policy attached. Therefore, you can read the secrets at kv-customer-info/ in the us-west-org namespace.

$ VAULT_NAMESPACE=us-west-org VAULT_TOKEN=$(cat token.txt) \
   vault kv get kv-customer-info/customer-001

Example output:

=========== Secret Path ===========
kv-customer-info/data/customer-001

======= Metadata =======
Key                Value
---                -----
created_time       2023-02-17T17:47:06.147303Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======== Data ========
Key              Value
---              -----
contact_email    admin@example.com
name             Example LLC

Revert back the parameter setting

Set the group_policy_application_mode parameter value back to within_namespace_hierarchy.

$ vault write sys/config/group-policy-application \
     group_policy_application_mode="within_namespace_hierarchy"

Try to read the secrets again.

$ VAULT_NAMESPACE=us-east-org VAULT_TOKEN=$(cat token.txt) \
   vault kv get kv-marketing/campaign

Output:

Error making API request.

Namespace: us-east-org/
URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/kv-marketing/campaign
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "kv-marketing/campaign/"

When the group_policy_application_mode parameter is set to within_namespace_hierarchy, you cannot access the secrets in the namespaces that has no hierarchical relationship.

Clean up

You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```
Unset the VAULT_TOKEN and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR
```
Remove the entity / token / accessor files.
```
$ rm accessor.txt entity_id.txt token.txt
```

Make sure that VAULT_TOKEN and VAULT_ADDR environment variables are set.
```
$ echo $VAULT_ADDR; echo $VAULT_TOKEN
```

Destroy the Vault resources created by Terraform.

$ terraform destroy -auto-approve

...snip...

Destroy complete! Resources: 14 destroyed.

Remove the terraform state files.
```
$ rm *.tfstate.*
```
Unset the VAULT_TOKEN and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR
```
Remove the entity / token / accessor files.
```
$ rm accessor.txt entity_id.txt token.txt
```

Summary

This tutorial demonstrated the new API endpoint, sys/config/group-policy-application and its group_policy_application_mode parameter introduced in Vault 1.13.0 to allow Vault clients to manage secrets across multiple independent namespaces.

Although secrets engines, auth methods, policies, and tokens are tied to each namespaces, the entity group can pull entities and groups from other namespaces. If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.

Help and reference

Vault namespace and mount structuring guide

Move secrets engines and auth methods

Challenge

Solution

Prerequisites

Lab setup

Scenario setup

Enable secrets sharing

Verification: Read secrets

Revert back the parameter setting

Clean up

Summary

Help and reference