Secure multi-tenancy with namespaces
Enterprise Only
The namespaces feature requires a Vault Enterprise Standard license, or HCP Vault cluster.
Everything in Vault is path-based, and often uses the terms path
and
namespace
interchangeably. The application namespace pattern is a useful
construct for providing Vault as a service to internal customers, giving them
the ability to implement secure multi-tenancy within Vault in order to provide
isolation and ensure teams can self-manage their own environments.
Personas
The scenario described in this tutorial introduces the following personas:
operations
is the cluster-level administrator with privileged policiesorg-admin
is the organization-level administratorteam-admin
is the team-level administrator
Challenge
When Vault is primarily used as a central location to manage secrets, multiple organizations within a company may need to be able to manage their secrets in a self-serving manner. This means that a company needs to implement a Vault as a Service model allowing each organization (tenant) to manage their own secrets and policies. The most importantly, tenants should be restricted to work only within their tenant scope.
Solution
Create a namespace dedicated to each team, organization, or app where they can perform all necessary tasks within their tenant namespace.
Each namespace can have its own:
- Policies
- Auth Methods
- Secrets Engines
- Tokens
- Identity entities and groups
Tokens are locked to a namespace. Identity groups can pull in entities and groups from other namespaces.
Prerequisites
To perform the tasks described in this tutorial, you need to have a Vault Enterprise environment.
Note
The creation of namespaces should be performed by a user with a
highly privileged token such as root
to set up isolated environments for
each organization, team, or application.
Lab setup
Open a terminal and start a Vault dev server with root
as the root token.
The Vault dev server defaults to running at 127.0.0.1:8200
. The server is
also initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault
CLI to address the Vault server.
Export an environment variable for the vault
CLI to authenticate with the
Vault server.
The Vault server is ready.
Scenario introduction
In this tutorial, you are going to create a namespace dedicated to the Education organization which has Training and Certification teams. Delegate operational tasks to the team admins so that the Vault cluster operators won't have to be involved.
The following steps are demonstrated:
- Create namespaces
- Write policies
- Set up entities and groups
- Test the organization admin user
- Test the team admin user
- Audit ambient credentials
Create namespaces
(Persona: operations)
Important Notes
This tutorial focuses on the use of Vault namespaces. Refer to Vault Limits and Maximums to understand the known upper limits on namespaces. Also, read Vault Namespace and Mount Structuring Guide for additional guidance on structuring Vault namespaces.
To create a new namespace, run: vault namespace create <namespace_name>
Create a namespace dedicated to the
education
organizations.In this example, the
education
namespace ID is72cdg
.Create child namespaces called
training
under theeducation
namespace.Create child namespaces called
certification
under theeducation
namespace.List the existing namespaces on the root.
List the existing namespace on the
education
namespace.
Custom metadata
Note
This section is informational. You can proceed to the Write policies section without performing the command below.
If you are running Vault 1.12.0 or later, you can add custom metadata to each
namespace using -custom-metadata
flag. Custom metadata can be useful when you
have machine-generated namespace names. You can set custom metadata to provide
human-friendly tags or names to the namespace.
Example output:
You can add custom metadata to an existing namespaces using the patch
sub-command.
You can pass multiple -custom-metadata=<key>=<value>
to set more than one
custom metadata with vault namespace create
or vault namespace patch
commands.
For example:
Write policies
(Persona: operations)
In this scenario, there is an organization-level administrator who is a
superuser within the scope of the education
namespace. Also, there is a
team-level administrator for training
and certification
.
Policy for education admin
Requirements:
- Create and manage namespaces
- Create and manage policies
- Enable and manage secrets engines
- Create and manage entities and groups
- Manage tokens
Policy for training admin
Requirements:
- Create and manage child-namespaces
- Create and manage policies
- Enable and manage secrets engines
Note
Refer to the Additional Discussion section to learn more about policy authoring with namespaces.
To target a specific namespace, you can do one of the following:
Option 1: Set VAULT_NAMESPACE
so that all subsequent CLI commands will
be executed against that particular namespace.
Option 2: Specify the target namespace with -namespace
flag
Since you have to deploy policies onto education
and education/training
namespaces, use -namespace
flag instead of environment variable.
Create
edu-admin
policy undereducation
namespace.Create
training-admin
policy undereducation/training
namespace.
Tip
To codify the creation of namespaces and policies under each namespaces, read the Codify Management of Vault Enterprise Using Terraform tutorial.
Set up entities and groups
(Persona: operations)
In the education
namespace, create an entity, Bob Smith with edu-admin
policy attached. Add a userpass
user, bob as an entity alias. By default, Bob Smith has no visibility into the education/training
namespace since the bob
user was defined in the education
namespace.
You are going to create an internal group named, Training Admin in the
education/training
namespace with training-admin
policy attached. To grant
the training-admin
policy for bob
, add the Bob Smith
entity to the
Training Admin
group as a member entity.
Tip
This step only demonstrates CLI commands and Web UI to create entities and groups. Refer to the Identity - Entities and Groups tutorial if you need the full details. Also, read the Additional Discussion section for an example of setting up external groups.
Enable the
userpass
auth method.Create a user
bob
under theeducation
namespace.Create an entity for
Bob Smith
withedu-admin
policy attached. Save the generated entity ID in a file namedentity_id.txt
.Get the mount accessor for userpass auth method and save it in
accessor.txt
.Create an entity alias for
Bob Smith
to attachbob
.Create a group, "Training Admin" in
education/training
namespace withBob Smith
entity as its member.
Test the Bob Smith entity
(Persona: org-admin)
Log in as
bob
into theeducation
namespace.Example output:
User
bob
only hasdefault
policy attached to his token (token_policies
); however, he inherited theedu-admin
policy from theBob Smith
entity (identity_policies
).Store the generated token value in
BOB_TOKEN
environment variable.Example:
Test to make sure that
bob
can create a namespace, enable secrets engine, and whatever else that you want to verify against theeducation
namespace.Verify that you can create a new namespace called
web-app
.Verify that you can enable key/value v2 secrets engine at
edu-secret
.Optionally, you can create new policies to test that
bob
can perform the operations as expected.Unset the VAULT_NAMESPACE environment variable.
Test the training admin group
(Persona: team-admin)
Look up the token details.
Example output:
Notice that the
external_namespace_policies
parameter liststraining-admin
policy. The userbob
inherited this policy from theTraining Admin
group defined in theeducation/training
namespace althoughbob
user was created in theeducation
namespace.Verify that
bob
can perform the operations permitted by thetraining-admin
policy.Set the target namespace as an environment variable.
Create a new namespace called
vault-training
.Enable key/value v1 secrets engine at
team-secret
.Unset the VAULT_NAMESPACE environment variable.
Summary
As this tutorial demonstrated, each namespace you created behaves as an
isolated Vault environment. By default, there is no visibility into other
namespaces regardless of its hierarchical relationship. In order for Bob to
operate in education/training
namespace, you can enable an auth method in the education/training
namespace so that he can log in. Or, as demonstrated in this tutorial, you can use Vault identity to associate entities and groups defined in different namespaces.
Note
Bob still needs to log into the education
namespace since his
token is tied to the education
namespace and it is invalid to log into the
education/training
namespace.
Audit ambient credentials
(Persona: operator)
Many auth and secrets providers, such as AWS, Azure, GCP, and AliCloud, use ambient credentials to authenticate API calls. For example, AWS may:
- Use an access key and secret key configured in Vault.
- If not present, check for environment variables such as
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
. - If not present, load credentials configured in
~/.aws/credentials
. - If not present, use instance metadata.
This becomes a problem if these ambient credentials are not intended to be used within a particular namespace.
For example, suppose that your Vault server is running on an AWS EC2 instance. You give the owner of a namespace a particular set of permissions to use for AWS. However, that owner does not configure them. So, Vault falls back to using the credentials available in instance metadata, leading to a privilege escalation.
To handle this:
- Ensure no environment variables are available that could grant a privilege escalation.
- Ensure that any privileges granted through instance metadata (in this example) or other ambient identity info represent a loss of privilege.
- Directly configure the correct credentials in namespaces, and restrict access to that endpoint so credentials can't later be edited to use ambient credentials.
Summary
As this tutorial demonstrated, each namespace you created behaves as an isolated Vault environment. Once you sign into a namespace, there is no visibility into other namespaces regardless of its hierarchical relationship. Tokens, policies, and secrets engines are tied to its namespace; therefore, each client must acquire a valid token for each namespace to access their secrets.
Auth methods and identity groups
For simplicity, this tutorial used the username and password (userpass
) auth
method which was enabled in the education
namespace as well as the
education/training
namespace. However, most likely, your organization uses an
auth method such as ldap
or okta
and map appropriate policies to those
externally defined groups.
For these types of auth methods, you have two options:
Option 1: Enable the auth method in each namespace the same way you performed in the
set up entities and groups step with userpass
.
Option 2: Enable the auth method in the root namespace and use Identity Groups to pull in external groups and map policies in each namespace.
Note
As mentioned in the Solution section, identity groups can pull in entities and groups from other namespaces.
The second option enables an auth method in the root namespace rather than enable in multiple namespaces.
The following steps demonstrate the Option 2 to create the "Training Admin" group as described in this tutorial.
Enable and configure the desired auth method (e.g. LDAP) in the root namespace.
Configure the LDAP auth method.
For each LDAP group to which you want to map policies, create an external group in the root namespace with an alias whose name exactly matches the LDAP group name.
Get the mount accessor for ldap auth method and save it in
accessor.txt
.Create an external group and save the generated group ID in
group_id.txt
.Create a group alias whose name exactly matches the LDAP group name. The following example uses a group name in LDAP is
ops_training
.In the
education/training
namespace, create an internal group which has the external group (training_admin_root
) as its member. Attach thetraining-admin
policy to this internal group. Remember that you created this policy inside of theeducation/training
namespace in the write policies step.Note
Remember that the auth methods are enabled in an explicit namespace. Therefore, the users must authenticate into the namespace where the auth method is configured (in this case, the
ldap
auth method is configured in the root namespace).Assuming the user, "bob_smith" belongs to the LDAP
ops_training
group.When you check the returned token's properties, the output should display
training-admin
as itsexternal_namespace_policies
.Since you did not assign any policy when you created the
training_admin_root
external group, theidentity_policies
showsnil
. Therefore, bob_smith's token only has thedefault
policy in the root namespace. However, it hastraining-admin
policy in the external namespace of ID,9dKXw
(in this example, it is the ID ofeducation/training
namespace).Set the target namespace as an environment variable for convenience, and
bob_smith
is ready to operate against theeducation/training
namespace.
Tip
The namespaces feature is designed to create an isolation around each namespace so that Vault tenants can operate independently by having a mini-Vault environment of their own. However, the second option leaves some dependency on the root namespace. The root-level admin is responsible for creating the mapping between the LDAP groups and the Vault's identity groups. The users log into the root namespace instead of logging in the target namespace.
Policy with namespaces
In this tutorial, you created policies in each namespace (education
and
education/training
). Therefore, you did not have to specify the target
namespace in the policy paths.
If you want to create policies in the root namespace to control education
and
education/training
namespaces, prepend the namespace in the paths.
For example:
In the write policies step, you deployed the training-admin
policy in
the education/training
namespace. The path is relative to the working
namespace. So, if you want to create the training-admin
policy in the
education
namespace instead, the paths starts with training/
rather than
education/training/
.
Note
Policies are tied to its namespace. Therefore, if you created the
training_admin
policy in the education
namespace, it is available only
to those clients authenticated in the education
namespace.
Administrative namespace
Vault version
The administrative namespace requires Vault Enterprise 1.15 or later.
The Vault API includes system backend endpoints, which are mounted under the
sys/
path. Those system backend endpoints let you interact with the internal
features of your Vault. Some of those endpoints are restricted that you can
reach only from the root namespace, or require a token in the root namespace
with elevated permissions.
However, there may be instances where a Vault operator needs to provide access
to a subset of those restricted endpoints without granting access to the full
set of privileged sys/
paths.
Vault 1.15 introduced administrative namespace to let Vault operators grant access to a subset of privileged endpoints by setting a parameter in the Vault configuration file. An administrative namespace is a second namespace you can choose to have access to some otherwise root-only API endpoints. This allows access to these namespaces to be possible outside the root namespace.
Stop the running Vault dev server.
Create a lab setup script named
lab-setup.sh
with the following content.lab-setup.shMake sure that the file is executable.
Run the script.
The
lab-setup.sh
script:- Created a Vault configuration file (
vaut-config.hcl
) which uses integrated storage - Started a Vault server and outputs the system log in a file named
vault-1.log
- Initialized the Vault server with 1 key share and the threshold of 1. The
generated unseal key and the initial root token are saved in a file named
key.txt
- Unsealed the Vault server
- Login with Vault using the initial root token
- Create
admin
andns_test
namespaces - Stop the server
- Created a Vault configuration file (
Update the generated
vault-config.hcl
file by adding theadministrative_namespace_path
value.vault-config.hclRe-start the Vault server.
Unseal Vault.
Verify the new permissions
To verify permissions for the administrative namespace, compare API responses from a restricted endpoint from your new namespace and another namespace without elevated permissions.
Use the monitor command to call the
/sys/monitor
endpoint from your test namespace.The command returns an unsupported path error:
Use the
monitor
command to call thesys/monitor
endpoint from your administrative namespace.You should see log data from your Vault instance streaming to the terminal.
Example output:
Press Ctrl + C to stop the monitor.
Clean up
Unset the
VAULT_TOKEN
,VAULT_ADDR
andVAULT_NAMESPACE
environment variables.You can stop the Vault dev by executing the following command.
Delete the generated files.
Delete the
./vault
directory.