Batch tokens

14min
|
Vault
Interactive

Tokens are the core method for authentication within Vault. Tokens can be used directly or dynamically generated by the auth methods. Regardless, clients need valid tokens to interact with Vault.

There are two types of tokens: service tokens and batch tokens. The service tokens are persisted; therefore, they can be renewed or revoked before reaching their time-to-live (TTL) value. On the other hand, batch tokens are not persisted. They are encrypted binary large objects (blobs) that carry enough information for them to be used for Vault actions. Therefore, batch tokens are extremely lightweight and scalable; however, they lack most of the flexibility and features of service tokens. Batch tokens cannot be listed or manually revoked.

Service tokens vs. batch tokens

As the number of machines and apps using Vault for secret management scales, Vault must manage the growing number of client tokens. The creation of service tokens can affect Vault performance since they must be replicated across the primary and secondary clusters. On the other hand, batch tokens are neither persisted to disk nor live in memory; they are not a part of the data replication process.

Due to the lack of features with batch tokens, it's preferable to use service tokens in most use cases. However, think of a scenario where you have a large number of containers (e.g. 100,000s) start up and all request a token from Vault. The use of batch tokens can reduce the stress on the storage backend and improve the overall performance.

Batch tokens are designed to be lightweight with limited flexibility. The following table highlights the differences between batch tokens and service tokens.

	Service tokens	Batch tokens
Can be root tokens	Yes	No
Can create child tokens	Yes	No
Renewable	Yes	No
Manually Revocable	Yes	No
Can be periodic	Yes	No
Can have explicit Max TTL	Yes	No (always uses a fixed TTL)
Has accessors	Yes	No
Has Cubbyhole	Yes	No
Revoked with parent (if not orphan)	Yes	Stops Working
Dynamic secrets lease assignment	Self	Parent (if not orphan)
Can be used across Performance Replication clusters	No	Yes (if orphan)
Creation scales with performance standby node count	No	Yes
Cost	Heavyweight; multiple storage writes per token creation	Lightweight; no storage cost for token creation

Cross cluster usage

To use a batch token across performance replication clusters, it has to be an orphan token since the secondary cluster will not be able to ensure the validity of its parent token.

Prerequisites

To perform the tasks described in this tutorial, you need a Vault environment. Refer to the Getting Started tutorial to install Vault.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Lab setup

Open a terminal and start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.

Insecure operation

Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

Note

For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

The Vault server is ready.

Create batch tokens

Create a test policy.

$ vault policy write test -<<EOF
path "auth/token/create" {
   capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

If you are not familiar with policies, complete the policies tutorial.

Create a batch token with the test policy attached, and set a TTL so the token is valid for 20 minutes.

$ vault token create -type=batch -policy=test -ttl=20m

Key                  Value
---                  -----
token                hvb.AAAAAQJyBEVE-vTWUrg0hcoIPuvKjjNxXXZ5MfsYVg2gJ0fGZpVi0IGTFfh4TqsoQIWaocNRXD1qzGXvhIHWJBM_rWU9YJY8sXOYVy_s1JAHasXJwGmZ_fBLJfSG6aCwQkCGwtAhYw
token_accessor       n/a
token_duration       20m
token_renewable      false
token_policies       ["default" "test"]
identity_policies    []
policies             ["default" "test"]

The generated token value starts with b. or hvb. to indicate that it is a batch token. Refer to the Tokens for Vault version specific token prefix.

Store the generated token in an environment variable, BATCH_TOKEN.

Example:

$ export BATCH_TOKEN="hvb.AAAAAQJyBEVE-vTWUrg0hcoIPuvKjjNxXXZ5MfsYVg2gJ0fGZpVi0IGTFfh4TqsoQIWaocNRXD1qzGXvhIHWJBM_rWU9YJY8sXOYVy_s1JAHasXJwGmZ_fBLJfSG6aCwQkCGwtAhYw"

Lookup the token details.

$ vault token lookup $BATCH_TOKEN

Example output:

Key                 Value
---                 -----
accessor            n/a
creation_time       1646698338
creation_ttl        20m
display_name        token
entity_id           n/a
expire_time         2022-03-07T16:32:18-08:00
explicit_max_ttl    0s
id                  hvb.AAAAAQJyBEVE-vTWUrg0hcoIPuvKjjNxXXZ5MfsYVg2gJ0fGZpVi0IGTFfh4TqsoQIWaocNRXD1qzGXvhIHWJBM_rWU9YJY8sXOYVy_s1JAHasXJwGmZ_fBLJfSG6aCwQkCGwtAhYw
issue_time          2022-03-07T16:12:18-08:00
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [default test]
renewable           false
ttl                 13m58s
type                batch

Notice that renewable is set to false.

Test batch tokens

Create some secrets in the Cubbyhole secrets engine.

$ VAULT_TOKEN=$BATCH_TOKEN vault write cubbyhole/token value="1234567890"

Output:

Error writing data to cubbyhole/token: Error making API request.

URL: PUT $VAULT_ADDR/v1/cubbyhole/token
Code: 400. Errors:

cubbyhole operations are only supported by "service" type tokens

Batch tokens do not have a cubbyhole associated with it.

Create a child token.

$ VAULT_TOKEN=$BATCH_TOKEN vault token create -policy=default

Error creating token: Error making API request.

URL: POST $VAULT_ADDR/v1/auth/token/create
Code: 400. Errors:

batch tokens cannot create more tokens

Batch tokens cannot create child tokens even if their policies have the capabilities to do so.

Try revoking the batch token.
```
$ vault token revoke $BATCH_TOKEN

Error revoking token: Error making API request.

URL: PUT $VAULT_ADDR/v1/auth/token/revoke
Code: 400. Errors:

Batch tokens cannot be revoked.
```
The TTL values of batch tokens are fixed. In this example, the TTL is set to 20 minutes. After 20 minutes, the token expires and Vault will revoke it. Batch tokens cannot be renewed.

Create a test policy.

First, create the HTTP request payload.

$ tee payload.json <<EOF
{
  "policy": "path \"auth/token/create\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n"
}
EOF

Use the sys/policies/acl endpoint to create test policy.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request PUT \
    --data @payload.json \
    $VAULT_ADDR/v1/sys/policies/acl/test

To create a batch token, you need to explicitly set the token type to be batch. This cannot be done as root. Create a batch token with test policy attached.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
   --data '{"type": "batch", "policies": ["test"], "ttl": "20m"}' \
   $VAULT_ADDR/v1/auth/token/create | jq .auth

Note

This example uses jq to process the JSON output for readability.

Example output:

{
  "client_token": "hvb.AAAAAQL4ppLlkrV2vNeOda-WR2qWfqJ0q4ka7wp5xH_EZnBj3Pz9Ymu5miMcO9LVnxL2f9PwzNBtB9qQmWb9WMliPhX3-EFafTjDw0b5b3WEcRQf-A-6E_f3k39m_YgT0G08cgqKiQ",
  "accessor": "",
  "policies": [
    "default",
    "test"
  ],
  "token_policies": [
    "default",
    "test"
  ],
  "metadata": null,
  "lease_duration": 1200,
  "renewable": false,
  "entity_id": "",
  "token_type": "batch",
  "orphan": false,
  "mfa_requirement": null,
  "num_uses": 0
}

The generated token value starts with b. or hvb. to indicate that it is a batch token. Refer to the Tokens for Vault version specific token prefix.

Store the generated token in an environment variable, BATCH_TOKEN.

Example:

$ export BATCH_TOKEN="hvb.AAAAAQL4ppLlkrV2vNeOda-WR2qWfqJ0q4ka7wp5xH_EZnBj3Pz9Ymu5miMcO9LVnxL2f9PwzNBtB9qQmWb9WMliPhX3-EFafTjDw0b5b3WEcRQf-A-6E_f3k39m_YgT0G08cgqKiQ"

Lookup the token details.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" \
    $VAULT_ADDR/v1/auth/token/lookup-self | jq .data

Example output:

{
  "accessor": "",
  "creation_time": 1646699963,
  "creation_ttl": 1200,
  "display_name": "token",
  "entity_id": "",
  "expire_time": "2022-03-07T16:59:23-08:00",
  "explicit_max_ttl": 0,
  "id": "hvb.AAAAAQL4ppLlkrV2vNeOda-WR2qWfqJ0q4ka7wp5xH_EZnBj3Pz9Ymu5miMcO9LVnxL2f9PwzNBtB9qQmWb9WMliPhX3-EFafTjDw0b5b3WEcRQf-A-6E_f3k39m_YgT0G08cgqKiQ",
  "issue_time": "2022-03-07T16:39:23-08:00",
  "meta": null,
  "num_uses": 0,
  "orphan": false,
  "path": "auth/token/create",
  "policies": [
    "default",
    "test"
  ],
  "renewable": false,
  "ttl": 683,
  "type": "batch"
}

Notice that renewable is set to false.

Test batch tokens

Create some secrets in the Cubbyhole secrets engine.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" \
      --request POST \
      --data '{"value": "1234567890"}' \
      $VAULT_ADDR/v1/cubbyhole/token | jq

Output:

{
  "errors": [
    "cubbyhole operations are only supported by \"service\" type tokens"
  ]
}

Batch tokens do not have a cubbyhole associated with it.

Create a child token.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" --request POST \
      --data '{"policies": ["default"]}' \
      $VAULT_ADDR/v1/auth/token/create | jq

Output:

{
  "errors": ["1 error occurred:\n\t* permission denied\n\n"]
}

If the batch token has a policy allowing to create tokens, the error message will be different.

{
  "errors": ["batch tokens cannot create more tokens"]
}

Batch token cannot create child tokens, even if its policy grants permission.

Try to revoke the batch token.

First, create the HTTP request payload.

$ tee payload_token_revoke.json <<EOF
{
   "token": "$BATCH_TOKEN"
}
EOF

Try revoking the batch token.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
      --data @payload_token_revoke.json \
      $VAULT_ADDR/v1/auth/token/revoke | jq

Output:

{
  "errors": ["batch tokens cannot be revoked"]
}

The TTL values of batch tokens are fixed. In this example, the TTL is set to 20 minutes. After 20 minutes, the token expires and Vault will revoke it. Batch tokens cannot be renewed.

Create a test policy.

First, create the HTTP request payload.

$ tee payload.json <<EOF
{
  "policy": "path \"auth/token/create\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n"
}
EOF

Use the sys/policies/acl endpoint to create test policy.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request PUT \
    --data @payload.json \
    $VAULT_ADDR/v1/sys/policies/acl/test

To create a batch token, you need to explicitly set the token type to be batch. This cannot be done as root. Create a batch token with test policy attached.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"type": "batch", "policies": ["test"], "ttl": "20m"}' \
   $VAULT_ADDR/v1/auth/token/create | jq .auth

Note

This example uses jq to process the JSON output for readability.

Example output:

{
  "client_token": "b.AAAAAQJ5VqYthSwEWwrqL7MboI3tm0LvaRz3zbbXJZSJdvU6MHMJa76emx24byI1vCp5EYI61JzWlYoM9IhME6j1u26SLhLECD5doIxo2YtfLIHcVrjf4U8W_ytaYFH702LC2z77d6JMR5A54H_0Qx5fFZ2CDE0tiFMsJwlvYAAu3CKMrYqIJsSpdGlm8q3_O6R7ayBjkBtn_09TAK-aas9N2Sbe4HNbV3g.SFJnc",
  "accessor": "",
  "policies": [
    "default",
    "test"
  ],
  "token_policies": [
    "default",
    "test"
  ],
  "metadata": null,
  "lease_duration": 1200,
  "renewable": false,
  "entity_id": "bbd4f028-aab1-8146-9a75-e0a78241b662",
  "token_type": "batch",
  "orphan": false
}

The generated token value starts with b. or hvb. to indicate that it is a batch token. Refer to the Tokens for Vault version specific token prefix.

Store the generated token in an environment variable, BATCH_TOKEN.

Example:

$ export BATCH_TOKEN="b.AAAAAQJ5VqYthSwEWwrqL7MboI3tm0LvaRz3zbbXJZSJdvU6MHMJa76emx24byI1vCp5EYI61JzWlYoM9IhME6j1u26SLhLECD5doIxo2YtfLIHcVrjf4U8W_ytaYFH702LC2z77d6JMR5A54H_0Qx5fFZ2CDE0tiFMsJwlvYAAu3CKMrYqIJsSpdGlm8q3_O6R7ayBjkBtn_09TAK-aas9N2Sbe4HNbV3g.SFJnc"

Lookup the token details.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    $VAULT_ADDR/v1/auth/token/lookup-self | jq .data

Example output:

{
  "accessor": "",
  "creation_time": 1646701407,
  "creation_ttl": 1200,
  "display_name": "token",
  "entity_id": "bbd4f028-aab1-8146-9a75-e0a78241b662",
  "expire_time": "2022-03-08T01:23:27Z",
  "explicit_max_ttl": 0,
  "id": "b.AAAAAQJ5VqYthSwEWwrqL7MboI3tm0LvaRz3zbbXJZSJdvU6MHMJa76emx24byI1vCp5EYI61JzWlYoM9IhME6j1u26SLhLECD5doIxo2YtfLIHcVrjf4U8W_ytaYFH702LC2z77d6JMR5A54H_0Qx5fFZ2CDE0tiFMsJwlvYAAu3CKMrYqIJsSpdGlm8q3_O6R7ayBjkBtn_09TAK-aas9N2Sbe4HNbV3g.SFJnc",
  "issue_time": "2022-03-08T01:03:27Z",
  "meta": null,
  "namespace_path": "admin/",
  "num_uses": 0,
  "orphan": false,
  "path": "auth/token/create",
  "policies": [
    "default",
    "test"
  ],
  "renewable": false,
  "ttl": 1014,
  "type": "batch"
}

Notice that renewable is set to false. Also, note that tokens are tied to the namespace it was created in.

Test batch tokens

Create some secrets in the Cubbyhole secrets engine.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"value": "1234567890"}' \
   $VAULT_ADDR/v1/cubbyhole/token | jq

Output:

{
  "errors": [
    "cubbyhole operations are only supported by \"service\" type tokens"
  ]
}

Batch tokens do not have a cubbyhole associated with them.

Create a child token.

$ curl --header "X-Vault-Token: $BATCH_TOKEN" --request POST \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --data '{"policies": ["default"]}' \
   $VAULT_ADDR/v1/auth/token/create | jq

Output:

{
  "errors": ["1 error occurred:\n\t* permission denied\n\n"]
}

If the batch token has a policy allowing to create tokens, the error message will be different.

{
  "errors": ["batch tokens cannot create more tokens"]
}

A batch token cannot create child tokens, even if its policy grants permission.

Try to revoke the batch token.

First, create the HTTP request payload.

$ tee payload_token_revoke.json <<EOF
{
   "token": "$BATCH_TOKEN"
}
EOF

Try revoking the batch token.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --data @payload_token_revoke.json \
   $VAULT_ADDR/v1/auth/token/revoke | jq

Output:

{
  "errors": ["batch tokens cannot be revoked"]
}

The TTL values of batch tokens are fixed. In this example, the TTL is set to 20 minutes. After 20 minutes, the token expires and Vault will revoke it. Batch tokens cannot be renewed.

There are some trade-offs for using batch tokens; however, depending on your use case, batch tokens can significantly improve the performance of your Vault environment.

Apply batch tokens

Similar to what you did in the Tokens tutorial, use the AppRole auth method as an example to generate a batch token upon a successful login.

Enable the approle auth method if not already enabled.
```
$ vault auth enable approle
```

Create a role called "shipping", which generates a batch token with a TTL of 20 minutes.

$ vault write auth/approle/role/shipping policies="shipping" \
     token_type="batch" \
     token_ttl="20m"

Output:

Success! Data written to: auth/approle/role/shipping

Verification

Note

If you are not familiar with the AppRole auth method, read the AppRole Pull Authentication tutorial.

Retrieve the RoleID for the shipping role and save it to a file.

$ vault read -format=json auth/approle/role/shipping/role-id \
    | jq -r ".data.role_id" > role_id.txt

Generate a SecretID for the shipping role and save it to a file as well.

$ vault write -f -format=json auth/approle/role/shipping/secret-id \
    | jq -r ".data.secret_id" > secret_id.txt

Authenticate with Vault using the generated role_id and secret_id. Store the generated client token in a file named, shipping_token.txt.

$ vault write -format=json auth/approle/login role_id=$(cat role_id.txt) \
   secret_id=$(cat secret_id.txt) | jq -r ".auth.client_token" > shipping_token.txt

View the token properties.

$ vault token lookup $(cat shipping_token.txt)

Example output:

Key                 Value
---                 -----
accessor            n/a
creation_time       1646699466
creation_ttl        20m
display_name        approle
entity_id           eca5b5ca-b927-6ae0-63a0-f1e6e29e36e7
expire_time         2022-03-07T16:51:06-08:00
explicit_max_ttl    0s
id                  hvb.AAAAAQKLaR6oaiRuA6i20-SP4iNtQa4hFUXf8OLmgwC97z5mQA381KecMo4GlL-P_Q9DupzmZANHUmIfHO3133oYlKvnWkugiarvrhApTb9iv-0SUzL2MSH0ks6HNY9b7mRK1JHgHZuiwUfUeUrJxPrVjVNdukDsealGiV72FU4xD3KCiPuTijJHL3fkU2FKNuwdHWAfobFIfHarEOYrfrcsPXkD
issue_time          2022-03-07T16:31:06-08:00
meta                map[role_name:shipping]
num_uses            0
orphan              true
path                auth/approle/login
policies            [default shipping]
renewable           false
ttl                 18m22s
type                batch

The output shows the type is batch.

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Unset the BATCH_TOKEN environment variable.
```
$ unset BATCH_TOKEN
```
Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```

Delete the files used to test tokens.

$ rm role_id.txt secret_id.txt payload.json payload_token_revoke.json shipping_token.txt

If you are running Vault locally in -dev mode, you can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```

Next steps

You learned the characteristics of batch tokens. Next, go through the Token management tutorial to learn the basic operational tasks for the Token auth method.

Reference materials

Tokens

Token management