Vault Agent - secrets as environment variables

14min
|
Beta
Vault

Consul Template and Envconsul tools have been widely used by the Vault practitioners to help integrate Vault in their existing solutions. Vault 1.3.0 introduced the Vault Agent Template feature which provides the workflow that Consul Template provides.

Vault 1.14 introduced the process supervisor mode to retrieve secrets from Vault as environment variables using Consul Template markup.

Prerequisites

To complete this tutorial, you will need:

Vault binary version 1.14.0 or later

Lab setup

Open a terminal and start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```

Log into Vault.

$ vault login root

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                root
token_accessor       9v9YS37o8lvXELvEQY6NqTtV
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Tip

Vault server is ready and the token value (root) is stored in the $HOME/.vault-token file.

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

$ vault login $VAULT_TOKEN

WARNING! The VAULT_TOKEN environment variable is set! The value of this
variable will take precedence; if this is unwanted please unset VAULT_TOKEN or
update its value accordingly.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.CAESIGFwyvuxqc319bArV9jVMV_3gZH3OV434JYu8niYpDRRGigKImh2cy5oRjlaWVhtYWtBN1U1QUV4cnpZV1Bjb2kuWDhXaDIQyIAM
token_accessor       Ri7jMehrYjAWvnN7EufsRX9O.X8Wh2
token_duration       5h56m42s
token_renewable      false
token_policies       ["default" "hcp-root"]
identity_policies    []
policies             ["default" "hcp-root"]

Tip

Vault server is ready and the token value is store in the $HOME/.vault-token file.

Setup test secrets

In your terminal, clone the learn-vault-agent-envconsul repository which contains the example configuration used in this tutorial.
```
$ git clone https://github.com/hashicorp-education/learn-vault-agent-envconsul.git
```

You can explore this repository by changing directories.

$ cd learn-vault-agent-envconsul

The folder includes the following files:

.
├── README.md
├── kv-demo.sh
├── pki-agent-config.hcl
├── pki-demo.sh
└── setup-secrets.sh

0 directories, 5 files

Run the setup-secrets.sh script. This script enables kv-v2 secrets engine at web-team/ and dev-app/ paths and creates initial test data. Also, it configures PKI secrets engine to generate certificate which you will leverage in later section.
```
$ ./setup-secrets.sh
```

Read the secrets created at web-team/data/api-keys.

$ vault kv get -mount=web-team api-keys

===== Secret Path =====
web-team/data/api-keys

======= Metadata =======
Key                Value
---                -----
created_time       2023-06-13T21:05:24.965199Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key     Value
---     -----
key1    4984h33sd
key2    o48r3eqf

Read the secrets created at dev-app/data/creds.

$ vault kv get -mount=dev-app creds

=== Secret Path ===
dev-app/data/creds

======= Metadata =======
Key                Value
---                -----
created_time       2023-06-13T21:05:25.042033Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    my-long-password
user        tester1

Read the secrets created at dev-app/data/creds/database/db-admin.

$ vault kv get -mount=dev-app creds/database/db-admin

============ Secret Path ============
dev-app/data/creds/database/db-admin

======= Metadata =======
Key                Value
---                -----
created_time       2023-06-13T21:05:25.10932Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    o3ir23ir
username    admin

Generate a Vault Agent config file

Vault Agent introduced the generate-config subcommand to auto generate the agent configuration file based on the user input.

Note

Currently, the generate-config subcommand supports kv-v1 and kv-v2 secrets paths.

Usage:

$ vault agent generate-config [options] <path_to_config>

Option	Description
-type	Currently, `env-template` is the only supported type of agent configuration
-exec	Pass the command to execute in agent process. The default is `env`
-path	Path to a kv-v1 or kv-v2 secret. Multiple paths and tail '*' wildcards are allowed.

Run the generate-config subcommand

Generates a Vault Agent configuration file named agent-config.hcl at the current working directory. If your application needs to pull secrets from multiple paths, you can provide multiple -path options as well as the wildcards (*).

Generate a Vault Agent configuration file named agent-config.hcl. Pass the kv-demo.sh as the command to execute.

$ vault agent generate-config -type="env-template" \
    -exec="./kv-demo.sh" \
    -path="web-team/api-keys" \
    -path="dev-app/*" \
    agent-config.hcl

Output:

Successfully generated "agent-config.hcl" configuration file!
Warning: the generated file uses 'token_file' authentication method, which is
not suitable for production environments.

In the absence of the output file name, the command will generate a configuration file named agent.hcl.

Open the generated configuration file with your preferred text editor to examine.

agent-config.hcl

auto_auth {
   method {
      type = "token_file"

      config {
         token_file_path = "/Users/username/.vault-token"
      }
   }
}

template_config {
   static_secret_render_interval = "5m"
   exit_on_retry_failure         = true
}

vault {
   address = "http://127.0.0.1:8200"
}

env_template "API_KEYS_KEY1" {
   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}"
   error_on_missing_key = true
}
env_template "API_KEYS_KEY2" {
   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}"
   error_on_missing_key = true
}
env_template "CREDS_PASSWORD" {
   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}"
   error_on_missing_key = true
}
env_template "CREDS_USER" {
   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}"
   error_on_missing_key = true
}
env_template "DB_ADMIN_PASSWORD" {
   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}"
   error_on_missing_key = true
}
env_template "DB_ADMIN_USERNAME" {
   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}"
   error_on_missing_key = true
}

exec {
   command                   = ["./kv-demo.sh"]
   restart_on_secret_changes = "always"
   restart_stop_signal       = "SIGTERM"
}

agent-config.hcl

auto_auth {
   method {
      type = "token_file"

      config {
         token_file_path = "/Users/username/.vault-token"
      }
   }
}

template_config {
   static_secret_render_interval = "5m"
   exit_on_retry_failure         = true
}

vault {
   address = "https://vault-cluster-public-vault-XXXXXXXXX.XXXXXXXXX.XX.hashicorp.cloud:8200"
}

env_template "API_KEYS_KEY1" {
   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key1 }}{{ end }}"
   error_on_missing_key = true
}
env_template "API_KEYS_KEY2" {
   contents             = "{{ with secret \"web-team/data/api-keys\" }}{{ .Data.data.key2 }}{{ end }}"
   error_on_missing_key = true
}
env_template "CREDS_PASSWORD" {
   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.password }}{{ end }}"
   error_on_missing_key = true
}
env_template "CREDS_USER" {
   contents             = "{{ with secret \"dev-app/data/creds\" }}{{ .Data.data.user }}{{ end }}"
   error_on_missing_key = true
}
env_template "DB_ADMIN_PASSWORD" {
   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.password }}{{ end }}"
   error_on_missing_key = true
}
env_template "DB_ADMIN_USERNAME" {
   contents             = "{{ with secret \"dev-app/data/creds/database/db-admin\" }}{{ .Data.data.username }}{{ end }}"
   error_on_missing_key = true
}

exec {
   command                   = ["./kv-demo.sh"]
   restart_on_secret_changes = "always"
   restart_stop_signal       = "SIGTERM"
}

The auto_auth stanza is using token_file as a placeholder. You can change the stanza to use desired auth method. See the following tutorials for other auto auth examples:

Vault Agent with AWS demonstrates AWS auth method
Vault Agent with Kubernetes demonstrates Kubernetes auth method

Process supervisor mode

Each generated env_template block maps to a secret key of a KV path that you passed in the command.

Environment Variable Naming Convention

The exec block sets the kv-demo.sh script as the commands that Vault Agent to execute.

Open and examine the kv-demo.sh file. It reads secrets as environment variables.

#!/bin/sh

echo
echo "==== KV DEMO APP ===="
echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"
echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"
echo "CREDS_USER        = ${CREDS_USER}"
echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"
echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"
echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"

This application does not need to know about Vault.

Start a Vault Agent

Start a Vault Agent instance that connects to the Vault server running at VAULT_ADDR.

Start a Vault Agent.

$ vault agent -config=agent-config.hcl -log-level=error

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: error
               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a

==== KV DEMO APP ====
API_KEYS_KEY1     = 4984h33sd
API_KEYS_KEY2     = o48r3eqf
CREDS_USER        = tester1
CREDS_PASSWORD    = my-long-password
DB_ADMIN_USERNAME = admin
DB_ADMIN_PASSWORD = o3ir23ir

Restarts on secrets change

The secret values could change while your application is running. Examine the behavior when the secret values change.

Add sleep 100 in the kv-demo.sh to mock a long running application.

kv-demo.sh

#!/bin/sh

echo
echo "==== DEMO APP ===="
echo "API_KEYS_KEY1     = ${API_KEYS_KEY1}"
echo "API_KEYS_KEY2     = ${API_KEYS_KEY2}"
echo "CREDS_USER        = ${CREDS_USER}"
echo "CREDS_PASSWORD    = ${CREDS_PASSWORD}"
echo "DB_ADMIN_USERNAME = ${DB_ADMIN_USERNAME}"
echo "DB_ADMIN_PASSWORD = ${DB_ADMIN_PASSWORD}"

sleep 100

Edit the agent-config.sh file to set the static_secret_render_interval to 10 seconds for the purpose of demonstration. This makes Vault Agent to pull the secrets every 10 seconds.

agent-config.hcl

auto_auth {
   method {
      type = "token_file"

      config {
         token_file_path = "/Users/username/.vault-token"
      }
   }
}

template_config {
   static_secret_render_interval = "10s"
   exit_on_retry_failure         = true
}

vault {
   address = "$VAULT_ADDR"
}

...snip...

Start the Vault Agent again.

$ vault agent -config=agent-config.hcl -log-level=error

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: error
               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a

==== KV DEMO APP ====
API_KEYS_KEY1     = 4984h33sd
API_KEYS_KEY2     = o48r3eqf
CREDS_USER        = tester1
CREDS_PASSWORD    = my-long-password
DB_ADMIN_USERNAME     = admin
DB_ADMIN_PASSWORD = o3ir23ir

Open another terminal, and connect to the target Vault server.
Tip
Refer back to the Lab setup section.
Set the VAULT_ADDR environment variable.
$ export VAULT_ADDR=http://127.0.0.1:8200
Authenticate with Vault.
$ vault login root
Set the VAULT_ADDR environment variable
$ export VAULT_ADDR=<Public_Cluster_URL>
Set the VAULT_TOKEN environment variable.
$ export VAULT_NAMESPACE=admin
Authenticate with Vault Dedicated using the admin token.
$ vault login <admin_token>

Update the secret values at web-team/data/api-keys.

$ vault kv put web-team/api-keys \
   key1="new-key-value-1" \
   key2="new-key-value-2"

Return to the terminal where Vault Agent is running and watch the output.

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: error
               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a

==== KV DEMO APP ====
API_KEYS_KEY1     = 4984h33sd
API_KEYS_KEY2     = o48r3eqf
CREDS_USER        = tester1
CREDS_PASSWORD    = my-long-password
DB_ADMIN_USERNAME = admin
DB_ADMIN_PASSWORD = o3ir23ir

==== KV DEMO APP ====
API_KEYS_KEY1     = new-key-value-1
API_KEYS_KEY2     = new-key-value-2
CREDS_USER        = tester1
CREDS_PASSWORD    = my-long-password
DB_ADMIN_USERNAME = admin
DB_ADMIN_PASSWORD = o3ir23ir

Press Ctrl + C to stop the running Vault Agent.

Work with dynamic secrets

Currently, the generate-config subcommand only supports kv-v1 and kv-v2 secrets engines; however, the Vault Agent's process supervisor mode supports dynamic secrets that Envconsul supports today. The difference is that you would have to manually define the env_template blocks.

The setup-secrets.sh script configured the PKI secrets engine and it is ready to serve.

$ vault secrets list

Example output:

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_595f8b0f    per-token private secret storage
dev-app/      kv           kv_f16c3aac           n/a
identity/     identity     identity_cd1c4e99     identity store
pki/          pki          pki_ca9e7398          n/a
pki_int/      pki          pki_331d3c63          n/a
secret/       kv           kv_1bb31236           key/value secret storage
sys/          system       system_a6bb36e2       system endpoints used for control, policy and debugging
web-team/     kv           kv_c5dd184d           n/a

(Optional) Request a certificate.

$ vault write -field=certificate pki_int/issue/example-dot-com \
   common_name="test.example.com" ttl="24h"

Example output:

Open the pki-agent-config.hcl to examine the env_template block it defines.

pki-agent-config.hcl

env_template "CERT" {
   contents = "{{ with pkiCert \"pki_int/issue/example-dot-com\" \"common_name=test.example.com\" \"ttl=24h\"}}{{ .Data.Cert }}{{ end }}"
   error_on_missing_key = true
}

exec {
   command                   = ["./pki-demo.sh"]
   restart_on_secret_changes = "always"
   restart_stop_signal       = "SIGTERM"
}

Open the pki-demo.sh file to review.
```
#!/bin/sh

echo
echo "==== CERT DEMO APP ===="
echo "CERT = ${CERT}"
```
This mock application gets certificate stored in the CERT environment variable. Similar to the kv-demo.sh, this application does not need to know anything about Vault.

Run a Vault Agent with additional env_template definition.

$ vault agent -config=agent-config.hcl \
   -config=pki-agent-config.hcl \
   -log-level=error

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: error
               Version: Vault v1.14.0-rc1, built 2023-06-06T18:12:53Z
            Version Sha: 1327dc6728853611f62708e28dbac3ce6cabad1a

==== CERT DEMO APP ====
CERT = -----BEGIN CERTIFICATE-----
MIIDZjCCAk6gAwIBAgIUdTJEGBZtcnxlo8f0mMLVzKvXbtgwDQYJKoZIhvcNAQEL
BQAwLTErMCkGA1UEAxMiZXhhbXBsZS5jb20gSW50ZXJtZWRpYXRlIEF1dGhvcml0
eTAeFw0yMzA2MTMyMzQ3MDNaFw0yMzA2MTQyMzQ3MzNaMBsxGTAXBgNVBAMTEHRl
c3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3
GXTT5UlMnH5OJKZbx7n9DJs4K4lgrKdirpD4mVL7O4ba3KUA/mam9M6LAqn5YzB6
UX6y1+zIyNaakqltxLVxiyLWodqix3fjN9g6jzis5JAQM3UIgVYPBpg5ptKzpVzH
ASvuDonnPflBYQG609rD56RK8ycyhq0bcUVCa6pEQI3EtJaqj2woOTVWyauo+r7g
jrwr49BL0ye0T1IoTSs3l0E5Pk6ZcTDEbK3rNZ+YWSzwcg6bAL1U8kOW3B3jINwI
GYXPKVVLM2v6TTYpeYB/fZ6FeQeDasVU3iy3N16HdIHbI2vAQqUkTZcv/LM1ALgf
DbEYOUvAcdNCZKifhDTdAgMBAAGjgY8wgYwwDgYDVR0PAQH/BAQDAgOoMB0GA1Ud
JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUBIsJvc4reYPkq1Vy
tuf3NMAHvD0wHwYDVR0jBBgwFoAUOiYGxwIjDoTz389uuIGb9VWmEKUwGwYDVR0R
BBQwEoIQdGVzdC5leGFtcGxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAZyN5dSKo
9E8G6kaArX6Jfq8CGdX1jhaqFH8SljYZ5/WngzJwuQNkvCU42TpAbvHsBfen93T/
VfGgPPh6GohwhvLbVQjTLzGBlFTq+oQJs0ntQxjL/CMEfqg0aLH1OnZuVULc9J6z
LaOHhsV4EHx7c5iSuwwun7NOJgysadboS8EE4BOqkKehaEz/ZXVVM5sJhRpi8KfR
I9EA+FNG6URKllW72OM2kbg9HVQsxsIPdG+o/JDsCz48yMwxJDyLAxMH/YtOVrbF
j55N7ooFPw/h3BhMP+0zzDrQrl9GPYRCe//10zmTA+4/q2vQ3nPMVeInEa+QdTBi
hxz5k4LuDjhuoA==
-----END CERTIFICATE-----

Similar to what you observed in the restarts on secrets change section, if the certificate expires and the application is still running, Vault Agent can fetch a new certificate.

Clean up

Press Ctrl + C to stop the running Vault Agent.
You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```

Press Ctrl + C to stop the running Vault Agent.
Disable the kv-v2 secrets engine enabled at web-team.
```
$ vault secrets disable web-team
```
Disable the kv-v2 secrets engine enabled at dev-app.
```
$ vault secrets disable dev-app
```
Disable the PKI secrets engine enabled at pki.
```
$ vault secrets disable pki
```
Disable the PKI secrets engine enabled at pki_int.
```
$ vault secrets disable pki_int
```
Unset the VAULT_TOKEN, VAULT_NAMESPACE and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR VAULT_NAMESPACE
```

Summary

You learned the use of the generate-config subcommand and the process supervisor mode to take advantage of secrets presented as environment variables.

Vault Agent can handle the authentication and secrets retrieval so that your application can remain Vault unaware. This reduces the barrier to adopting Vault and keep your applications secure.

There are several tutorials demonstrates the use of Vault Agent. See the available Vault Agent tutorials.

Help and reference

Vault Agent Templates

Vault Agent Windows service