Vault Agent Windows service

16min
|
Vault

All authenticated requests to Vault must include a valid token, which means that Vault clients must first authenticate with Vault, and receive a token.

The Secure Introduction of Vault Clients tutorial details available approaches for solving the secret zero problem.

One comprehensive and robust way to solve this problem is to use Vault Agent. Vault Agent can also manage the lifecycle of additional secrets beyond secret zero.

Challenge

Vault Agent is most effective when available to the client application or user environment on a continuous basis and operated as a system service instead of a foreground user process.

Solution

You can configure Vault Agent to operate as a Windows service to automatically start when Windows starts, and become available for use when the client application or user needs it.

Scenario introduction

Diagram showing Vault Agent connection to Vault server

You can register Vault as a Windows service in different ways. You will use either the Service Control tool, sc.exe or the New-Service cmdlet to register Vault Agent as a Windows service in this tutorial.

For this scenario, you will use PowerShell sessions and command line invocations to start a Vault dev mode server and configure the AppRole auth method for use by Vault Agent.

You will then configure Vault Agent and register it as a service.

You will also confirm that the Agent service can start and authenticate with the dev mode Vault server.

Prerequisites

You need the following to complete this section of the tutorial:

A Windows host (Tutorial last tested with Windows Server 2019).
An Administrator level user account on the Windows host that has the capability to register system services. Follow this tutorial while logged in as the local administrator account.
Vault version 1.7 or later; you can follow the Install Vault tutorial to install Vault on Windows. The community edition is suitable for completing the example scenario.
jq for consuming Vault JSON formatted output and capturing specific fields for output to files; you need jq to follow specific steps in the example scenario.

Lab setup

Vault Agent in production will typically use an external Vault cluster, but for the purposes of this example scenario, you will start and prepare a Vault dev mode server and the Vault Agent on the same host.

Tip

The dev mode server stores everything in memory, so you can just stop it when finished to clean up.

Start a PowerShell session. Start the Vault dev mode server, and specify root as the initial root token value.

$ vault server -dev -dev-root-token-id=root

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.15.10
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.7.0
             Version Sha: d77a09d565024d3c8bef98a101752ef620ed3063
...
snip
...
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

PowerShell:
    $env:VAULT_ADDR="http://127.0.0.1:8200"
cmd.exe:
    set VAULT_ADDR=http://127.0.0.1:8200

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: ahOWBI3/vLhSJuW5eYhq8rdEM8g0JGROiaSzVSUY794=
Root Token: root

Development mode should NOT be used in production installations!

Insecure operation

Do not run a Vault dev server in production. You are using this approach to simplify the unsealing process for the example scenario.

Start a second PowerShell session and set an environment variable with the Vault server address.
```
$ $env:VAULT_ADDR="http://127.0.0.1:8200"
```
Set an environment variable with the Vault token.
```
$ $env:VAULT_TOKEN="root"
```
Verify the connection to the Vault server.
```
$ vault status
```

Leave the two PowerShell terminals open with the Vault server running. The Vault dev mode server is now ready for use.

When using HCP Vault Dedicated, the Vault server will run in the HashiCorp Cloud Platform while the Vault Agent runs on your Windows host.

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
Start a PowerShell session, set the VAULT_ADDR environment variable to the copied address.
```
$ $env:VAULT_ADDR="<Public_Cluster_URL>"
```
Return to the Overview page and click Generate token.
Vault generates a new admin token within moments.
Copy the Admin Token.
Return to your PowerShell session, set the VAULT_TOKEN environment variable to store the token value.
```
$ $env:VAULT_TOKEN="<Admin_Token>"
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ $env:VAULT_NAMESPACE="admin"
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.
Verify the connection to the Vault server.
```
$ vault status
```

Leave the PowerShell terminal open. The Vault Dedicated server is ready.

Configure Vault

Vault Agent needs an enabled auth method for automatically authenticating to Vault. Enable the AppRole auth method.
```
$ vault auth enable approle

Success! Enabled approle auth method at: approle/
```
The success message indicates that you enabled the AppRole auth method at the default path approle/.
Create a role named vault-agent-role for later use by the Agent.
```
$ vault write auth/approle/role/vault-agent-role secret_id_ttl=90m token_num_uses=10 token_ttl=60m token_max_ttl=120m secret_id_num_uses=20

Success! Data written to: auth/approle/role/vault-agent-role
```
Note
The restrictions on token use from this example role emphasize the importance of short lived secrets.
In the vault-agent-role, the AppRole secret ID has a 90 minute time to live (TTL) and you can use it 20 times. You can use tokens acquired from this role 10 times, with a maximum TTL of 2 hours.
Vault Agent will use this role by passing a role ID and secret ID when it performs the automatic authentication.

Configure Vault Agent

In this scenario, Vault Agent will execute with an example configuration instructing it to connect to the Vault dev mode server and authenticate using the AppRole auth method.

You need to write the vault-agent-role role ID and a matching secret ID to files in the Vault agent directory to authenticate with Vault using the AppRole auth method. When the Agent service starts, it will consume these files as part of its automatic authentication functionality.

Note

The default behavior for each automatic authentication that Agent tries when using AppRole is to delete the secret ID file regardless of whether authentication succeeds. You should expect the agent-secret-id file to be missing after starting the service.

From your existing PowerShell session, create a top-level directory to contain the configuration and other data created when following the example scenario.
```
$ New-Item -path c:\vault-agent -ItemType "directory"

Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        3/18/2021   8:35 AM                vault-agent
```
Note
These examples use the path C:\vault-agent, but you can use any name and have access to, so long as you substitute it for any reference of the scenario directory name.

Write the role ID to file in the scenario directory called agent-role-id.

$  vault read auth/approle/role/vault-agent-role/role-id -format=json | jq -r '.data.role_id' | Out-File -encoding ascii C:\vault-agent\agent-role-id -NoNewline

Write the secret ID to file in the scenario directory called agent-secret-id.

$ vault write -f auth/approle/role/vault-agent-role/secret-id -format=json | jq -r '.data.secret_id' | Out-File -encoding ascii C:\vault-agent\agent-secret-id -NoNewline

Check the file contents, keeping in mind that this will reveal the secret ID value.

$  Get-Content c:\vault-agent\agent-role-id ; Get-Content c:\vault-agent\agent-secret-id

28a05ccc-99bd-babd-9be0-29e890c279f3
8f941d1a-ed3b-bd3b-a939-2836b633e541

The Vault Agent service uses a configuration file to connect to the Vault server. Assign the file content to a variable named AgentConfiguration.

$AgentConfiguration = @"
pid_file = "/vault-agent/agent.pid"

vault {
  address = "${env:VAULT_ADDR}"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/vault-agent/agent-role-id"
      secret_id_file_path = "/vault-agent/agent-secret-id"
    }
  }

  sink "file" {
    config = {
      path = "/vault-agent/agent-token"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = true
}
"@

If you have Vault binary version 1.13.0 or later, you can specify a file to write Vault Agent log messages.

$AgentConfiguration = @"
pid_file = "/vault-agent/agent.pid"

log_file = "/vault-agent/agent-trace.log"

vault {
  address = "${env:VAULT_ADDR}"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/vault-agent/agent-role-id"
      secret_id_file_path = "/vault-agent/agent-secret-id"
    }
  }

  sink "file" {
    config = {
      path = "/vault-agent/agent-token"
    }
  }
}

cache {
  use_auto_auth_token = true
}

listener "tcp" {
  address     = "127.0.0.1:8100"
  tls_disable = true
}
"@

A log file named vault-agent-{timestamp}.log will be generated. To see the full list of available parameters, refer to the Vault Agent documentation.

Write the AgentConfiguration variable contents to the Vault Agent configuration file.
```
$ Set-Content C:\vault-agent\vault-agent.hcl $AgentConfiguration
```
Confirm the Vault address in the Agent configuration file.
$ Get-Content C:\vault-agent\vault-agent.hcl | Select-String -Pattern 'address'
The first line is for your Vault server - http://127.0.0.1.8200. The second line is the listener address for the Vault Agent - 127.0.0.1:8100.
address = "http://127.0.0.1:8200" address = "127.0.0.1:8100"
$ Get-Content C:\vault-agent\vault-agent.hcl | Select-String -Pattern 'address'
The first line is for your Vault server matching the public address retrieved from the HCP Portal ending with port 8200. The second line is the listener address for the Vault Agent - http://127.0.0.1:8100.
address = "https://vault-cluster.vault.1b3d5f-1234-5678-ba99-s3asam3str.aws.hashicorp.cloud:8200" address = "127.0.0.1:8100"

Unset existing environment variables.

$ Remove-Item Env:\VAULT_ADDR ; Remove-Item Env:\VAULT_TOKEN

Verify removal of the environment variables.
```
$ Get-Item Env:
```
You are now prepared to register the Agent service.

Register Vault Agent as a Windows service

One way to register the service is to use Service Control.

Service Control works best if the path to your Vault binary and the associated Agent configuration file do not contain space characters. Service Control can be difficult to configure if your path has spaces, as you must quote paths containing spaces, and escaping quotes is often non-trivial.

Another alternative is to use the New-Service cmdlet. New-Service is more tolerant about escaping quotes in paths containing spaces. It can sometimes be easier to configure if your path has spaces or you prefer not to use Service Control.

Choose the method you prefer to learn about, and register the Vault Agent service.

Heads up

When you use the Service Control tool, ensure that you explicitly invoke sc.exe, and not just sc so that you do not invoke the Set-Content built-in alias by mistake.

The following example command registers a Vault Agent service that uses "Vault Agent" as the display name and automatically starts when Windows boots.

The binPath argument should include the fully qualified path to the Vault binary executable, and all required arguments. In this case, it includes the -config flag to specify the Agent configuration that you just created.

Note

The spacing after the equals sign (=) in all arguments is intentional and required to register the service without error. The binPath value here is for the vault.exe location when Vault installed with Chocolatey, and you should adjust it to match your Vault executable path if you installed Vault another way.

$ sc.exe create VaultAgent binPath="C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl" displayName="VaultAgent" start=auto
[SC] CreateService SUCCESS

$ sc.exe create VaultAgent binPath="C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -namespace=admin -config=C:\vault-agent\vault-agent.hcl" displayName="VaultAgent" start=auto
[SC] CreateService SUCCESS

If the output shows "[SC] CreateService SUCCESS", then service registration is complete.

If you observe an error, verify the path to the Vault binary, and check the arguments, by running the contents of binPath= directly in a PowerShell session and observing the results.

The following example command registers a Vault Agent service that uses "Vault Agent" as the display name and automatically starts when Windows boots.

Note

The binPath value here is for the vault.exe location when Vault installed with Chocolatey, and you should adjust it to match your Vault executable path if you installed Vault another way.

$ New-Service -Name "VaultAgent" -BinaryPathName "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -config=C:\vault-agent\vault-agent.hcl" -DisplayName "VaultAgent" -StartupType "Automatic"

$ New-Service -Name "VaultAgent" -BinaryPathName "C:\ProgramData\chocolatey\lib\vault\tools\vault.exe agent -namespace=admin -config=C:\vault-agent\vault-agent.hcl" -DisplayName "VaultAgent" -StartupType "Automatic"

As noted earlier, New-Service is easier to use if the path to your Vault executable or agent configuration has spaces in the name.

Here is an example of how to configure Vault Agent as a service using a path with spaces for your reference.

$ New-Service -Name "VaultAgent" -BinaryPathName '"C:\my dir\vault.exe" agent -config="C:\my dir\agent-config.hcl"' -DisplayName "Vault Agent" -StartupType "Automatic"

Note

You double-quote the paths themselves, but wrap the entire BinaryPathName in single quotes to escape the double quotes used for the paths.

If anything goes wrong during the service registration process, and you need to manually edit the path later, use the Registry Editor to find the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VaultAgent.

You can then edit the ImagePath value at that key to correct the path value.

Start Vault Agent service

You can start the service with Service Control, the Start-Service cmdlet, or with the UI in the Windows Service Manager.

Start the Agent service with sc.exe.

$ sc.exe start VaultAgent

SERVICE_NAME: VaultAgent
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 2  START_PENDING
                                (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x7d0
        PID                : 4728
        FLAGS              :

You can use the graphical Service Manager and start the Agent service by following these steps.

Click the Windows menu, and type Services.
Select Services to start the application.
Find the VaultAgent service in the list and double-click it.
In the Vault Agent Properties dialog, click Start.
When you start the service, it will have a Running status in the listing.

The Vault Agent service is now started and running.

Verify the Agent token

If the Agent is operating properly, it will automatically authenticate with Vault using AppRole and get a token that it will write to the file sink at C:\vault-agent\agent-token.

Using this token ID value and the Vault Agent address, you can further authenticate with Vault.

Set the Vault Agent address as the value to the VAULT_ADDR environment variable so that you are communicating directly to the Agent and not the Vault dev mode server.

$ $env:VAULT_ADDR="http://127.0.0.1:8100"

Capture the token contained in the agent-token file as the VAULT_TOKEN environment variable value.

$ $env:VAULT_TOKEN = Get-Content C:\vault-agent\agent-token -Raw

Try a token lookup.

$ vault token lookup

Example output:

Key                  Value
---                  -----
creation_time        1616100193
creation_ttl         1h
display_name         approle
entity_id            c0a7378e-1d42-dade-dfb0-d6d448228008
expire_time          2021-03-18T15:25:13.1964324-07:00
explicit_max_ttl     0s
issue_time           2021-03-18T13:43:13.1406039-07:00
last_renewal         2021-03-18T14:25:13.1964324-07:00
last_renewal_time    1616102713
meta                 map[role_name:vault-agent-role]
num_uses             7
orphan               true
path                 auth/approle/login
policies             [default]
renewable            true
ttl                  20m22s
type                 service

Insecure operation

Displaying plaintext values in terminal sessions is typically discouraged; this command is for illustrative purposes.

Note that the role_name contained in the meta is vault-agent-role, and the token has just the default policy attached. This is confirmation that the token originated from the AppRole auth method on the dev server.

IMPORTANT: `VAULT_ADDR` usage

In this tutorial, you set the VAULT_ADDR environment variable to the target Vault address. Keep in mind that Vault Agent has its own client for Vault server connections.

This is how Vault Agent transparently proxies connections from the CLI to the Vault server. If you set a VAULT_ADDR environment variable, the Vault Agent will use that value to connect to Vault. This can create an infinite loop where Vault Agent uses the value of VAULT_ADDR to make a connection, and ends up trying to connect to itself instead of the server.

When the connection fails, the Vault Agent increments the port and tries again. The agent repeats these attempts, which leads to port exhaustion.

This problem is a result of the precedence order of the 3 different ways to configure the Vault address. They are, in increasing order of priority:

Configuration files
Environment variables
CLI flags

Warning

The configuration files have the lowest priority and CLI flags have the highest priority. A value of VAULT_ADDR in the environment will override the Vault server address configured in the Vault Agent configuration file. A CLI flag will override everything.

How to avoid infinite loop problem

Do NOT set VAULT_ADDR globally in the environment so that the Vault Agent will use the value in its configuration file to connect to the target Vault server instead. This means that you need to manually set VAULT_ADDR for any sessions involving the Vault CLI.
If VAULT_ADDR is set globally in the environment, add the -address="<server address>" CLI flag to the command invocation when creating the Windows service (specified as binPath during service creation). This CLI flag will override both the configuration file and the environment variable, and ensure that Vault Agent is always talking to the correct server.
Example command:
```
$ New-Service -Name "VaultAgent" -BinaryPathName '"C:\my dir\vault.exe" agent -config="C:\my dir\agent-config.hcl" -address="https://production-vault-server:8200"' -DisplayName "Vault Agent" -StartupType "Automatic"
```

Next steps

You have learned how to configure Vault Agent as a Windows service and confirm that the service registers and starts.

You showed that the Vault Agent could connect to the Vault dev mode server, and authenticate by checking for a Vault token at the configured sink location.

You also used the Agent and token ID value to authenticate with Vault as a secondary confirmation that Agent acquired a token from the AppRole auth method.

From here, you can expand your Vault Agent knowledge, and learn about more advanced features, such as Agent Caching and Agent Templates.

Help and reference

Blog post: Why Use the Vault Agent for Secrets Management?
Video: Streamline Secrets Management with Vault Agent and Vault 0.11
Vault Agent
sc.exe create
New-Service

Vault Agent process supervisor mode

Using Vault Agent with .NET Core

Challenge

Solution

Scenario introduction

Prerequisites

Lab setup

Configure Vault

Configure Vault Agent

Register Vault Agent as a Windows service

Start Vault Agent service

Verify the Agent token

IMPORTANT: VAULT_ADDR usage

How to avoid infinite loop problem

Next steps

Help and reference

IMPORTANT: `VAULT_ADDR` usage