Install Vault to Kubernetes with Integrated Storage

14min
|
Vault

Running Vault on Kubernetes is generally the same as running it anywhere else. Kubernetes, as a container orchestration engine, eases some of the operational burdens and Helm charts provide the benefit of a refined interface when it comes to deploying Vault in a variety of different modes.

In this tutorial, you will set up Vault and its dependencies with a Helm chart. You will then integrate a web application that uses the Kubernetes service account token to authenticate with Vault and retrieve a secret.

Prerequisites

This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, minikube, the Vault Helm charts, the sample web application, and additional configuration to bring it all together.

First, follow the directions to install minikube, including VirtualBox or similar.

Next, install kubectl CLI and helm CLI.

NOTE: This tutorial was last tested in May 2022 on a macOS 12.2.1 using this configuration.

Docker version.

$ docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
  ## ...

Display minikube version.

$ minikube version
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

Helm version.

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Start minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters running inside Virtual Machines (VM) on your local system.

Start a Kubernetes cluster.

$ minikube start
😄  minikube v1.25.2 on Darwin 12.2.1
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the minikube cluster.

$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Install the Vault Helm chart

Vault manages the secrets that are written to these mountable volumes. To provide these secrets a single Vault server is required. For this demonstration Vault can be run in development mode to automatically handle initialization, unsealing, and setup of a KV secrets engine.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "secrets-store-csi-driver" chart repository
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

To verify, search repositories for vault in charts.

$ helm search repo hashicorp/vault
NAME            CHART VERSION   APP VERSION DESCRIPTION
hashicorp/vault 0.20.0          1.10.3      Official HashiCorp Vault Chart

Create a file named helm-vault-raft-values.yml with the following contents:

$ cat > helm-vault-raft-values.yml <<EOF
server:
   affinity: ""
   ha:
      enabled: true
      raft: 
         enabled: true
         setNodeId: true
         config: |
            cluster_name = "vault-integrated-storage"
            storage "raft" {
               path    = "/vault/data/"
            }

            listener "tcp" {
               address = "[::]:8200"
               cluster_address = "[::]:8201"
               tls_disable = "true"
            }
            service_registration "kubernetes" {}
EOF

Recommendation

If you are using Prometheus for monitoring and alerting, we recommend to set the cluster_name in the HCL configuration. With the Vault Helm chart, this is accomplished with the config parameter.

Install the latest version of the Vault Helm chart with Integrated Storage.

$ helm install vault hashicorp/vault --values helm-vault-raft-values.yml

Example output:

NAME: vault
LAST DEPLOYED: Wed May 18 20:19:15 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

This creates three Vault server instances with an Integrated Storage (Raft) backend.

The Vault pods and Vault Agent Injector pod are deployed in the default namespace.

Display all the pods within the default namespace. This may take a moment.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          2m12s
vault-1                                 0/1     Running   0          2m12s
vault-2                                 0/1     Running   0          2m12s
vault-agent-injector-56b65c5cd4-k7lbt   1/1     Running   0          2m13s

Initialize vault-0 with one key share and one key threshold.
```
$ kubectl exec vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > cluster-keys.json
```
The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a file named cluster-keys.json.
Display the unseal key found in cluster-keys.json.
```
$ jq -r ".unseal_keys_b64[]" cluster-keys.json
rrUtT32GztRy/pVWmcH0ZQLCCXon/TxCgi40FL1Zzus=
```
Insecure operation
Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.
Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.
```
$ VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" cluster-keys.json)
```
After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Unseal Vault running on the vault-0 pod.

$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

Insecure operation

Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.

The operator unseal command reports that Vault is initialized and unsealed.

Example output:

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.15.2
Build Date              2023-11-06T11:33:28Z
Storage Type            raft
Cluster Name            vault-integrated-storage
Cluster ID              f9762f89-fa6b-5420-1595-0a335513d7d1
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    31
Raft Applied Index      31

The Vault server is initialized and unsealed.

Join the vault-1 pod to the Raft cluster.

$ kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

Join the vault-2 pod to the Raft cluster.

$ kubectl exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200

Use the unseal key from above to unseal vault-1.

$ kubectl exec -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
Unseal Key (will be hidden):
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.10.3
Storage Type       raft
HA Enabled         true

Use the unseal key from above to unseal vault-2.

$ kubectl exec -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY

Set a secret in Vault

The web application that you deploy in the Launch a web application section, expects Vault to store a username and password at the path secret/webapp/config. To create this secret requires you to login with the root token, enable the key-value secret engine, and store a secret username and password at that defined path.

Vault generated an initial root token when it was initialized.

Display the root token found in cluster-keys.json.

$ jq -r ".root_token" cluster-keys.json
hvs.HJmsajgGlWPTx6YNHoUljuOO

Start an interactive shell session on the vault-0 pod.
```
$ kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
/ $
```
Your system prompt is replaced with a new prompt / $.
Vault is now ready for you to login with the initial root token.
Vault requires clients to authenticate first before it allows any further actions. An unsealed Vault starts with the Token Auth Method enabled and generates an initial root token.

$ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.HJmsajgGlWPTx6YNHoUljuOO
token_accessor       JVsMJHVu6rTWbPLlYmWQTq1R
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Enable an instance of the kv-v2 secrets engine at the path secret.
```
$ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/
```
Learn more
This tutorial focuses on Vault's integration with Kubernetes and not interacting with the key-value secrets engine. For more information refer to the Versioned Key/value secrets engine tutorial.

Create a secret at path secret/webapp/config with a username and password.

$ vault kv put secret/webapp/config username="static-user" password="static-password"
====== Secret Path ======
secret/data/webapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-07T05:15:19.402740412Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Verify that the secret is defined at the path secret/webapp/config.

$ vault kv get secret/webapp/config
====== Secret Path ======
secret/data/webapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-07T05:15:19.402740412Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    static-password
username    static-user

You successfully created the secret for the web application.

Exit the vault-0 pod.
```
$ exit
```

Configure Kubernetes authentication

The initial root token is a privileged user that can perform any operation at any path. The web application only requires the ability to read secrets defined at a single path. This application should authenticate and be granted a token with limited access.

We recommend that root tokens are used only for initial setup of an authentication method and policies. Afterwards they should be revoked. This tutorial does not show you how to revoke the root token.

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Start an interactive shell session on the vault-0 pod.
```
$ kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
/ $
```
Your system prompt is replaced with a new prompt / $.
Enable the Kubernetes authentication method.
```
$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
```
Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint.
Configure the Kubernetes authentication method to use the location of the Kubernetes API.
For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.
```
$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
```
Successful output from this command resembles this example:
```
Success! Data written to: auth/kubernetes/config
```
The environment variable KUBERNETES_PORT_443_TCP_ADDR is defined and references the internal network address of the Kubernetes host.
For a client to access the secret data defined, at secret/webapp/config, requires that the read capability be granted for the path secret/data/webapp/config. This is an example of a policy. A policy defines a set of capabilities.
Write out the policy named webapp that enables the read capability for secrets at path secret/data/webapp/config.
```
$ vault policy write webapp - <<EOF
path "secret/data/webapp/config" {
  capabilities = ["read"]
}
EOF
```
Successful output from this command resembles this example:
```
Success! Uploaded policy: webapp
```
Define an auth method role that uses the webapp policy. A role binds policies and environment parameters together to create a login for the web application.
Create a Kubernetes authentication role, named webapp, that connects the Kubernetes service account name and webapp policy.
```
$ vault write auth/kubernetes/role/webapp \
        bound_service_account_names=vault \
        bound_service_account_namespaces=default \
        policies=webapp \
        ttl=24h
```
Successful output from this command resembles this example:
```
Success! Data written to: auth/kubernetes/role/webapp
```
The role connects the Kubernetes service account, vault, and namespace, default, with the Vault policy, webapp. The tokens returned after authentication are valid for 24 hours.
Exit the vault-0 pod.
```
$ exit
```

Launch a web application

You have created a web application, published it to DockerHub, and created a Kubernetes deployment that will run the application in your existing cluster. The example web application performs the single function of listening for HTTP requests. During a request it reads the Kubernetes service token, logs into Vault, and then requests the secret.

Use your preferred text editor and review the contents of deployment-01-webapp.yml.

deployment-01-webapp.yml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      serviceAccountName: vault
      containers:
        - name: app
          image: hashieducation/simple-vault-client:latest
          imagePullPolicy: Always
          env:
            - name: VAULT_ADDR
              value: 'http://vault:8200'
            - name: JWT_PATH
              value: '/var/run/secrets/kubernetes.io/serviceaccount/token'
            - name: SERVICE_PORT
              value: '8080'

The web application deployment defines a list of environment variables.

JWT_PATH sets the path of the JSON web token (JWT) issued by Kubernetes. This token is used by the web application to authenticate with Vault.
VAULT_ADDR sets the address of the Vault service. The Helm chart defined a Kubernetes service named vault that forwards requests to its endpoints (i.e. The pods named vault-0, vault-1, and vault-2).
SERVICE_PORT sets the port that the service listens for incoming HTTP requests.

Deploy the webapp in Kubernetes by applying the file deployment-01-webapp.yml.
```
$ kubectl apply --filename deployment-01-webapp.yml
deployment.apps/webapp created
```
The webapp runs as a pod within the default namespace.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS              RESTARTS   AGE
vault-0                                 1/1     Running             0          15m
vault-1                                 1/1     Running             0          15m
vault-2                                 1/1     Running             0          15m
vault-agent-injector-659b4488df-4hhpz   1/1     Running             0          15m
webapp-784b8d9979-vn57d                 0/1     ContainerCreating   0          47s

The webapp pod is displayed here as the pod prefixed with webapp.

The deployment of the service requires the retrieval of the web application container from Docker Hub. This displays the STATUS of ContainerCreating. The pod reports that it is not ready (0/1).

Wait until the webapp pod is running and ready (1/1).

The webapp pod runs an HTTP service that is listening on port 8080.

In another terminal, port forward all requests made to http://localhost:8080 to the webapp pod on port 8080.

$ kubectl port-forward \
    $(kubectl get pod -l app=webapp -o jsonpath="{.items[0].metadata.name}") \
    8080:8080

In the original terminal, perform a curl request at http://localhost:8080.
```
$ curl http://localhost:8080
password:static-secret username:static-user
```
The web application running on port 8080 in the webapp pod:
- authenticates with the Kubernetes service account token
- receives a Vault token with the read capability at the secret/data/webapp/config path
- retrieves the secrets from secret/data/webapp/config path
- displays the secrets as JSON

Clean up

First, stop the running local Kubernetes cluster.

$ minikube stop

This deactivates minikube, and all pods still exist at this point.

Delete the local Kubernetes cluster.

Be aware that minikube delete removes the minikube deployment including all pods. Be sure you want everything removed before continuing.

$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/mrken/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

Next steps

You launched Vault in high-availability mode with a Helm chart. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

Also, you can run vault on Kubernetes on different platforms. Explore setting up Vault in high-availability with Integrated Storage in the Vault Installation to Google Kubernetes Engine via Helm tutorial.

Then you deployed a web application that authenticated and requested a secret directly from Vault. Explore how pods can retrieve secrets through the Vault Injector service via annotations or secrets mounted on ephemeral volumes.

Vault on Kubernetes deployment guide

Vault installation to minikube with Consul