Vault installation to Azure Kubernetes Service via Helm

14min
|
Vault

Azure Kubernetes Service (AKS) can run Vault in a managed Kubernetes cluster with the Vault UI enabled for web-based secrets management.

In this tutorial, you create a cluster in AKS, install Vault with the Web UI enabled, and then configure the authentication between Vault and the cluster. Then you deploy a web application with deployment annotations so the application's secrets are installed via the Vault Agent injector service.

Prerequisites

This tutorial requires a Azure account, Azure command-line interface (CLI), Kubernetes CLI and the Helm CLI.

First, create a Azure account.

Next, install Azure CLI, kubectl CLI and helm CLI.

Install az with Homebrew.

$ brew install azure-cli

Install kubectl with Homebrew.

$ brew install kubernetes-cli

Install helm with Homebrew.

$ brew install helm

Next, connect the az CLI with your Azure account.

$ az login

This command launches the browser and requires that you authenticate with your Microsoft account credentials.

The Kubernetes cluster that you run needs to be created in a Azure resource group. A resource group is a container that holds related resources for an Azure solution. This resource group is created with a name and a location.

Display all the locations available to your account.

$ az account list-locations | jq -r ".[].name"
eastus
eastus2
southcentralus
westus2
australiaeast
southeastasia
northeurope
uksouth
westeurope
centralus
...snip...

The output displays a list of all the location names. Select a name of a geographical location that serves you best.

Create a resource group named learn-vault with the location you selected.

$ az group create --name learn-vault --location eastus

Example output:

{
  "id": "/subscriptions/70cd78a4-2182-4077-9b93-2e199f08630b/resourceGroups/learn-vault",
  "location": "eastus",
  "managedBy": null,
  "name": "learn-vault",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

The learn-vault resource group is created in the eastus location.

Start cluster

Kubernetes clusters may run one or more nodes. Each node contains the services to run pods. Vault running in standalone mode requires one node.

Create a cluster named learn-vault-cluster with 1 node in the learn-vault resource group.

$ az aks create --resource-group learn-vault \
    --name learn-vault-cluster \
    --node-count 1 \
    --enable-addons monitoring \
    --generate-ssh-keys

The cluster is created.

High availability

Vault supports a high-availability mode. This requires a minimum of three nodes. Refer to the Vault installation to minikube via Helm with Consul tutorial for a Consul backend and Vault Installation to Google Kubernetes Engine via Helm tutorial for an integrated storage backend.

When the cluster is ready, the kubectl CLI requires configuration to communicate with this cluster.

Configure the kubectl CLI to communicate to the learn-vault-cluster cluster.

$ az aks get-credentials --resource-group learn-vault --name learn-vault-cluster
Merged "learn-vault-cluster" as current context in /Users/USERNAME/.kube/config

Managing multiple clusters

kubectl enables you to manage multiple clusters through the context configuration. You display the available contexts kubectl config get-contexts and set the context by name kubectl config use-context NAME.

Display the nodes of the cluster.

$ kubectl get nodes
NAME                                STATUS   ROLES   AGE    VERSION
aks-nodepool1-12172524-vmss000000   Ready    agent   118s   v1.18.10

The one node cluster is ready.

Install the Vault Helm chart

The recommended way to run Vault on Kubernetes is via the Helm chart.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Search for all the Vault Helm chart versions.

$ helm search repo vault --versions
NAME            CHART VERSION   APP VERSION DESCRIPTION
hashicorp/vault 0.8.0           1.5.4       Official HashiCorp Vault Chart
hashicorp/vault 0.7.0           1.5.2       Official HashiCorp Vault Chart
hashicorp/vault 0.6.0           1.4.2       Official HashiCorp Vault Chart
hashicorp/vault 0.5.0                       Install and configure Vault on Kubernetes.
hashicorp/vault 0.4.0                       Install and configure Vault on Kubernetes.

The Vault Helm chart contains all the necessary components to run Vault in several different modes.

Default behavior

By default, it launches Vault in standalone mode with a file storage backend on a single pod. Enabling the Vault web UI requires that you override the defaults.

Install the latest version of the Vault Helm chart with the Web UI enabled.

$ helm install vault hashicorp/vault \
  --set='ui.enabled=true' \
  --set='ui.serviceType=LoadBalancer'

The Vault pod, Vault Agent Injector pod, and Vault UI Kubernetes service are deployed in the default namespace.

Get all the pods within the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          30s
vault-agent-injector-56bf46695f-crqqn   1/1     Running   0          30s

The vault-0 pod deployed runs a Vault server and reports that it is Running but that it is not ready (0/1). This is because the status check defined in a readinessProbe returns a non-zero exit code.

The vault-agent-injector pod deployed is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if specific annotations exist within the request.

Retrieve the status of Vault on the vault-0 pod.

$ kubectl exec vault-0 -- vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        false
Sealed             true
Total Shares       0
Threshold          0
Unseal Progress    0/0
Unseal Nonce       n/a
Version            n/a
HA Enabled         false
command terminated with exit code 2

The status command reports that Vault is not initialized and that it is sealed. For Vault to authenticate with Kubernetes and manage secrets requires that that is initialized and unsealed.

Initialize and unseal Vault

Vault starts uninitialized and in the sealed state. The process of initializing and unsealing Vault can be performed via the exposed Web UI.

Command-line interface

The Vault CLI on the vault-0 is able to initialize and unseal the Vault server. For more information refer to the Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar tutorial.

The Vault web UI is available through a Kubernetes service.

Display the vault-ui service in the default namespace.

$ kubectl get service vault-ui
NAME       TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)          AGE
vault-ui   LoadBalancer   10.0.1.209   20.62.223.255   8200:32656/TCP   9m15s

The EXTERNAL-IP displays the IP address of the Vault UI. In this example output this address is 20.62.223.255.

Launch a web browser, and enter the EXTERNAL-IP with the port 8200 in the address. For example: http://20.62.223.255:8200.
Enter 5 in the Key shares and 3 in the Key threshold text fields.
Click Initialize.
When the root token and unseal key is presented, scroll down to the bottom and select Download keys. Save the generated unseal keys file to your computer.
The unseal process requires these keys and the access requires the root token.
Click Continue to Unseal to proceed.

Open the downloaded file.

Example key file:

{
  "keys": [
    "ecfb4ef59f9a2570f856c471cd3b0580e2b7d99962d5c9af7a25b80138affe935a",
    "807e9bbfb984c631becc526c621c9852f82d88b2347f7398ef7af3c1fbfbbe9fd0",
    "561a7ff6b44b88f96a2d9faca1ae514d1557008ce19283dcfe2fb746ed4f0f7d94",
    "3671e9e817177d79d3c004e0745e5f1d1a5cbfcd9fd6ad22505d4bc538176fa3f9",
    "313fffc1c848276fffe1e3fcfce4d3472d104cda466227ca155e4f693cfbaa36b9"
  ],
  "keys_base64": [
    "7PtO9Z+aJXD4VsRxzTsFgOK32Zli1cmveiW4ATiv/pNa",
    "gH6bv7mExjG+zFJsYhyYUvgtiLI0f3OY73rzwfv7vp/Q",
    "Vhp/9rRLiPlqLZ+soa5RTRVXAIzhkoPc/i+3Ru1PD32U",
    "NnHp6BcXfXnTwATgdF5fHRpcv82f1q0iUF1LxTgXb6P5",
    "MT//wchIJ2//4eP8/OTTRy0QTNpGYifKFV5PaTz7qja5"
  ],
  "root_token": "s.p3L38qZwmnHUgIHR1MBmACfd"
}

Copy one of the keys (not keys_base64) and enter it in the Master Key Portion field. Click Unseal to proceed.
The Unseal status shows 1/3 keys provided.
Enter another key and click Unseal.
The Unseal status shows 2/3 keys provided.
Enter another key and click Unseal.
After 3 out of 5 unseal keys are entered, Vault is unsealed and is ready to operate.
Copy the root_token and enter its value in the Token field. Click Sign In.

Set a secret in Vault

The web application that you deploy in the Deploy web application step, expects Vault to store a username and password at the path secret/webapp/config. To create this secret requires you to enable the key-value secret engine, and store a secret username and password at that defined path.

Select the Secrets tab in the Vault UI.
Under Secrets Engines, select Enable new engine.
Under Enable a Secrets Engine, select KV and Next.
Enter secret in the Path text field.
Select Enable Engine.
The view changes to display all the secrets for this secrets engine.
Select the Create secret action.
Enter devwebapp/config in the Path for this secret.
Under Version data, enter username in the key field and giraffe in the value field.
Select Add to create another key and value field in Version data.
Enter password in the key field and salsa in the value field.
Select Save to create the secret.
The view displays the contents of the newly created secret.

You successfully created the secret for the web application.

Configure Kubernetes authentication

The initial root token is a privileged user that can perform any operation at any path. The web application only requires the ability to read secrets defined at a single path. This application should authenticate and be granted a token with limited access.

Best practice

We recommend that root tokens are used only for initial setup of an authentication method and policies. Afterwards they should be revoked. This tutorial does not show you how to revoke the root token.

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

Select the Access tab in the Vault UI.
The view displays all the authentication methods that are enabled.
Under Authentication Methods, select Enable new method.
Under Enable an Authentication Method, select Kubernetes and Next.
The view displays the method options configuration for the authentication method.
Select Enable Method to create this authentication method with the default method options configuration.
The view displays the configuration settings that enable the auth method to communicate with the Kubernetes cluster. The Kubernetes host, CA Certificate, and Token Reviewer JWT require configuration. These values are defined on the vault-0 pod.
Enter the address returned from the following command in Kubernetes host field.
```
$ echo "https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"
https://10.0.0.1:443
```
The command displays the environment variables defined on the node, filters to only the KUBERNETES_PORT_443_TCP_ADDR address and then prefixes the https:// protocol and appends the 443 port.
For the Kubernetes CA Certificate field, toggle the Enter as text.

Enter the certificate returned from the following command in Kubernetes CA Certificate entered as text.

$ kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-----BEGIN CERTIFICATE-----
MIIEyjCCArKgAwIBAgIRAI20oNPOAqRDhNnOzHjtkAYwDQYJKoZIhvcNAQELBQAw
DTELMAkGA1UEAxMCY2EwIBcNMjAxMjE0MjAwNTM0WhgPMjA1MDEyMTQyMDE1MzRa
MA0xCzAJBgNVBAMTAmNhMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
sScrgikc12Tif7Py+8cbyuQ82D+D14734JkJuf1Z5pMHJEJzXydjRDKpf1ut+yO0
IdZMrn4fjyRe8qrzjpH6pOD2WdSqZlzb23XIRV1byVqsWwir77I6p6KO6DNPyAcN
YQ92W9SDlwKHUoyByJxwOTIwcobpZcentUVd9ld4ExA5uFXu7VzKfxVOmHW0m1aO
xgOKZEZJrPnWpZBMHK14Cqq1yxV5LxZEF3xfb6ONYnXsY+IgddEkSDgiUxTkkDZv
5wYLiWIY/DhfJvlFDH1c7nYfIbukpmzOKqempKI/aWvJxgtHKYnW7ydvdPkv2dMv
nAH5puNrsdKXN2HNfdytZGX9tpQIssRpgOCgjKV0+9liyE2MJ1/vmndlKomamC1L
GD7LrRPUlf0hlXXTIgMAXaUtNjaS4KT37mZI9Z5yUZUH5Unfu19WDsRvYp4ZuYqC
zEvTy06aJ5maUYmdobK0fIbpHr5Vbiu3J54Co7w1EJvZlwM91D6VhlzkodsKI4Jn
+WEmZSewvUgTCJDPWgHhgahsl9ZuOW47M9ZQkR/HXKtqm4iygNcPX2eN1LlZpFrG
D213WFmW6g5P6j0wILaZxfk9ufEsPmh5dptx1WGSgtJ0yfwNlvwhRBrJFuksKMpX
ZziT/ALbDe7hPnkxGZcyEJCjXCqkfQphbIT1FJpNjlsCAwEAAaMjMCEwDgYDVR0P
AQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggIBABWJ
wCW4i/S4DJ+othnoDyy30rSxkKl4RIczJ+/TFL6iqDcK4bElasxxWCgXV15fDpH4
z6dI5Ic5jyu+pK/7cwY3C1Y3uZq8e/8/+4Q1UJs5xRb7BPAQLlHlHF74Z4Ho3THn
uHjfN2k/V+atQEZiZ5CCvtR3tvVCGV3iO9i21pVvA+ypt61qGlXGt1ENoxZJX8ts
j8kBo+Ud1lwiWblV1Ba6yses7lYCX/GAiK75uvuKGW51m6oU6ItvXie3og6YDW7Y
05YhqJYhQnbYV4tTAjBmuR1S4BeQUJBkArbwHnMxsSwUSHPFSIj+VrjexS+K0H9g
vx2bXrQILivGxmysjJlA0YwcTLLDhjX6hlW1AR1fVWOt8odTGZ5Vk24bqWBw3fY1
W68Gm230L5T4fdb0CwLKeowbIBraYD6z/E9L3pJCA8ZUYPWeWdZm55ZKqp5M8A8f
0QLn15Vbb57EKLhPndBJlgomLuaIHD1K9je3lhlWYeoJjDvKGq6Jjw8OAThRNsBp
RRx/X5HD2pLACpgLlcqjTcsSa+wdI5lFGrR7uubt+z8igsmJT5pP1jL5AFrzHqOQ
WAm4ifPDF1bAgUhJMhvPL5B2JJPYjwlKZ4qbrLLDifg3KNBJ9D0B9tldWTw1oD3p
UTZbPZUwkIFK9UygG8na5fZFqQG+8sCjOF2hXfou
-----END CERTIFICATE-----

Access - Kubernetes auth config with certificate as text

Expand the Kubernetes Options section.

Enter the token returned from the following command in Token Reviewer JWT field.

$ echo $(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
eyJhbGciOiJSUzI1NiIsImtpZCI6ImdFUjRkcWlValh5SXJ2Q3NvO ... snipped ...

The command displays the JSON Web Token (JWT) mounted by Kubernetes on the vault-0 pod.

Access - Kubernetes auth config with JWT

Select Save.

The Kubernetes authentication method is configured to work within the cluster. The web application requires a Vault policy to read the secret and a Kubernetes role to grant it that policy.

Configure web application authentication

For a client of the Vault server to read the secret data defined in the Set a secret in Vault step requires that the read capability be granted for the path secret/data/devwebapp/config.

Select the Policies tab in the Vault UI.
Under ACL Policies, select the Create ACL policy action.
Enter devwebapp in the Name field.

Enter this policy in the Policy field.

# Read the configuration secret
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}

Policies - create devwebapp policy

Select Create policy.
The policy is created and the view displays its name and contents.
The policy is assigned to the web application through a Kubernetes role. This role also defines the Kubernetes service account and Kubernetes namespace that is allowed to authenticate.
Select the Access tab in the Vault UI.
Under Authentication Methods, click the ... for the kubernetes/ auth method. Select View configuration.
Under the kubernetes method, choose the Roles tab.
This view displays all the roles defined for this authentication method.
Select Create role.
Enter devweb-app in the Name field.
Enter internal-app in the Bound service account names field.
This field contains one or more Kubernetes service accounts that this role supports. This Kubernetes service account is created in the Deploy web application step.
Enter default in the Bound service account namespaces field and select Add.
This field contains one or Kubernetes namespaces that this role supports.
Expand the Tokens section.
Enter devwebapp in the Generated Token's Policies.
Select Save.
The role is created and the view displays its name in the roles view for this authentication method.

Deploy web application

The web application pod requires the creation of the internal-app Kubernetes service account specified in the Vault Kubernetes authentication role created in the Configure Kubernetes authentication step.

Define a Kubernetes service account named internal-app.

$ cat > internal-app.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
EOF

Create the internal-app service account.

$ kubectl apply --filename internal-app.yaml

Define a pod named devwebapp with the web application.

$ cat > devwebapp.yaml <<EOF
---
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  labels:
    app: devwebapp
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "devweb-app"
    vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
spec:
  serviceAccountName: internal-app
  containers:
    - name: devwebapp
      image: jweissig/app:0.0.1
EOF

This definition creates a pod with the specified container running with the internal-app Kubernetes service account. The container within the pod is unaware of the Vault cluster. The Vault Injector service reads the annotations to find the secret path, stored within Vault at secret/data/devwebapp/config and the file location, /vault/secrets/secret-credentials.txt, to mount that secret with the pod.

Learn more

For more information about annotations refer to the Injecting Secrets into Kubernetes Pods via Vault Agent Injector tutorial.

Create the devwebapp pod.

$ kubectl apply --filename devwebapp.yaml

Get all the pods within the default namespace.

$ kubectl get pods
devwebapp                               2/2     Running   0          6s
vault-0                                 1/1     Running   0          8m48s
vault-agent-injector-56bf46695f-gdpsz   1/1     Running   0          8m48s

Wait until the devwebapp pod reports that is running and ready (2/2).

Display the secrets written to the file /vault/secrets/secret-credentials.txt on the devwebapp pod.

$ kubectl exec --stdin=true --tty=true devwebapp --container=devwebapp \
    -- cat /vault/secrets/credentials.txt

Example output:

data: map[password:salsa username:giraffe]
metadata: map[created_time:2020-12-11T19:14:05.170436863Z deletion_time: destroyed:false version:1]

The result displays the unformatted secret data present on the container.

Format data

A template can be applied to structure this data to meet the needs of the application.

Clean up

Destroy the cluster.

$ az group delete --name learn-vault --yes --no-wait

The cluster is destroyed.

Next steps

You launched Vault in standalone mode with a Helm chart. Learn more about the Vault Helm chart by reading the documentation or exploring the project source code.

To install Vault in high-availability (HA) mode via the Helm chart, refer to the Vault Installation to Google Kubernetes Engine via Helm tutorial or the Vault Installation to Minikube via Helm with Consul tutorial for step-by-step instructions.

The pod you deployed used annotations to inject the secret into the file system. Explore how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.

Vault installation to GKE via Helm

Deploy Vault on Amazon EKS Anywhere