Manage codified Vault on HCP Vault Dedicated with Terraform

Challenge

In the first part of this lab, you learned how to migrate a codified configuration for Vault Community Edition to HCP Vault Dedicated.

Migrating to the cloud, however, is only part of the challenge for organizations. Once you have migrated from self-hosted Vault to Vault Dedicated, there are still operational tasks that need to be managed.

For example, once you have migrated your self-hosted instance of Vault to HCP, you will eventually need to scale Vault Dedicated or setup replication to meet increased demand in other regions.

Before Vault Dedicated this might require weeks, if not months of planning to ensure applications that relied on Vault still had access to the necessary Vault resources during any operational tasks.

Solution

By following the principles covered in Deploy HCP Vault Dedicated with Terraform, you can use Terraform to manage manage the lifecycle of your Vault Dedicated cluster - from initial creation, scaling up to a larger tier and size of the cluster, adding performance replication, and scaling the cluster down to a smaller tier.

One of the many benefits of Vault Dedicated is you can accomplish this with no downtime by making just a few changes to your existing Terraform configuration.

Scenario introduction

After successfully migrating your self-hosted cluster to HCP, your organization wants to increase its utilization of Vault. Based on your current application load, it is necessary to scale your Vault Dedicated cluster to multi-region configuration during peak business hours. After peak business hours you want to scale down your cluster to manage cost.

In this lab you will:

1. Use your existing Terraform configuration to scale your cluster from the dev tier to the plus tier to support increased Vault Dedicated utilization.

2. After scaling your Vault Dedicated cluster, you will add performance replication in a new HCP region to support workloads in multiple locations.

3. Once performance replication has been set up, you will delete the replica and scale the cluster back down to a lower tier.

   

Prerequisites

• A macOS or Linux development host from which you perform most of the tasks; (this lab was lasted tested on macOS 11.6.5)

• Vault CLI installed.

• Terraform CLI installed.

• Tree CLI installed.

• Learn lab manage codified HCP Vault Dedicated repository.

• Existing Vault Dedicated cluster based on this sample Terraform configuration.

Personas

This scenario involves 1 persona:

• admin persona runs Vault, applies configuration with Terraform, edits configuration and environment, and applies configuration to Vault Dedicated.

HCP requirements

• An HCP account where you have deployed a Vault Dedicated development tier cluster. You should use the sample Terraform configuration to deploy the Vault cluster to follow this lab.

• A service principal account assigned the admin role in the HCP Portal.

Prepare scenario environment

You can find the Terraform configuration that you will use for this scenario in the learn-manage-codified-hcp-vault- terraform GitHub repository.

1. Clone the repository.

   $ git clone https://github.com/hashicorp/learn-manage-codified-hcp-vault-terraform.git
   

   Copy

2. Change into the repository directory and examine the contents.

   $ cd learn-manage-codified-hcp-vault-terraform/
   

   Copy

Deploy HCP Vault Dedicated cluster

To follow this lab you must have a Vault Dedicated dev tier cluster available using the provided sample Terraform configuration.

Please expand and complete each section below if you do not already have an Vault Dedicated cluster available.

Add HCP Vault Dedicated environment variables

Setup your local environment to authenticate with Vault Dedicated. By using the example Terraform configuration, you can get the required values from the Terraform output.

1. Export VAULT_ADDR environment variable for the Vault Dedicated public URL.

   $ export VAULT_ADDR=$(terraform output -json | jq -r '.primary_vault_public_endpoint_url.value')
   

   Copy

2. Export VAULT_TOKEN environment variable to contain the initial admin token.

   $ export VAULT_TOKEN=$(terraform output -json | jq -r '.primary_token.value')
   

   Copy

3. Export VAULT_NAMESPACE environment variable to contain the top level Vault Dedicated namespace.

   $ export VAULT_NAMESPACE=$(terraform output -json | jq -r '.namespace.value')
   

   Copy

4. Check the Vault server status.

   $ vault status
   

   Copy

   Example output:

   Key                      Value
   ---                      -----
   Recovery Seal Type       shamir
   Initialized              true
   Sealed                   false
   Total Recovery Shares    1
   Threshold                1
   Version                  1.10.3+ent
   Storage Type             raft
   Cluster Name             vault-cluster-2c82cb91
   Cluster ID               16b68eeb-4a25-3523-ae0a-4d9fc6eb448c
   HA Enabled               true
   HA Cluster               https://172.25.17.200:8201
   HA Mode                  active
   Active Since             2022-06-06T14:51:28.184879441Z
   Raft Committed Index     5381
   Raft Applied Index       5381
   Last WAL                 1046
   

5. Validate that your admin token is working to provide correct access to Vault.

   $ vault token lookup | grep policies
   policies            [default hcp-root]
   

   Copy

You are ready to proceed with managing your Vault Dedicated cluster.

If you deployed a new Vault Dedicated cluster and did not follow the migration lab, apply a sample Vault configuration.

Review Terraform configuration

1. Examine the current Terraform configuration.

   $ tree
   .
   ├── README.md
   ├── hcpvault.tf
   ├── main.tf
   ├── outputs.tf
   ├── part1-config
   │   ├── README.md
   │   └── config
   │       ├── acl-policies.tf
   │       ├── auth-methods.tf
   │       ├── main.tf
   │       ├── policies
   │       │   ├── admin-policy.hcl
   │       │   └── student-secrets.hcl
   │       ├── secrets-engines.tf
   │       ├── static-secrets.tf
   │       ├── terraform.tfstate
   │       └── variables.tf
   ├── terraform.tfstate
   ├── variables.tf
   
   3 directories, 16 files
   

   Copy

   This is a sample Terraform configuration that can be used to deploy an Vault Dedicated cluster.

   Note

   The configuration included in the part1-config directory was supplied for those who did not attend the Migrating codified self-hosted Vault to HCP Vault Dedicated portion of this lab and want to be able to follow this lab step-by-step.

2. Examine the main Terraform configuration.

   $ cat main.tf
   

   Copy

   This file defines the HCP provider and the version of the provider that should be used.

   main.tf

   terraform {
     required_providers {
       hcp = {
         source = "hashicorp/hcp"
         version = ">=0.30.0"
       }
     }
   }
   
   provider "hcp" {
     # Configuration options
   }
   
   provider "time" {
     # Configuration options
   }
   

3. Examine the variables used to define your Vault Dedicated cluster.

   $ cat variables.tf
   

   Copy

   The variables.tf file is used by other Terraform configurations to gather required inputs for different resources. This is useful because you can reference these values where needed, without having to account for them multiple times in different resources or configuration files.

   For example, you can define the tier for your Vault Dedicated cluster and when you want to scale the cluster up to another size, you can make the change in one place.

   variables.tf

   variable "cloud_provider" {
     description = "The cloud provider of the HCP HVN and Vault cluster."
     type        = string
     default     = "aws"
   }
   
   variable "tier" {
     description = "Tier of the Vault Dedicated cluster."
     type        = string
     default     = "dev"
   }
   
   variable "primary_region" {
     description = "The region of the primary cluster HCP HVN and Vault cluster."
     type        = string
     default     = "us-east-1"
   }
   
   variable "primary_cluster_hvn" {
     description = "The ID of the HCP HVN."
     type        = string
     default     = "hvn-aws-us-east-1"
   }
   
   variable "primary_cluster_hvn_cidr" {
     description = "The CIDR range of the HCP HVN."
     type        = string
     default     = "172.25.16.0/20"
   }
   
   variable "primary_cluster_id" {
     description = "The ID of the Vault Dedicated cluster."
     type        = string
     default     = "vault-cluster-primary"
   }
   

4. Examine the Vault Dedicated configuration.

   $ cat hcpvault.tf
   

   Copy

   There are 2 resources required to create an Vault Dedicated cluster - a HashiCorp Virtual Network (HVN), represented in the hcp_hvn resource, and the Vault cluster represented in the hcp_vault_cluster resource.

   The region you create your HVN in will determine where your Vault cluster is created. Refer to the variables.tf file to review the values that will be used.

   The time_sleep and hcp_vault_cluster_admin_token resources are not required, but provided here as an example for how you might retrieve the initial admin token after provisioning a new Vault Dedicated cluster.

   hcpvault.tf

   resource "hcp_hvn" "primary_cluster_hvn" {
     hvn_id         = var.primary_cluster_hvn
     cloud_provider = var.cloud_provider
     region         = var.primary_region
     cidr_block     = var.primary_cluster_hvn_cidr
   }
   
   resource "hcp_vault_cluster" "primary_cluster" {
     hvn_id     = hcp_hvn.primary_cluster_hvn.hvn_id
     cluster_id = var.primary_cluster_id
     tier       = var.tier
     public_endpoint = true
   }
   
   resource "time_sleep" "wait_30_primary" {
     depends_on = [hcp_vault_cluster.primary_cluster]
     create_duration = "30s"
   }
   
   resource "hcp_vault_cluster_admin_token" "primary_cluster_token" {
     cluster_id      = var.primary_cluster_id
     depends_on      = [time_sleep.wait_30_primary]
   }
   
   

5. Examine the Terraform outputs.

   $ cat outputs.tf
   

   Copy

   Outputs are a useful way to collect and use information from one Terraform configuration and use it in another configuration or to use the information as a reference.

   In this example, you may need to know the public URL of your Vault cluster to manually access it or to have another Terraform configuration applied after it is available.

   outputs.tf

   output "primary_vault_public_endpoint_url" {
     value       = hcp_vault_cluster.primary_cluster.vault_public_endpoint_url
     description = "The public IP address of the cluster."
   }
   
   output "namespace" {
     value       = hcp_vault_cluster.primary_cluster.namespace
     description = "The default namespace of the cluster."
   }
   
   output "primary_token" {
     value       = hcp_vault_cluster_admin_token.primary_cluster_token.token
     description = "Token"
     sensitive = true
   }
   

Review HCP Vault Dedicated deployment

Because you deployed your Vault Dedicated cluster using Terraform, you can easily review details about the deployment with the show command.

1. Run terraform show.

   $ terraform show
   

   Copy

   Example output:

   # hcp_hvn.primary_cluster_hvn:
   resource "hcp_hvn" "primary_cluster_hvn" {
       cidr_block          = "172.25.16.0/20"
       cloud_provider      = "aws"
       created_at          = "2022-06-03T17:10:25.000Z"
       hvn_id              = "hvn-aws-us-east-1"
       id                  = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.network.hvn/hvn-aws-us-east-1"
       organization_id     = "a2c4e6g-88d2-69fb-8cc1-0242ac110014"
       project_id          = "a2c4e6g-8920-4714-ba99-0242ac11000e"
       provider_account_id = "091862471264"
       region              = "us-east-1"
       self_link           = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.network.hvn/hvn-aws-us-east-1"
   }
   
   # hcp_vault_cluster.primary_cluster:
   resource "hcp_vault_cluster" "primary_cluster" {
       cloud_provider             = "aws"
       cluster_id                 = "vault-cluster-primary"
       created_at                 = "2022-06-03T17:11:06.097Z"
       hvn_id                     = "hvn-aws-us-east-1"
       id                         = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary"
       namespace                  = "admin"
       organization_id            = "a2c4e6g-88d2-69fb-8cc1-0242ac110014"
       project_id                 = "a2c4e6g-8920-4714-ba99-0242ac11000e"
       public_endpoint            = true
       region                     = "us-east-1"
       self_link                  = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary"
       tier                       = "DEV"
       vault_private_endpoint_url = "https://vault-cluster-primary-private-vault-ec951ba3.98ebdb39.z1.hashicorp.cloud:8200"
       vault_public_endpoint_url  = "https://vault-cluster-primary-public-vault-ec951ba3.98ebdb39.z1.hashicorp.cloud:8200"
       vault_version              = "v1.10.3"
   }
   
   # hcp_vault_cluster_admin_token.primary_cluster_token:
   resource "hcp_vault_cluster_admin_token" "primary_cluster_token" {
       cluster_id = "vault-cluster-primary"
       created_at = "2022-06-03T13:17:54-04:00"
       id         = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary/token"
       token      = (sensitive value)
   }
   
   # time_sleep.wait_30_seconds:
   resource "time_sleep" "wait_30_seconds" {
       create_duration = "30s"
       id              = "2022-06-03T17:17:52Z"
   }
   
   Outputs:
   
   namespace = "admin"
   token = (sensitive value)
   vault_public_endpoint_url = "https://vault-cluster-primary-public-vault-ec951ba3.98ebdb39.z1.hashicorp.cloud:8200"
   

   Observe that the cluster tier is currently DEV.

Scale up the HCP Vault Dedicated cluster

After migrating your self-hosted Vault cluster to HCP, you need to increase the capacity to support additional applications accessing Vault.

If you needed to scale up a self hosted cluster on your own, you might need to spend weeks or even months planning. For example you might need to validate your on-premises compute resources have enough resources to support larger Vault instances or verify the configuration has a valid CPU and memory configuration that is supported by the underlying infrastructure.

With Vault Dedicated, you can scale your Vault cluster with a change to your variables.tf file and re-applying the configuration. The underlying infrastructure and associated managed of that infrastructure is handled by HashiCorp.

1. Change the tier from dev to plus_small in the variables.tf file.

   $ sed -ibak "s/dev/plus_small/g" variables.tf
   

   Copy

   Note

   Once you scale a cluster is scaled above the dev tier, it cannot be scaled back down to the dev tier.

2. Verify the file file has been updated.

   $ cat variables.tf | grep plus_small
   
   default     = "plus_small"
   

   Copy

3. Re-apply the configuration to change the tier.

   $ terraform apply
   

   Copy

   Example output:

   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     ~ update in-place
   
   Terraform will perform the following actions:
   
     # hcp_vault_cluster.primary_cluster will be updated in-place
     ~ resource "hcp_vault_cluster" "primary_cluster" {
           id                         = "/project/a2c4e6g-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary"
         ~ tier                       = "DEV" -> "plus_small"
           # (14 unchanged attributes hidden)
       }
   
   Plan: 0 to add, 1 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   

   The example output shows that the cluster will be updated in-place. The cluster does not need to be replaced or stopped to change the tier or size of the cluster.

4. Enter yes at the prompt to proceed with scaling the cluster.

Access the HCP Vault Dedicated cluster

One of the many benefits of running your Vault cluster on HCP is most of the infrastructure related maintenance is managed by HashiCorp. While your cluster is scaling, applications that rely on Vault can still access Vault resources.

1. Log into the HCP Portal and navigate to the Vault clusters list while Terraform is upgrading the cluster.

2. Observe that your cluster is an Updating state.

3. Click vault-cluster-primary to access details about the cluster.

4. Information about the cluster is still available while the upgrade is in process.

5. Click the Public link under Cluster URLs to copy the URL.

6. Open a new browser tab and navigate to the copied URL. Vault is still accessible even during the upgrade.

7. Click the Method pull down menu and select Username.

8. Log in as the admin user that was migrated from the self-hosted Vault instance to Vault Dedicated with the password superS3cret!.

9. Navigate to Secrets and click api-credentials.

Even though the cluster is in the process of being upgraded, Vault is still available and accessible for incoming requests.

Add performance replication

As the Vault admin, you have successfully scaled your cluster up from the dev tier to the plus tier to support increased Vault utilization. To meet the demands of your applications, you now need to enable performance replication in a different region.

Adding performance replication to an existing cluster involves just a few steps and can be performed on-demand with no downtime. Similar to creating a new Vault Dedicated cluster using Terraform, you will need to define the parameters for your new cluster and associate the new cluster with your primary cluster.

1. Add the required values for the new replica cluster to the variables.tf file.

   $ cat >> variables.tf <<EOF
   
   variable "secondary_region" {
     description = "The region of the secondary cluster HCP HVN and Vault cluster."
     type        = string
     default     = "us-west-2"
   }
   
   variable "secondary_cluster_hvn" {
     description = "The ID of the HCP HVN."
     type        = string
     default     = "hvn-aws-us-west-2"
   }
   
   variable "secondary_cluster_hvn_cidr" {
     description = "The CIDR range of the HCP HVN."
     type        = string
     default     = "172.24.16.0/20"
   }
   
   variable "secondary_cluster_id" {
     description = "The ID of the Vault Dedicated cluster."
     type        = string
     default     = "vault-cluster-secondary"
   }
   EOF
   

   Copy

2. Review the variables.tf file.

   $ cat variables.tf
   

   Copy

   The variables.tf file now contains the necessary information for the new cluster.

   variables.tf

   variable "cloud_provider" {
     description = "The cloud provider of the HCP HVN and Vault cluster."
     type        = string
     default     = "aws"
   }
   
   variable "tier" {
     description = "Tier of the Vault Dedicated cluster."
     type        = string
     default     = "dev"
   }
   
   variable "primary_region" {
     description = "The region of the primary cluster HCP HVN and Vault cluster."
     type        = string
     default     = "us-east-1"
   }
   
   variable "primary_cluster_hvn" {
     description = "The ID of the HCP HVN."
     type        = string
     default     = "hvn-aws-us-east-1"
   }
   
   variable "primary_cluster_hvn_cidr" {
     description = "The CIDR range of the HCP HVN."
     type        = string
     default     = "172.25.16.0/20"
   }
   
   variable "primary_cluster_id" {
     description = "The ID of the Vault Dedicated cluster."
     type        = string
     default     = "vault-cluster-primary"
   }
   
   variable "secondary_cluster_hvn" {
     description = "The ID of the HCP HVN."
     type        = string
     default     = "hvn-aws-us-west-2"
   }
   
   variable "secondary_cluster_hvn_cidr" {
     description = "The CIDR range of the HCP HVN."
     type        = string
     default     = "172.24.16.0/20"
   }
   
   variable "secondary_cluster_id" {
     description = "The ID of the Vault Dedicated cluster."
     type        = string
     default     = "vault-cluster-secondary"
   }
   
   variable "secondary_region" {
     description = "The region of the secondary cluster HCP HVN and Vault cluster."
     type        = string
     default     = "us-west-2"
   }
   

3. Update hcpvault.tf to add the required resource blocks for the new cluster.

   $ cat >> hcpvault.tf <<EOF
   
   resource "time_sleep" "wait_30_secondary" {
     depends_on = [hcp_vault_cluster.secondary_cluster]
     create_duration = "30s"
   }
   
   resource "hcp_hvn" "secondary_cluster_hvn" {
     hvn_id         = var.secondary_cluster_hvn
     cloud_provider = var.cloud_provider
     region         = var.secondary_region
     cidr_block     = var.secondary_cluster_hvn_cidr
   }
   
   resource "hcp_vault_cluster" "secondary_cluster" {
     hvn_id          = hcp_hvn.secondary_cluster_hvn.hvn_id
     cluster_id      = var.secondary_cluster_id
     tier            = var.tier
     primary_link    = hcp_vault_cluster.primary_cluster.self_link
     public_endpoint = true
   }
   
   resource "hcp_vault_cluster_admin_token" "secondary_cluster_token" {
     cluster_id      = var.secondary_cluster_id
     depends_on      = [time_sleep.wait_30_secondary]
   }
   EOF
   

   Copy

4. Review the hcpvault.tf file.

   $ cat hcpvault.tf
   

   Copy

   The Terraform configuration for Vault Dedicated now includes the resources to create the new cluster. Performance replication is enabled by using the primary_link parameter to associate the new cluster with the primary cluster.

   hcpvault.tf

   resource "time_sleep" "wait_30_seconds" {
     depends_on = [hcp_vault_cluster.primary_cluster]
     create_duration = "30s"
   }
   
   resource "hcp_hvn" "primary_cluster_hvn" {
     hvn_id         = var.primary_cluster_hvn
     cloud_provider = var.cloud_provider
     region         = var.primary_region
     cidr_block     = var.primary_cluster_hvn_cidr
   }
   
   resource "hcp_vault_cluster" "primary_cluster" {
     hvn_id     = hcp_hvn.primary_cluster_hvn.hvn_id
     cluster_id = var.primary_cluster_id
     tier       = var.tier
     public_endpoint = true
   }
   
   resource "hcp_vault_cluster_admin_token" "primary_cluster_token" {
     cluster_id      = var.primary_cluster_id
     depends_on      = [time_sleep.wait_30_seconds]
   }
   
   resource "hcp_hvn" "secondary_cluster_hvn" {
     hvn_id         = var.secondary_cluster_hvn
     cloud_provider = var.cloud_provider
     region         = var.secondary_region
     cidr_block     = var.secondary_cluster_hvn_cidr
   }
   
   resource "hcp_vault_cluster" "secondary_cluster" {
     hvn_id          = hcp_hvn.secondary_cluster_hvn.hvn_id
     cluster_id      = var.secondary_cluster_id
     tier            = var.tier
     primary_link    = hcp_vault_cluster.primary_cluster.self_link
     public_endpoint = true
   }
   
   resource "hcp_vault_cluster_admin_token" "secondary_cluster_token" {
     cluster_id      = var.secondary_cluster_id
     depends_on      = [time_sleep.wait_30_seconds]
   }
   

5. Add an output for the new clusters URL.

   $ cat >> outputs.tf <<EOF
   
   output "secondary_vault_public_endpoint_url" {
     value       = hcp_vault_cluster.secondary_cluster.vault_public_endpoint_url
     description = "The public IP address of the secondary cluster."
   }
   EOF
   

   Copy

6. Create the new cluster by re-running terraform apply.

   terraform apply -auto-approve
   

   Copy

   Example output:

   hcp_hvn.primary_cluster_hvn: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.network.hvn/hvn-aws-us-east-1]
   hcp_vault_cluster.primary_cluster: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary]
   time_sleep.wait_30_primary: Refreshing state... [id=2022-06-07T17:11:47Z]
   hcp_vault_cluster_admin_token.primary_cluster_token: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary/token]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     + create
   
   Terraform will perform the following actions:
   
     # hcp_hvn.secondary_cluster_hvn will be created
     + resource "hcp_hvn" "secondary_cluster_hvn" {
   
   ...snip...
   
   hcp_vault_cluster.secondary_cluster: Still creating... [17m1s elapsed]
   hcp_vault_cluster.secondary_cluster: Creation complete after 17m8s [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-secondary]
   time_sleep.wait_30_secondary: Creating...
   time_sleep.wait_30_secondary: Still creating... [10s elapsed]
   time_sleep.wait_30_secondary: Still creating... [20s elapsed]
   time_sleep.wait_30_secondary: Creation complete after 30s [id=2022-06-07T21:37:16Z]
   hcp_vault_cluster_admin_token.secondary_cluster_token: Creating...
   hcp_vault_cluster_admin_token.secondary_cluster_token: Creation complete after 3s [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-secondary/token]
   
   Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   namespace = "admin"
   primary_token = <sensitive>
   primary_vault_public_endpoint_url = "https://vault-cluster-primary-public-vault-f0694e72.5f8f89a2.z1.hashicorp.cloud:8200"
   secondary_vault_public_endpoint_url = "https://vault-cluster-secondary-public-vault-d9915240.da3e83c2.z1.hashicorp.cloud:8200"
   
   

7. Log into the HCP Portal and navigate to the Vault clusters list. Both the primary and secondary cluster are available.

8. Unset the VAULT_TOKEN environment variable used to authenticate with the primary cluster.

   $ unset VAULT_TOKEN
   

   Copy

9. Log into the secondary cluster using the Vault CLI.

   $ vault login \
      -address=$(terraform output -json | jq -r '.secondary_vault_public_endpoint_url.value') \
      -method=userpass \
      username=admin \
      password=superS3cret!
   

   Copy

10. Verify the status of the secondary Vault cluster.

    $ vault status \
       -address=$(terraform output -json | jq -r '.secondary_vault_public_endpoint_url.value')
    

    Copy

    Example output:

    Key                      Value
    ---                      -----
    Recovery Seal Type       shamir
    Initialized              true
    Sealed                   false
    Total Recovery Shares    1
    Threshold                1
    Version                  1.10.3+ent
    Storage Type             raft
    Cluster Name             vault-cluster-e0356e94
    Cluster ID               901c4f34-df29-ea41-1c70-ffe86d492a77
    HA Enabled               true
    HA Cluster               https://172.24.25.116:8201
    HA Mode                  active
    Active Since             2022-06-07T21:28:44.120674828Z
    Raft Committed Index     11356
    Raft Applied Index       11356
    Last WAL                 3531
    

    You can verify this is the secondary cluster because the IP address is part of the address space defined for the secondary cluster HVN.

11. View available secrets engines.

    $ vault secrets list \
       -address=$(terraform output -json | jq -r '.secondary_vault_public_endpoint_url.value')
    

    Copy

    Example output:

    Path                Type            Accessor                 Description
    ----                ----            --------                 -----------
    api-credentials/    kv              kv_e80f580b              n/a
    cubbyhole/          ns_cubbyhole    ns_cubbyhole_ebc1e938    per-token private secret storage
    identity/           ns_identity     ns_identity_647c3433     identity store
    sys/                ns_system       ns_system_b24fbdca       system endpoints used for control, policy and debugging
    transit/            transit         transit_d83dca00         n/a
    

    All secrets engines were replicated from the primary cluster.

Add a path filter

Now that you have deployed performance replication, you have been informed that the transit secrets engine should not have been replicated to the secondary cluster.

You can solve for this by adding a path filter that denies a specific path from being replicated to the secondary cluster.

1. Add the paths_filter parameter to the secondary clusters resource block in hcpvault.tf.

   $ sed -ibak '/primary_link/a\
     paths_filter    = ["transit"]
    ' hcpvault.tf
   

   Copy

2. Re-apply the Terraform configuration to prevent the transit secrets engine from being replicated to the secondary cluster.

   $ terraform apply -auto-approve
   

   Copy

3. Verify that the transit secrets engine is no longer available on the secondary cluster.

   $ vault secrets list \
      -address=$(terraform output -json | jq -r '.secondary_vault_public_endpoint_url.value')
   

   Copy

   Example output:

   Path                Type            Accessor                 Description
   ----                ----            --------                 -----------
   api-credentials/    kv              kv_e80f580b              n/a
   cubbyhole/          ns_cubbyhole    ns_cubbyhole_ebc1e938    per-token private secret storage
   identity/           ns_identity     ns_identity_647c3433     identity store
   sys/                ns_system       ns_system_b24fbdca       system endpoints used for control, policy and debugging
   

   The transit secrets engine is no longer listed.

Disable performance replication and scale down the cluster

You have successfully scaled your Vault Dedicated cluster from the dev tier to the plus tier and enabled performance replication. During peak business hours your applications have the necessary access to Vault Dedicated.

To help manage cost, you have been asked to scale down your cluster and remove the secondary cluster during non-peak hours.

Removing performance replication requires removing the resources block for the secondary cluster. Similarly, scaling the cluster down to a smaller tier requires changing the tier value in variables.tf.

1. Remove the secondary cluster resources from hcpvault.tf.

   $ sed -ibak '24,$ d' hcpvault.tf
   

   Copy

2. Remove the secondary cluster URL from outputs.tf

   $ sed -ibak '16,$ d' outputs.tf
   

   Copy

3. Change the tier from plus_small to starter_small.

   $ sed -ibak "s/plus_small/starter_small/g" variables.tf
   

   Copy

4. Re-apply the Terraform configuration.

   $ terraform apply -auto-approve
   

   Copy

   Example output:

   time_sleep.wait_30_secondary: Refreshing state... [id=2022-06-08T18:23:13Z]
   hcp_vault_cluster_admin_token.secondary_cluster_token: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-secondary/token]
   hcp_hvn.secondary_cluster_hvn: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.network.hvn/hvn-aws-us-west-2]
   hcp_hvn.primary_cluster_hvn: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.network.hvn/hvn-aws-us-east-1]
   hcp_vault_cluster.secondary_cluster: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-secondary]
   hcp_vault_cluster.primary_cluster: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary]
   time_sleep.wait_30_primary: Refreshing state... [id=2022-06-08T13:54:09Z]
   hcp_vault_cluster_admin_token.primary_cluster_token: Refreshing state... [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary/token]
   
   Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
     ~ update in-place
     - destroy
   
   Terraform will perform the following actions:
   
     # hcp_hvn.secondary_cluster_hvn will be destroyed
     # (because hcp_hvn.secondary_cluster_hvn is not in configuration)
   
   ...snip...
   
   hcp_vault_cluster.primary_cluster: Still modifying... [id=/project/11eb3a47-8920-4714-ba99-0242ac...rp.vault.cluster/vault-cluster-primary, 15m21s elapsed]
   hcp_vault_cluster.primary_cluster: Modifications complete after 15m27s [id=/project/11eb3a47-8920-4714-ba99-0242ac11000e/hashicorp.vault.cluster/vault-cluster-primary]
   
   Apply complete! Resources: 0 added, 1 changed, 4 destroyed.
   
   Outputs:
   
   namespace = "admin"
   primary_token = <sensitive>
   primary_vault_public_endpoint_url = "https://vault-cluster-primary-public-vault-d7a8ff49.edd3051f.z1.hashicorp.cloud:8200"
   

5. Log into the HCP Portal and navigate to the Vault clusters list. The secondary cluster has been removed and the cluster scaled down from plus_small to starter_small.

One more thing...

Before we wrap up - did anyone notice an operational and security benefit when enabling performance replication and path filters for Vault Dedicated?

Summary

By using simple, modular, and composable infrastructure as code with Terraform, you can easily manage the lifecycle of your Vault Dedicated clusters.

Clean up

Here are the steps you need to clean up the scenario content from your local environment.

1. Delete the Vault Dedicated cluster instance and HVN.

   $ terraform destroy -auto-approve
   

   Copy

2. Unset the environment variables.

   $ unset VAULT_ADDR VAULT_TOKEN LEARN_VAULT_PID VAULT_NAMESPACE
   

   Copy

Resources

• Codify Management of Vault Using Terraform
• Migration Strategies and Considerations Guidelines
• Terraform S3 State Storage
• Terraform Vault Provider
• Terraform HCP Provider
• Deploy HCP Vault Dedicated with Terraform
• Create a Vault Cluster on HCP