Storage migration tutorial - Consul to integrated storage

This tutorial provides steps to migrate your Vault storage from Consul to Integrated Storage. (NOTE: Integrated Storage requires Vault 1.4 or later.)

Before continuing with this tutorial, be sure to read the Migration checklist first.

Vault storage migration workflow:

• Backup Vault data
• Migrate Vault storage
• Vault Enterprise Replication
  • Recommended approach for Vault Enterprise
• Post-migration health check

Backup Vault data

Before performing the Vault server upgrade, it is recommended to take a snapshot of the Vault data using the Consul Snapshots.

Consul Enterprise users may be using the automated Snapshot Agent to capture snapshot periodically and retained in a specified destination.

Snapshot Vault data

On the Consul server cluster that contains the Vault data to be saved in a snapshot, execute the following command from either a Consul server directly or any system running a Consul client agent connected to the server cluster.

$ consul snapshot save backup.snap
Saved and verified snapshot to index 1394

The snapshot file backup.snap will be present in the current working directory.

Inspecting the Snapshot

The snapshot file is simply a gzip compressed archive. You can perform some operational inspection on the snapshot file via the consul snapshot inspect command and also manually by decompressing the file and examining its contents.

$ consul snapshot inspect backup.snap

ID           2-1394-1515172423763
Size         481887
Index        1394
Term         2
Version      1

This output shows the snapshot ID, size in bytes, plus the snapshot index, term, and version, which can be compared with the server (e.g. via consul info) and is useful to detect corruption.

Migrate Vault storage

If you have a multi-datacenter Vault Enterprise Replication deployments, skip to the Vault Enterprise Replication section.

Perform the migration step on one of the nodes in the cluster which will become the leader node.

To walkthrough the migration steps, assume that the following is your new Vault server configuration.

# Storage configuration
storage "raft" {
  path = "/vault/raft/"
  node_id = "node_1"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

api_addr = "https://13.57.14.206:8200"
cluster_addr = "https://10.0.101.22:8201"
disable_mlock = true
ui=true

Notice that the path is set to /vault/raft/ and node_id is set to node_1. (Refer to the server configuration documentation for details.)

1. Create a migration configuration file (e.g. migrate.hcl).

   storage_source "consul" {
   address = "127.0.0.1:8500"
   path    = "vault"
   }
   
   storage_destination "raft" {
     path = "/vault/raft/"
     node_id = "node_1"
   }
   
   cluster_addr = "https://10.0.101.22:8201"
   

   Copy

   The storage_source stanza should be the current storage backend (consul) configuration, and the storage_destination points to the Integrated Storage (raft) configuration.

   The path and node_id must match the values you set in the server configuration file.

   Note

   The /vault/raft/ path must exist on the host machine. The migration command will not create the folder for you.

2. Execute the vault operator command to perform the migration.

   $ vault operator migrate -config=migrate.hcl
   

   Copy

   Refer to the Vault command documentation on operator migrate for more information.

3. You can start the Vault server using the new server configuration pointing to the raft storage and unseal.

   At this point, there is only one raft cluster member.

   $ vault operator raft list-peers
   
   Node       Address                    State     Voter
   ----       -------                    -----     -----
   node_1     https://10.0.101.22:8201   leader    true
   

   Copy

4. Start the remaining Vault nodes in the cluster and add each node to the cluster using the vault operator raft join command. If retry_join is configured, there is no need to invoke the raft join command. The follower nodes should be added to the cluster automatically.

   $ vault operator raft join https://13.57.14.206:8200
   

   Copy

   While https://13.57.14.206:8200 is the leader node's api_addr.

   If you are not familiar with how the HA cluster with Integrated Storage works, read the Vault HA Cluster with Integrated Storage tutorial to familiarize yourself with the Integrated Storage.

Vault Enterprise Replications

If you have multi-datacenter Vault Enterprise Replication deployments such as the diagram below, read the recommendation in this section.

Recommended approach for Vault Enterprise

1. Stop the DR secondary cluster (Cluster 4 in the diagram above as an example).

2. Create a new Vault cluster configured with Integrated Storage (Cluster 5 in the diagram below) and add it as a new DR secondary.

   

3. As some workload comes through, monitor that the DR secondary cluster is catching up with its primary cluster (Cluster 2) which is still using Consul as the storage backend. If there is no issue, you can stop and terminate the old secondary cluster (Cluster 4) at this point.

4. Stop the primary cluster (Cluster 2) and promote the DR secondary (Cluster 5) to be the new primary.

   If the cluster is a performance secondary, check to make sure that it is syncing up with its performance primary (Cluster 1).

5. Repeat the step by creating a new cluster configured with Integrated Storage (Cluster 6) and add it as a DR secondary to the new primary (Cluster 5).

   

   Monitor that it syncs up with its primary as some workload comes through. Finally, you can terminate the old primary cluster (Cluster 2).

Repeat the same workflow to migrate the Vault data to Integrated Storage on the performance primary (Cluster 1) and its DR secondary (Cluster 3).

Post-migration health check

Once the storage is migrated, validate that the cluster is healthy and check logs for any unusual errors that are related to the cluster health. Refer to the following guides:

• Troubleshooting Vault - Vault Logs
• If you have Vault Enterprise Replication environment, refer to the Monitoring Vault Replication

Help and Reference

• Vault command documentation on operator migrate
• Integrated Storage documentation
• Integrated Storage Concepts
• Upgrading Vault guide
• Inspect Data in Integrated Storage