Terraform Enterprise Automated Recovery

This guide explains how to configure automated recovery for a Terraform Enterprise installation. The goal is to provide a short Mean-Time-To-Recovery (MTTR) in the event of an outage. There are two steps in the automated recovery process:

Configure snapshots on your current install to backup data
Provision a new Terraform Enterprise instance using the latest snapshot

This guide will walk through both of these steps.

Configure snapshots

Snapshots are taken on the Terraform Enterprise instance. That instance can have two types of data on it:

Terraform Enterprise application data: The core product data such as run history, configuration history, state history. This data changes frequently.
Terraform Enterprise installer data: The data used to configure Terraform Enterprise itself such as installation type, database connection settings, hostname. This data rarely changes.

In the Mounted Disk and External Services operational modes, only installer data is stored on the instance. Application data is stored in the mounted disk or in an external PostgreSQL instance.

Automated snapshots are more effective when using Mounted Disk or External Services as the amount of backed up data is smaller and less risky.

You can configure snapshots in the dashboard under Console Settings, in the Snapshot & Restore section. We recommend selecting Enable Automatic Scheduled Snapshots. The interval depends on the operational mode, but we recommend Daily for Mounted Disk or External Services because the snapshots contain only configuration data, not application data.

Restore a snapshot in a new Terraform Enterprise instance

Version Checking

Note: replicatedctl is located at /usr/local/bin/replicatedctl. On some operating systems, /usr/local/bin is not in the user's $PATH. In these cases, either add /usr/local/bin to the path or refer to replicatedctl with the full path.

Using the restore mechanism requires Replicated version 2.17.0 or greater. You can check the version using replicatedctl version.

Script

Below are examples of restore scripts that are run on machine boot. There are many mechanisms that can run a script on boot (cloud-init, systemd, /etc/init.d). Which one is used is up to the user.

This script is presented as an example. Anyone using it needs to understand what it's doing and is free to modify it to meet any additional needs they have.

S3

This example uses S3 as the mechanism to store snapshots on a debian-based system.

#!/bin/bash

set -e -u -o pipefail

bucket=your_bucket_to_store_snapshots
region=region_of_the_bucket
access_key=aws_access_key_id
secret_key=aws_secret_access_key

access="--store s3 --s3-bucket $bucket --s3-region $region --s3-key-id $access_key --s3-secret-key $secret_key"

# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.

apt-get update
apt-get install -y jq

# Run the installer.

curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts

# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
  sleep 1
  echo "Replicated is not yet ready."
done
echo "Replicated is ready."

# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json

# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)

# If there are no snapshots available, exit out
if test "$id" = ""; then
  echo "No snapshots found"
  exit 1
fi

echo "Restoring snapshot: $id"

# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"

# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
  sleep 1
done

echo
echo "Application booted!"

SFTP

This example uses sftp to store the snapshots.

#!/bin/bash

set -e -u -o pipefail

key_file=path_to_your_ssh_key
key="$(base64 -w0 "$key_file")"
host=sftp_server_hostname_or_ip
user=user_to_sftp_on_the_remote_server

access="--store sftp --sftp-host $host --sftp-user $user --sftp-key $key"

# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.

apt-get update
apt-get install -y jq

# Run the installer.

curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts

# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
  sleep 1
  echo "Replicated is not yet ready."
done
echo "Replicated is ready."

# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json

# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)

# If there are no snapshots available, exit out
if test "$id" = ""; then
  echo "No snapshots found"
  exit 1
fi

echo "Restoring snapshot: $id"

# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"

# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
  sleep 1
done

echo
echo "Application booted!"

Local directory

This example uses a local directory to store the snapshots. If this is intended to be run on a brand new system, the local directory would be either mounted block device or a network filesystem like NFS or CIFS.

#!/bin/bash

set -e -u -o pipefail

path=absolute_path_to_directory_of_snapshots

access="--store local --path $path"

# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.

apt-get update
apt-get install -y jq

# Run the installer.

curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts

# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
  sleep 1
  echo "Replicated is not yet ready."
done
echo "Replicated is ready."

# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json

# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)

# If there are no snapshots available, exit out
if test "$id" = ""; then
  echo "No snapshots found"
  exit 1
fi

echo "Restoring snapshot: $id"

# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"

# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
  sleep 1
done

echo
echo "Application booted!"

Airgap recovery considerations

The instructions above are tailored for the online install method. When restoring on an airgap instance, there are several additional considerations:

The minimum version of Replicated is 2.31.0, rather than 2.17.0.
The license file and airgap package must be in place on the new instance prior to restore. The restore process expects to find them in the same locations as they were on the original instance.
The snapshot being used must also be from an airgap instance.
The install.sh script and method used must be from the Replicated airgap installer boostrapper, using the process described for airgap installation.