Version remote state with the HCP Terraform API

The Terraform state file is the source of truth for your infrastructure. Protecting and backing up this file is critical for practitioners who use Terraform in production. Remote state storage with HCP Terraform offers fail-safes for your infrastructure in the event of disaster-recovery situations and local file corruption. Although Terraform takes steps to prevent state errors, your state file can get corrupted due to partial apply operations or incorrectly running terraform import or terraform taint. Using the HCP Terraform API, you can safely download, modify, and upload your state file to an HCP Terraform workspace.

In this tutorial, you will generate a state file by deploying an AWS instance with web access using the Terraform CLI. Then, you will download your remote state file and use the Terraform API to create a new state version.

Prerequisites

For this tutorial you will need:

• The Terraform CLI 0.15.0+ installed locally.
• Access to HCP Terraform or Enterprise.
• An AWS account with credentials configured for Terraform.
• The awscli configured.
• jq installed.

Clone the example configuration

Clone the example GitHub repository.

$ git clone https://github.com/hashicorp-education/learn-tfc-state-api

Change into the repository directory.

$ cd learn-tfc-state-api

Open the main.tf file to review the configuration. The main resources in this configuration are an AWS EC2 instance and a security group with port 8080 access.

First, update your configuration with your HCP Terraform information. Update <YOUR-ORGANIZATION-NAME> with your HCP Terraform organization name.

terraform {
  required_version = ">= 1.1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.24.1"
    }
  }
  cloud {
    organization = "<ORGANIZATION_NAME>"
    workspaces {
      name = "Example-Workspace"
    }
  }
}

##...

Save your changes.

Run the terraform login subcommand, and follow the prompts to authenticate to HCP Terraform.

$ terraform login
Terraform will request an API token for app.terraform.io using your browser.

If login is successful, Terraform will store the token in plain text in
the following file for use by subsequent commands:
    /Users/username/.terraform.d/credentials.tfrc.json

Do you want to proceed?
  Only 'yes' will be accepted to confirm.

  Enter a value:

Generate a token using your browser, and copy-paste it into this prompt.

Terraform will store the token in plain text in the following file
for use by subsequent commands:
    /Users/rachel/.terraform.d/credentials.tfrc.json

Token for app.terraform.io:
  Enter a value:

For more detailed instructions on logging in, reference the Authenticate the CLI with HCP Terraform tutorial.

After authenticating, initialize your Terraform configuration.

$ terraform init

Initializing HCP Terraform...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v3.37.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

Add AWS credentials to workspace variables

Navigate to your HCP Terraform state-versioning workspace. Click on "Variables" and add your region variable as a Terraform variable. Add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables as environment variables. Be sure to mark the AWS credentials environment variables as sensitive

Apply your configuration

In your terminal, apply your configuration. Enter yes when prompted to confirm your changes.

$ terraform apply

Running apply in HCP Terraform. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://app.terraform.io/app/hashicorp-learn/state-versioning/runs/run-Lm96BJVNXkRv7dNQ

Waiting for the plan to start...

##...

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

instance_id = "i-0bcc0850e611cfd6b"
public_ip = "3.143.170.167"

Verify that your state file contains your resources using terraform state list command.

$ terraform state list
data.aws_ami.ubuntu
aws_instance.example
aws_security_group.sg_web
aws_security_group_rule.sg_web

In the next section, you will configure your HCP Terraform workspace.

Configure the HCP Terraform workspace

You will need your workspace ID and a new HCP Terraform API token to access your workspace's state file.

Retrieve workspace ID

In HCP Terraform, navigate to your new workspace.

In your state-versioning workspace, navigate to "Settings" then "General" and copy your workspace ID.

Create a workspace ID environment variable in your terminal, replacing <YOUR-WORKSPACE-ID> with the workspace ID you just copied.

$ export WORKSPACE_ID=<YOUR-WORKSPACE-ID>

Retrieve HCP Terraform API token

Create a new user token by clicking the icon for your user in the top right corner, then "User settings", then "Tokens". Select "Create an API token".

Name your token "state-versioning" then select "Create API token".

Copy the API token.

Create an environment variable with your token, replacing <YOUR-TFC-TOKEN> with the token you just copied.

$ export TFC_TOKEN=<YOUR-TFC-TOKEN>

Lock your workspace

Click on the lock icon to lock your workspace. Locking your workspace prevents other operations from running and potentially corrupting the state file you are going to download. Your workspace needs to be locked before you can push a new state file via API. You must lock the workspace as the same user you generated the HCP Terraform token for in the previous step.

Edit configuration tags

Run the AWS CLI to add the Org tag to your EC2 resource. Your state file does not have a record of this value.

$ aws ec2 create-tags --resources $(terraform output -raw instance_id) --tags Key=Org,Value=HashiCorp

It may take a few minutes to update your instance.

Your new Org tag is HashiCorp in AWS while your Terraform state file is not aware of this change.

Later in this tutorial, you will reconcile this difference with the Terraform state.

Download the state file

In your terminal, navigate to the helper_scripts folder.

$ cd helper_scripts

The shell scripts in this directory construct your API queries, download your remote state file for editing, and create a payload for uploading your changes.

Open the getstate.sh file to review the API query.

#!/bin/bash

HTTP_RESPONSE=$(curl \
     --header "Authorization: Bearer "$TFC_TOKEN"" \
     --header "Content-Type: application/vnd.api+json" \
     "https://app.terraform.io/api/v2/workspaces/"$WORKSPACE_ID"/current-state-version" | jq -r '.data | .attributes | ."hosted-state-download-url"')

curl -o state.tfstate $HTTP_RESPONSE

If you are using Terraform Enterprise, change the URL from app.terraform.io to your personalized Terraform Enterprise domain.

This snippet uses your environment variables and authenticates to your HCP Terraform workspace to download the current remote state file. The hosted-state-download-url contains the URL that hosts your remote state file.

In your terminal, run the getstate.sh script.

$ ./getstate.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1995  100  1995    0     0   8711      0 --:--:-- --:--:-- --:--:--  8711
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8941    0  8941    0     0  33486      0 --:--:-- --:--:-- --:--:-- 33486

Verify your query downloaded the state file and open state.tfstate.

$ cat state.tfstate
{
  "version": 4,
  "terraform_version": "0.15.1",
  "serial": 0,
  "lineage": "acd79b18-a1a9-3e66-6325-8686bbc77566",
##...

Modify and create the state payload

Now that you have the remote state downloaded, open the state.tfstate file in your file editor. Increment your serial number by one then save the file.

 {
   "version": 4,
   "terraform_version": "0.15.1",
-  "serial": 0,
+  "serial": 1,
   "lineage": "acd79b18-a1a9-3e66-6325-8686bbc77566",
   "outputs": {
     "instance_id": {
       "value": "i-01b237003b468184d",
       "type": "string"
     },
     "public_ip": {
       "value": "3.19.246.180",
       "type": "string"
     }
   },
## ...
 }

This is your new state version number. Terraform uses the serial to keep track of the changes made in each new state file and uses it to make sure your operations run against the correct known state file in the HCP Terraform workspace. In standard operations, Terraform updates the serial for you automatically. However, since you're pushing a new state version, you need to manually increment this value.

Edit your instance tags to include the new Org tag. Search for terraform-learn-state-versioning in the state.tfstate file. Add a comma at the end of the Name tag. Terraform parses the JSON-formatting state file and adds elements to your resource records.

##...
            "tags": {
-              "Name": "terraform-learn-state-versioning"
+              "Name": "terraform-learn-state-versioning",
+              "Org": "HashiCorp"
            },
            "tenancy": "default",
##...

Save your file.

Now, you will construct your current state payload. Select the tab for your operating system for specific instructions.

#!/bin/bash

serial=$(cat state.tfstate | jq '.serial')
md5_compute=$(md5 -q state.tfstate)
md5=\"$md5_compute\"
lineage=$(cat state.tfstate | jq '.lineage')
base64_encode=$(base64 state.tfstate)
state=\"$base64_encode\"


echo "{
   \"data\": {
   \"type\": \"state-versions\",
     \"attributes\": {
       \"serial\": $serial,
       \"md5\": "$md5",
       \"lineage\": "$lineage",
       \"state\": "$state"
     }
   }
 }" > payload.json

$ ./createpayload.sh

$ cat payload.json
{
   "data": {
   "type": "state-versions",
     "attributes": {
       "serial": 1,
       "md5": "f51e44f5672b40725e283c1bd5556752",
       "lineage": "939c75bf-0872-6277-d273-3df86f7ac679",
       "state": "ewogICJ2ZXJzaW9uIjogNCwKICAidGVyc
## …
}

#!/bin/bash

serial=$(cat state.tfstate | jq '.serial')
md5_compute=$(md5sum state.tfstate | awk '{print $1}')
md5=\"$md5_compute\"
lineage=$(cat state.tfstate | jq '.lineage')
base64_encode=$(base64 state.tfstate)
state=\"$base64_encode\"


echo "{
   \"data\": {
   \"type\": \"state-versions\",
     \"attributes\": {
       \"serial\": $serial,
       \"md5\": "$md5",
       \"lineage\": "$lineage",
       \"state\": "$state"
     }
   }
 }" > payload.json

$ ./linux-createpayload.sh

$ cat payload.json
{
   "data": {
   "type": "state-versions",
     "attributes": {
       "serial": 1,
       "md5": "f51e44f5672b40725e283c1bd5556752",
       "lineage": "939c75bf-0872-6277-d273-3df86f7ac679",
       "state": "ewogICJ2ZXJzaW9uIjogNCwKICAidGVyc
## ...
}

Upload the new state file to HCP Terraform

Now that you have a JSON payload with your encrypted state file, upload the new state file to HCP Terraform.

In your helper_scripts directory, open the uploadstate.sh file.

#!/bin/bash

 HTTP_POST=$(curl \
     --header "Authorization: Bearer "$TFC_TOKEN"" \
     --header "Content-Type: application/vnd.api+json" \
     --request POST \
     --data @payload.json \
     "https://app.terraform.io/api/v2/workspaces/"$WORKSPACE_ID"/state-versions")

 echo $HTTP_POST

This API query uses the --data flag to upload the payload.json file to your workspace.

Run the uploadstate.sh script.

$ ./uploadstate.sh
##...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13247  100  1107  100 12140   1392  15270 --:--:-- --:--:-- --:--:-- 16641
{"data":{"id":"sv-VBU3yeG5XMLgK5K6","type":"state-versions","attributes":{"created-at":"2021-04-08T21:59:29.954Z","size":null,"hosted-state-download-url":"https://archivist.terraform.io/v1/object/dmF1bHQ6djE6ZXFONmlaYlVhcHVNOE9WWENZZkljdmJz..."
##...

In your HCP Terraform workspace, navigate to your "States" tab and select the most recent state.

In the "Changes in this version" section, confirm your new state file contains a new serial number and tag reference.

Unlock your workspace by clicking on the lock icon and confirming the unlock.

Update your configuration

Because you updated your resource outside of the Terraform workflow with an additional tag, you must update the configuration with the updated resource and run a terraform apply to maintain parity with your state file. Terraform will perform the apply, but will not make any resource changes. This is a "no-operation" or "no-op" apply.

Change into the root directory.

$ cd ..

Open the main.tf file and update your instance tag.

 resource "aws_instance" "example" {
   ami                    = data.aws_ami.ubuntu.id
   instance_type          = "t2.micro"
   vpc_security_group_ids = [aws_security_group.sg_web.id]
   user_data              = <<-EOF
              #!/bin/bash
              apt-get update
              apt-get install -y apache2
              sed -i -e 's/80/8080/' /etc/apache2/ports.conf
              echo "Hello World" > /var/www/html/index.html
              systemctl restart apache2
              EOF
   tags = {
     Name = "terraform-learn-state-versioning"
+    Org  = "HashiCorp"
   }
 }

Run terraform apply to consolidate your configuration with your remote state. This is a "no-op" apply.

$ terraform apply

Running apply in HCP Terraform. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://app.terraform.io/app/hashicorp-learn/state-versioning/runs/run-gGFN9Tdd6cGuaqyN

Terraform v0.15.1
on linux_amd64
Configuring remote state backend...
Initializing Terraform configuration...
aws_security_group.sg_web: Refreshing state... [id=sg-08df7f3f965e47a6a]
aws_security_group_rule.sg_web: Refreshing state... [id=sgrule-4136193275]
aws_instance.example: Refreshing state... [id=i-0a8f43386c25bc073]

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and the remote system(s). As a result, there are no actions to
take.

Now that your modified state file matches your infrastructure and configuration, delete the local version of your state file.

$ rm helper_scripts/state.tfstate

Your payload.json file also contains an encrypted version of your state. Delete your payload.json file.

$ rm helper_scripts/payload.json

Destroy your infrastructure

In HCP Terraform, destroy your remote workspace.

Navigate to "Settings" > "Destruction and Deletion".

At the bottom of the page, select "Queue Destroy Plan" and confirm.

Next, delete your workspace from HCP Terraform. At the bottom of the page, select "Delete workspace" and confirm.

Next steps

In this tutorial, you learned how to use the HCP Terraform API to interact with and update your HCP Terraform workspace's state. First, you created infrastructure in HCP Terraform. Then, you downloaded your current HCP Terraform state file and safely modified and versioned your state file. Finally, you uploaded your versioned and updated state file to HCP Terraform.

For more information about the HCP Terraform API or Terraform state, review the following resources:

• HCP Terraform API documentation.
• HCP Terraform Fundamentals tutorials.
• Terraform State tutorials.