Inject secrets into Terraform using the Vault provider

Traditionally, developers looking to safely provision infrastructure using Terraform are given their own set of long-lived, scoped AWS credentials. While this enables the developer's freedom, using long-lived credentials can be dangerous and difficult to secure.

1. Operators need to manage a large number of static, long-lived AWS IAM credentials with varying scope.

2. Long-lived credentials on a developer's local machine creates a large attack surface area. If a malicious actor gains access to the credentials, they could used them to damage resources.

You can address both concerns by storing your long-lived AWS credentials in HashiCorp's Vault's AWS Secrets Engine, then leverage Terraform's Vault provider to generate appropriately scoped & short-lived AWS credentials to be used by Terraform to provision resources in AWS.

As a result, operators (Vault Admin) are able to avoid managing static, long-lived secrets with varying scope and developers (Terraform Operator) are able to provision resources without having direct access to the secrets.

In this tutorial, you assume the role of both the Vault Admin and the Terraform Operator.

1. First, as a Vault Admin, you will configure AWS Secrets Engine in Vault.

2. Then, as a Terraform Operator, you will connect to the Vault instance to retrieve dynamic, short-lived AWS credentials generated by the AWS Secrets Engine to provision an Ubuntu EC2 instance.

3. Finally, as a Vault Admin, you will remove the Terraform Operator's ability to manipulate EC2 instances by modifying the policy for the corresponding Vault role.

Throughout this journey, you'll learn about the benefits and considerations this approach has to offer.

Prerequisites

In order to follow this tutorial, you should be familiar with the usual Terraform plan/apply workflow and Vault. If you're new to Terraform, refer first to the Terraform Getting Started tutorial. If you're new to Vault, refer first to the Vault Getting Started tutorial.

In addition, you will need the following:

• Terraform installed locally

• Vault installed locally

• an AWS account and AWS Access Credentials

  If you don't have AWS Access Credentials, create your AWS Access Key ID and Secret Access Key by navigating to your service credentials in the IAM service on AWS. Click "Create access key" here to view your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. You will need these values later.

Start Vault server

Start a Vault server in development mode with education as the root token. Leave this process running in your terminal window.

$ vault server -dev -dev-root-token-id="education"
==> Vault server configuration:

    Api Address: http://127.0.0.1:8200
            Cgo: disabled
Cluster Address: https://127.0.0.1:8201
     Go Version: go1.14.4
     Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
      Log Level: info
          Mlock: supported: false, enabled: false
  Recovery Mode: false
        Storage: inmem
        Version: Vault v1.4.3
    Version Sha: 491533b63ec9c1343eac3a24d8a7558185a0acb7+CHANGES

Your Vault server should now be up. Navigate to localhost:8200 and login into the instance using your root token: education.

Clone repository

In your terminal, clone the Inject Secrets repository and navigate into the directory. It contains the example configurations used in this tutorial.

$ git clone https://github.com/hashicorp/learn-terraform-inject-secrets-aws-vault && cd learn-terraform-inject-secrets-aws-vault

This directory should contain two Terraform workspaces — an vault-admin-workspace and a operator-workspace.

$ tree
.
├── README.md
├── operator-workspace
│   └── main.tf
│   └── versions.tf
└── vault-admin-workspace
    └── main.tf
│   └── versions.tf

Configure AWS Secrets Engine in Vault

In another terminal window (leave the Vault instance running), navigate to the Vault Admin directory.

$ cd vault-admin-workspace

In the main.tf file, you will find 2 resources:

1. the vault_aws_secret_backend.aws resource configures AWS Secrets Engine to generate a dynamic token that lasts for 2 minutes.

2. the vault_aws_secret_backend_role.admin resource configures a role for the AWS Secrets Engine named dynamic-aws-creds-vault-admin-role with an IAM policy that allows it iam:* and ec2:* permissions.

This role will be used by the Terraform Operator workspace to dynamically generate AWS credentials scoped to this IAM policy.

Before applying this configuration, set the required Terraform variable substituting <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your AWS Credentials. Notice that we're also setting the required Vault Provider arguments as environment variables: VAULT_ADDR & VAULT_TOKEN.

$ export TF_VAR_aws_access_key=<AWS_ACCESS_KEY_ID>
$ export TF_VAR_aws_secret_key=<AWS_SECRET_ACCESS_KEY>
$ export VAULT_ADDR=http://127.0.0.1:8200
$ export VAULT_TOKEN=education

Initialize the Vault Admin workspace.

$ terraform init

In your initialized directory, run terraform apply, review the planned actions, and confirm the run with a yes

$ terraform apply

## ...

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

backend = "dynamic-aws-creds-vault-admin-path"
role = "dynamic-aws-creds-vault-admin-role"

Notice that there are two output variables named backend and role. These output variables will be used by the Terraform Operator workspace in a later step.

If you go to the terminal where your Vault server is running, you should see Vault output something similar to the below. This means Terraform was successfully able to mount the AWS Secrets Engine at the specified path. The role has also been configured although it's not output in the logs.

[INFO]  core: successful mount: namespace= path=dynamic-aws-creds-vault-admin-path/ type=aws

Provision compute instance

Now that you have successfully configured Vault's AWS Secrets Engine, you can retrieve dynamic short lived AWS token to provision an EC2 instance.

Navigate to the Terraform Operator workspace.

$ cd ../operator-workspace

In the main.tf file, you should find the following data and resource blocks:

1. the terraform_remote_state.admin data block retrieves the Terraform state file generated from your Vault Admin workspace

2. the vault_aws_access_credentials.creds data block retrieves the dynamic, short-lived AWS credentials from your Vault instance. Notice that this uses the Vault Admin workspace's output variables: backend and role

3. the aws provider is initialized with the short-lived credentials retrieved by vault_aws_access_credentials.creds. The provider is configured to the us-east-1 region, as defined by the region variable

4. the aws_ami.ubuntu data block retrieves the most recent Ubuntu image

5. the aws_instance.main resource block creates an t2.micro EC2 instance

Initialize the Terraform Operator workspace.

$ terraform init

Navigate to the IAM Users page in AWS Console. Search for the username prefix vault-token-terraform-dynamic-aws-creds-vault-admin. Nothing should show up on your initial search. However, a user with this prefix should appear on terraform plan or terraform apply.

Apply the Terraform configuration, remember to confirm the run with a yes. Terraform will provision the EC2 instance using the dynamic credentials generated from Vault.

$ terraform apply

Refresh the IAM Users and search for the vault-token-terraform-dynamic-aws-creds-vault-admin prefix. You should see a IAM user.

This IAM user was generated by Vault with the appropriate IAM policy configured by the Vault Admin workspace. Because the default_lease_ttl_seconds is set to 120 seconds, Vault will revoke those IAM credentials and they will be removed from the AWS IAM console after 120 seconds.

Navigate to the EC2 page and search for dynamic-aws-creds-operator. You should see an instance provisioned by the Terraform Operator workspace using the short-lived AWS credentials.

Every Terraform run with this configuration will use its own unique set of AWS IAM credentials that are scoped to whatever the Vault Admin has defined.

The Terraform Operator doesn't have to manage long-lived AWS credentials locally. The Vault Admin only has to manage the Vault role rather than numerous, multi-scoped, long-lived AWS credentials.

After 120 seconds, you should see the following in the terminal running Vault.

2020-07-13T16:07:55.755-0700 [INFO]  expiration: revoked lease: lease_id=dynamic-aws-creds-vault-admin-path/creds/dynamic-aws-creds-vault-admin-role/z1PKR7Y623fk0ZQWW1kwaVVY

This shows that Vault has destroyed the short-lived AWS credentials generated for the apply run.

Destroy EC2 instance

Destroy the EC2 instance, remember to confirm the run with a yes.

$ terraform destroy

This run should have generated and used another set of IAM credentials. Verify that your EC2 instance has been destroyed by viewing the EC2 page of your AWS Console.

Restrict Vault role's permissions

If the Vault Admin wanted to remove the Terraform Operator's EC2 permissions, they would only need to update the Vault role's policy.

Navigate to the Vault Admin workspace.

$ cd ../vault-admin-workspace

Remove "ec2:*" from the vault_aws_secret_backend_role.admin resource in your main.tf file.

$ sed -i '' -e 's/, \"ec2:\*\"//g' main.tf

resource "vault_aws_secret_backend_role" "admin" {
  backend = vault_aws_secret_backend.aws.path
  name    = "${var.name}-role"
  credential_type = "iam_user"

  policy_document = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
-        "iam:*", "ec2:*"
+        "iam:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

This change restricts the Terraform Operator's ability to provision any AWS EC2 instance.

Apply the Terraform configuration, remember to confirm the run with a yes.

$ terraform apply

Verify restricted Terraform Operator permissions

Navigate to the Terraform Operator workspace.

$ cd ../operator-workspace

Run terraform plan. This plan should fail because the Terraform Operator no longer has CRUD permissions on EC2 instances due to changes to the dynamic-aws-creds-vault-admin role.

$ terraform plan

Error: UnauthorizedOperation: You are not authorized to perform this operation.
        status code: 403, request id: 8bb1d1f8-5667-456a-9fee-8387e0e2ceb0

Benefits and considerations

This approach to secret injection:

1. alleviates the Vault Admin's responsibility in managing numerous, multi-scoped, long-lived AWS credentials,

2. reduces the risk from a compromised AWS credential in a Terraform run (if a malicious user gains access to an AWS credential used in a Terraform run, that credential is only value for the length of the token's TTL),

3. allows for management of a role's permissions through a Vault role rather than the distribution/management of static AWS credentials,

4. enables development to provision resources without managing local, static AWS credentials

However, this approach may run into issues when applied to large multi-resource configurations. The generated dynamic AWS Credentials are only valid for the length of the token's TTL. As a result, if:

1. the apply process exceeds than the TTL and the configuration needs to provision another resource or

2. the apply confirmation time exceeds the TTL

the apply process will fail because the short-lived AWS Credentials have expired.

You could increase the TTL to conform to your situation; however, this also increases how long the temporary AWS credentials are valid, increasing the malicious actor's attack surface.

Summary

Congratulations! You have successfully:

1. configured Vault's AWS Secret Engine through Terraform,
2. used dynamic short-lived AWS credentials to provision infrastructure, and
3. restricted the AWS credential's permissions by adjusting the corresponding Vault role

Remember to clean up environment by destroying all resources in both Vault Admin and Terraform Operator workspaces.

$ terraform destroy --auto-approve && cd ../vault-admin-workspace && terraform destroy --auto-approve

Remember to stop your local Vault instance used in this tutorial by hitting Ctrl + C in the terminal window running Vault.

Now that you have inject secrets into Terraform using the Vault provider, you may like to:

• Watch a video exploring Best Practices for using Terraform with Vault.

• Learn how to codify management of Vault Community Edition and Vault Enterprise.

• Learn more about the various Vault secret engines

• You can take your security to the next level by leveraging Terraform Enterprise's Secure Storage of Variables to safely store sensitive variables like the Vault token used for authentication.

• Learn more about the Terraform Vault Provider.