Manage federated workload identities with AWS IAM and Vault Enterprise

14min
|
Enterprise
Vault
Terraform
Interactive

Note

Using AWS identity providers requires a Vault Enterprise license.

In this tutorial, you will learn how to configure and use AWS IAM OIDC identity providers to securely connect your Vault plugin.

Challenge

Plugins such as the AWS Secrets engine originally required static security credentials. The user supplies the long lived and highly privileged AWS credentials in the plugin configuration. For AWS this was the AWS_ACCESS_KEY and the AWS_SECRET_ACCESS_KEY.

$ vault write aws/config/root \
    access_key=<<AWS_ACCESS_KEY>> \
    secret_key=<<AWS_SECRET_KEY>> \
    region=us-east-1

Some organizations have strict requirements and must use of short lived, ephemeral credentials. For such organizations, this configuration is not acceptable.

Vault provides root credential rotations API, but this still requires an initial set of long lived, highly privileged credentials credentials. Even the possibility of mishandling credentials during configuration might be unacceptable.

Solution

A identity provider (IdP) with a trust relationship with Vault ensures no use of static credentials.

With an identity provider, you can manage your user identities in Vault and give these temporary user identities permissions to use AWS resources.

This tutorial instructs the user how to set up this trust relationship between a cloud identity provider, Vault Enterprise, and a Vault plugin. Identity provider provides a method to securely configure a Vault plugin. Through an IAM role, Vault has access to specific capabilities and provides temporary security credentials to access AWS.

Personas

The intended audience of this tutorial is an Vault or AWS system administrator with privileged access and the ability to create AWS IAM identity providers.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

This tutorial requires an administrator access to an AWS account, Terraform, Vault CLI and the example Terraform configuration to create a demonstration environment.

Vault Enterprise version 1.16 or later with premium license.
Vault CLI
AWS CLI
Optionally - the use the Terraform CLI to create and configure the AWS OIDC identity provider.
Amazon Web Services test account. This tutorial requires the creation of new cloud resources and but should incur minimal costs associated with the deployment and management of these resources. Your IAM user will need permission to create AWS IAM resources.
- An AWS IAM user with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY used to create an AWS IAM Provider.
Git
ngrok installed and configured with an auth token.
openSSL

Set up ngrok

Ngrok is a "reverse proxy". The set up in this tutorial exposes your dev Vault instance to the internet. Your local computer becomes a locally hosted server, anyone can access it through a subdomain of ngrok.com.

In this tutorial, ngrok provides a forwarding address that your local instance of Vault that will be accessible from AWS.

Note

Be aware that using ngrok creates a publicly routable address for the Vault dev instance. For more secure production configurations, see the Additional discussion section.

Open a terminal to run ngrok.
Follow the official instructions to download ngrok for your OS distribution and set up a free account: https://ngrok.com/download
After setting up your account and obtaining an authtoken, configure ngrok with the auth token:
```
$ ngrok config add-authtoken <<token>>
```

Start ngrok.

$ ngrok http 127.0.0.1:8200

Example output:

ngrok                                                                         (Ctrl+C to quit)

New guides https://ngrok.com/docs/guides/site-to-site-apis/

Session Status                online
Account                       Hashicorp Account #1 (Plan: Enterprise)
Update                        update available (version 3.10.1, Ctrl-U to update)
Version                       3.9.0
Region                        United States (us)
Latency                       33ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    https://XXNNNXXNNXXX.ngrok.app -> http://localhost:8200

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Copy the ngrok Forwarding address for later use. This is the first part of the https://XXNNNXXNNXXX.ngrok.app -> http://localhost:8200 before the ->.

Vault setup

Open a second terminal and export an environment variable with a valid Vault Enterprise license.
```
$ export VAULT_LICENSE=02MV4UU43BK5....
```
For the Vault to access the AWS Account, set AWS Access Key and Secret Access Key environment variables for the user you created for this tutorial.
```
$ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
```
Start a Vault dev server with the literal string root as the root token value, and temporarily setting the VAULT_API_ADDR to the ngrok forwarding address.
```
$ VAULT_API_ADDR=<<NGROK_FORWARDING_ADDRESS>> vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200, but through VAULT_API_ADDR ngrok provided a forwarding address that makes it available publicly. Now AWS IAM OIDC identity provider can access Vault on your local machine.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
The Vault server is ready.

Confirm ngrok configuration

Open a third terminal.
In the new terminal, export an environment variable for the vault CLI to the ngrok forwarding address.
```
$ export VAULT_ADDR=<<FORDARDING_ADDRESS_FROM_NGROK>>
```
Setting the VAULT_ADDR to the ngrok forwarding address forces the Vault CLI to use ngrok.com which forward to your local instance of Vault.
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

To validate the correct set up of ngrok and Vault, check the status of Vault.

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.16.2
Build Date      2024-04-22T16:25:54Z
Storage Type    inmem
Cluster Name    vault-cluster-beeda372
Cluster ID      cf3c7383-ff3a-3502-964c-9ef3d2f094f4
HA Enabled      false

A similar response confirms that ngrok is forwarding to your Vault instance.

Obtain thumbprint for intermediate certificate authority

During configuration, Vault requires a thumbprint from the top intermediate certificate authority that signed the certificate.

A certificate thumbprint is an unique identifier used to verify the certificate.

To get the thumbprint, ether follow the directions at Obtain the thumbprint for an OpenID Connect identity provider and copy down the thumbprint for use in this tutorial, or you can follow the steps below to retrieve the thumbprint.

Extract the domain from the URI of the JSON web key set of the IdP.

$ export DOMAIN=$(curl -s http://127.0.0.1:8200/v1/identity/oidc/provider/default/.well-known/openid-configuration | jq -r '.jwks_uri | sub("^https://";"") | sub("\/(.*)";"")')

Using the DOMAIN from above, this command extracts the certificate and calculates thumbprint.

$ export THUMBPRINT=$(openssl x509 -in <(openssl s_client -servername "keys.$DOMAIN" -showcerts -connect "keys.$DOMAIN:443" 2>/dev/null) -outform DER| sha1sum -z | awk '{print $1;}')

Create AWS OIDC identity provider

This tutorial provides two methods to set up the AWS open identity provider, AWS CLI and Terraform.

In a new terminal window, clone the repository at learn-vault-plugin-wif.

$ git clone https://github.com/hashicorp-education/learn-vault-plugin-wif.git

Move into the cloned repository.
```
$ cd  learn-vault-plugin-wif
```
For the Vault CLI to access the AWS Account, set AWS Access Key and Secret Access Key environment variables.
```
$ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
```
Using the ngrok forwarding address copied earlier, update the code block with your ISSUER and paste in the terminal. This is the
```
$ export ISSUER=<<ISSUER>>
```
Note: No not include the "https://" when setting this environment variable.
Using the fingerprint created during the prerequisites, create another TF_VAR environment variable.
```
$ export THUMBPRINT=<<THUMBPRINT>>
```

Determine the AWS account ID and set an environment variable.

$ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

Setting the TOKEN_AUDIENCE to make future commands easier.
```
$ export TOKEN_AUDIENCE="vault-aws-secrets-test-user"
```

Create an file with an AWS assume role policy in /policy directory.

$ tee policy/assume-role.json <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Effect": "Allow",
            "Principal": {
               "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/$ISSUER/v1/identity/oidc/plugins"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
               "StringEquals": {
                  "https://${ISSUER}/v1/identity/oidc/plugins:aud": "vault-aws-secrets-test-user"
               }
            }
      }
   ]
}
EOF

Create the AWS IAM role that the identity provider uses, then put the ARN in an environment variable.

$ export AWS_ROLE_ARN=$(aws iam create-role --role-name vault-aws-secrets-engine-wif --assume-role-policy-document file://policy/assume-role.json  | jq ".Role.Arn")

Create a document with an inline policy.

$ tee policy/config-role.json <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "sts:AssumeRole",
               "iam:UpdateOpenIDConnectProviderThumbprint",
               "iam:PutRolePolicy",
               "iam:ListRolePolicies",
               "iam:ListInstanceProfilesForRole",
               "iam:ListAttachedRolePolicies",
               "iam:GetUser",
               "iam:GetRolePolicy",
               "iam:GetRole",
               "iam:GetOpenIDConnectProvider",
               "iam:GetInstanceProfile",
               "iam:DeleteRolePolicy",
               "iam:DeleteRole",
               "iam:DeleteOpenIDConnectProvider",
               "iam:CreateRole",
               "iam:CreateOpenIDConnectProvider",
               "ec2:DescribeInstances"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/*"
      },
      {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:user/*"
      },
      {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/vault-aws-secrets-engine-wif"
      },
      {
            "Action": [
               "sts:AssumeRole",
               "iam:UpdateOpenIDConnectProviderThumbprint",
               "iam:UpdateAccessKey",
               "iam:PutRolePolicy",
               "iam:ListRolePolicies",
               "iam:ListInstanceProfilesForRole",
               "iam:ListAttachedRolePolicies",
               "iam:ListAccessKeys",
               "iam:GetUser",
               "iam:GetRolePolicy",
               "iam:GetOpenIDConnectProvider",
               "iam:GetAccessKeyLastUsed",
               "iam:DeleteRolePolicy",
               "iam:DeleteRole",
               "iam:DeleteOpenIDConnectProvider",
               "iam:DeleteAccessKey",
               "iam:CreateRole",
               "iam:CreateOpenIDConnectProvider",
               "iam:CreateAccessKey"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:user/*"
      }
   ]
}
EOF

Attach an inline policy to vault-secrets-engine-wif.

$ aws iam put-role-policy \
 --role-name vault-aws-secrets-engine-wif \
 --policy-name vault-aws-secrets-engine \
 --policy-document file://policy/config-role.json

Create the identity provider.

$ aws iam create-open-id-connect-provider --url https://$ISSUER/v1/identity/oidc/plugins --thumbprint-list $THUMBPRINT --client-id-list "vault-aws-secrets-test-user"

Example output:

{
   "OpenIDConnectProviderArn": "arn:aws:iam::NNNNNNNNNNNN:oidc-provider/ddddddddddddd.ngrok.app"
}

Move into the terraform subfolder of the newly cloned repository.
```
$ cd terraform
```
Using the $ISSUER environment variable set earlier, create another variable. This variable needs to start with TF_VAR_ to be automatically used by Terraform to set up the identity provider.
```
$ export TF_VAR_issuer=$ISSUER
```

Initialize your Terraform configuration to download providers and modules.

$ terraform init
Initializing the backend...
## ...
Initializing provider plugins...
## ...
Terraform has been successfully initialized!
## ...

Run the apply, and paste in the thumbprint you acquired in the prerequisites.

$ terraform apply -auto-approve
data.aws_caller_identity.current: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=<..>]

...

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

account_id = <...>
aws_role_arn = "arn:aws:iam::<...>:role/vault-aws-secrets-engine-wif"
token_audience = "vault-aws-secrets-032e2847"

This creates a file called output.env. Sourcing it will apply the environment variables needed next in this tutorial.
```
$ source output.env 
```

Set up AWS secrets engine

Using the ngrok forwarding address copied earlier, update the following code block and paste in your terminal.
```
$ export ISSUER=<<ISSUER>>
```
Note: No not include the "https://" when setting this environment variable.

In a new terminal, set some Vault environment variables.

$ export VAULT_TOKEN=root && export VAULT_ADDR="https://$ISSUER"

Configure the identity provider issuer.

$ vault write identity/oidc/config issuer=$VAULT_ADDR
WARNING! The following warnings were returned from Vault:

* If "issuer" is set explicitly, all tokens must be validated against that
address, including those issued by secondary clusters. Setting issuer to
"" will restore the default behavior of using the cluster's api_addr as the
issuer.

Enable the AWS Secrets engine.

$ vault secrets enable aws
Success! Enabled the aws secrets engine at: aws/

Configure the AWS secrets engine with a role allowing access to the AWS identity provider and the audience intended.

$ vault write aws/config/root \
   identity_token_audience="${TOKEN_AUDIENCE}" \
   role_arn="${AWS_ROLE_ARN}"

The AWS IAM user my-role with a policy document with to get the current user information.

$ vault write aws/roles/my-role \
   credential_type=iam_user \
   policy_document=-<<EOF
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": "iam:GetUser",
         "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:user/vault-token-my-role-*"
      }
   ]
}
EOF

Example output:

Success! Data written to: aws/roles/my-role

Get AWS credentials from Vault.

$ vault read aws/creds/my-role
---                -----
lease_id           aws/creds/my-role/Ws0fpy9DJAgusZUQt0BH0EeS
lease_duration     768h
lease_renewable    true
access_key         AKIA...
secret_key         NN...
session_token      <nil>

This creates a temporary user with my-role attached to it. The policy attached allows the IAM user to identify itself.

Use the access_key and secrets_key above to reset your AWS environment variables.

$ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>

Use the AWS CLI to check which user you are using the credentials of.

$ aws iam get-user
{
   "User": {
      "Path": "/",
      "UserName": "vault-token-my-role-1718035748-ITpjTB8nP7mQiaygqWLa",
      "UserId": "AIDA...",
      "Arn": "arn:aws:iam::NNNNNNNNNNNN:user/vault-token-my-role-1718035748-ITpjTB8nP7mQiaygqWLa",
      "CreateDate": "2024-06-10T16:09:08+00:00"
   }
}

Set up AWS authentication method

In addition to using an identity provider to configure the AWS secrets engine, the AWS authentication method can use an identity provider instead of static credentials for set up.

The last section included the IAM policies needed to set up the AWS authentication method, but the Vault configuration for the AWS authentication method is still required.

In the same terminal used in the last section, write the OIDC configuration.

$ vault write identity/oidc/config issuer="${VAULT_ADDR}"

Example output:

WARNING! The following warnings were returned from Vault:

* If "issuer" is set explicitly, all tokens must be validated against that
address, including those issued by secondary clusters. Setting issuer to
"" will restore the default behavior of using the cluster's api_addr as the
issuer.

Enable AWS authentication.

$ vault auth enable aws
Success! Enabled aws auth method at: aws/

Configure the AWS authentication to use the identity provider.

$ vault write auth/aws/config/client \
   identity_token_audience="${TOKEN_AUDIENCE}" \
   role_arn="${AWS_ROLE_ARN}"

Example output:

Success! Data written to: auth/aws/config/client

Attach dev-role-iam to the token provided by the AWS auth method.

$ vault write auth/aws/role/dev-role-iam \
   auth_type=iam \
   bound_iam_principal_arn="arn:aws:iam::${AWS_ACCOUNT_ID}:user/*"

Example output:

Success! Data written to: auth/aws/role/dev-role-iam

Open a new terminal, and set the Vault environment variables.

$ export VAULT_TOKEN=root && export VAULT_ADDR=<<FORDARDING_ADDRESS_FROM_NGROK>>

For the Vault CLI to access the AWS Account, set AWS Access Key and Secret Access Key environment variables.
```
$ export AWS_ACCESS_KEY_ID=<<AWS_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<AWS_SECRET_ACCESS_KEY>>
```

Authenticate with Vault using dev-role-iam.

$ vault login -method=aws role=dev-role-iam

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                      Value
---                      -----
token                    hvs...
token_duration           768h
token_renewable          true
token_policies           ["default"]
identity_policies        []
policies                 ["default"]
token_meta_role_id       4f39f8dc-1587-1b3b-cba7-5558b371bbb5
token_meta_account_id    794723625284
token_meta_auth_type     iam

You have authenticated with Vault using the AWS auth method from an identity provider.

The output displays details associated with the Vault role.

Clean up

Delete the identity provider and the role in your AWS account.
In your AWS account find any users whose name starts with vault-token-my-role- and delete them.
Delete the cloned repository named learn-vault-plugin-wif.
```
$ cd .. && rm -rf  learn-vault-plugin-wif
```

In the terminal you ran the Terraform config, destroy the infrastructure.
```
$ terraform destroy -auto-approve
```
Delete the cloned repository named learn-vault-plugin-wif.
```
$ cd ../.. && rm -rf  learn-vault-plugin-wif
```

In the terminal with Vault, press Ctrl + C to stop the running Vault dev server.
In the terminal with ngrok, press Ctrl + C to stop the running server.
Close all open terminals.

Additional discussion

This tutorial used ngrok as a reverse proxy to make the Vault dev instance publicly routable. However, this is undesirable for most production use-cases.

For production, we recommend using a proxy to enable access to only the API endpoints strictly necessary.

In the case of workload identity federation, these are:

- ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/oidc/plugins/.well-known/openid-configuration
- ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/oidc/plugins/.well-known/keys

Substitute in appropriate values for your VAULT_ADDR and VAULT_NAMESPACE.

Be sure to also update the issuer in identity/oidc/config.

Where possible, consider restricting access to the proxy to AWS.

Next steps

Organizations in a highly regulated industry have rules against the use of static credentials, and until plugin Workload Identity Federation, plugin set up required static credentials.

In this tutorial, you learned how to set up AWS OIDC identity provider and avoided any use of static credentials in the process. This was possible through a trust relationship between Vault and AWS OIDC identity provider. Vault will manage the identities on the cloud provider and give these temporary user identities permission to use AWS resources.

For more information, see individual plugin documentation for instructions on how to set up workload identity federation.

Sync secrets with Vault Enterprise

Next Collection

Monitor & troubleshoot