Build certificate authority (CA) in Vault with an offline root

17min
|
Vault
Terraform

This tutorial builds on the Build Your Own Certificate Authority (CA) tutorial. You will learn how to create a CA chain hierarchy that uses an offline root and online intermediate CAs in Vault.

Prerequisites

To perform the tasks described in this tutorial, you need:

Vault binary.
Certstrap tool by Square to create a simple offline Root CA.
Terraform CLI to provision Vault server.
jq tool to parse JSON output.
openssl tool which is generally packaged with most OS distributions for inspecting x509 certificates and certificate signing requests that you will create in this tutorial.
tree tool to view the folder structure.

Scenario introduction

In this tutorial, you will complete the following on your local machine:

Create an offline Root CA.
Start Vault server in development mode.
Create Intermediate Certificate Authority 1 (ICA1) in Vault signed with offline Root CA.
Create Intermediate Certificate Authority 2 (ICA2) in Vault signed with ICA1.
Issue a client x509 Certificate rooted in ICA2.

Step 1: Create an offline Root CA

Use the certstrap tool to create the offline Root CA, select a very strong password for protecting the CA key.

$ certstrap init \
     --organization "Test" \
     --organizational-unit "Test Org" \
     --country "US" \
     --province "MD" \
     --locality "Bethesda" \
     --common-name "Testing Root"

Output:

 Enter passphrase (empty for no passphrase):
 Enter same passphrase again:
 Created out/Testing_Root.key (encrypted by passphrase)
 Created out/Testing_Root.crt
 Created out/Testing_Root.crl

Inspect the offline Root CA certificate with openssl to ensure it has the expected subject.
```
$ openssl x509 -in out/Testing_Root.crt -noout  -subject -issuer
```
Output:
```
 subject= /C=US/ST=DC/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root
 issuer= /C=US/ST=DC/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root
```
Note
In this tutorial, you created the Root CA in your current local directory. However, you will generate the offline Root CA material with the root user, under a well defined root user owned directory.

The tree command outputs the out folder hierarchy.

$ tree out

Output:

out
├── Testing_Root.crl
├── Testing_Root.crt
└── Testing_Root.key

Step 2: Start Vault server in dev mode

To proceed with the tutorial, open two terminals on your local machine.

In one terminal, start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to address 127.0.0.1:8200. The server is initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
In a second terminal session, export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Additionally, export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

For this tutorial, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

The Vault server is ready.

Step 3: Generate ICA1 in vault

Next, use Terraform Vault provider to provision and manage intermediate PKI endpoints in Vault by creating below file hierarchy.

In your second terminal, create the necessary terraform files to manage ICA1 resources in Vault.

Create main.tf file which defines vault provider.

$ cat > main.tf << EOF
provider "vault" {}

locals {
 default_3y_in_sec   = 94608000
 default_1y_in_sec   = 31536000
 default_1hr_in_sec = 3600
}

EOF

Create test_org_ica1.tf file which enables and configures PKI secrets engine.

$ cat > test_org_ica1.tf << EOF
resource "vault_mount" "test_org_v1_ica1_v1" {
 path                      = "test-org/v1/ica1/v1"
 type                      = "pki"
 description               = "PKI engine hosting intermediate CA1 v1 for test org"
 default_lease_ttl_seconds = local.default_1hr_in_sec
 max_lease_ttl_seconds     = local.default_3y_in_sec
}

resource "vault_pki_secret_backend_intermediate_cert_request" "test_org_v1_ica1_v1" {
 depends_on   = [vault_mount.test_org_v1_ica1_v1]
 backend      = vault_mount.test_org_v1_ica1_v1.path
 type         = "internal"
 common_name  = "Intermediate CA1 v1 "
 key_type     = "rsa"
 key_bits     = "2048"
 ou           = "test org"
 organization = "test"
 country      = "US"
 locality     = "Bethesda"
 province     = "MD"
}
EOF

To ensure the file structure is properly configured, use the tree command to view the directory.
```
$ tree
.
├── main.tf
├── out
│   ├── Testing_Root.crl
│   ├── Testing_Root.crt
│   └── Testing_Root.key
└── test_org_ica1.tf

2 directories, 5 file
```
The test_org_ica1.tf file has the necessary code to enable a new PKI endpoint for the ICA1 in Vault and to generate a Certificate Signing Request (CSR). The CSR will be signed by the offline Root CA next.

First, initialize terraform; this downloads the necessary providers and initializes the backend.

$ terraform init

Initializing the backend...

Initializing provider plugins...
...snip...

Terraform has been successfully initialized!

Execute the apply command to configure Vault.

$ terraform apply

This displays the actions to be performed by Terraform.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # vault_mount.test_org_v1_ica1_v1 will be created
  + resource "vault_mount" "test_org_v1_ica1_v1" {
      + accessor                  = (known after apply)
      + default_lease_ttl_seconds = 3600
      + description               = "PKI engine hosting intermediate CA1 v1 for test org"
      + external_entropy_access   = false
      + id                        = (known after apply)
      + max_lease_ttl_seconds     = 94608000
      + path                      = "test-org/v1/ica1/v1"
      + seal_wrap                 = (known after apply)
      + type                      = "pki"
    }

  # vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica1_v1 will be created
  + resource "vault_pki_secret_backend_intermediate_cert_request" "test_org_v1_ica1_v1" {
      + backend            = "test-org/v1/ica1/v1"
      + common_name        = "Intermediate CA1 v1 "
      + country            = "US"
      + csr                = (known after apply)
      + format             = "pem"
      + id                 = (known after apply)
      + key_bits           = 2048
      + key_type           = "rsa"
      + locality           = "Bethesda"
      + organization       = "test"
      + ou                 = "test org"
      + private_key        = (sensitive value)
      + private_key_format = "der"
      + private_key_type   = (known after apply)
      + province           = "MD"
      + type               = "internal"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

When prompted, enter yes to accept the plan and proceed with Vault configuration.

   vault_mount.test_org_v1_ica1_v1: Creating...
   vault_mount.test_org_v1_ica1_v1: Creation complete after 0s [id=test-org/v1/ica1/v1]
   vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica1_v1: Creating...
   vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica1_v1: Creation complete after 0s [id=test-org/v1/ica1/v1/intermediate/generate/internal]

   Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Note

The path for the PKI endpoint is set to test-org/vX/ica1/Y. The X denotes the version of offline Root CA and Y denotes the version of ICA1. This ensures that the future versions of ICA1 past expiry can be added in parallel, and eventually clients of ICA1 can move off the earlier one. It is possible to use the same PKI path and reset ICA1, but it can be a breaking change depending upon your PKI use cases.

Create a new csr folder.
```
$ mkdir csr
```

Get the ICA1 CSR from the Terraform state file and store it under a new csr folder.

$ terraform show -json | jq '.values["root_module"]["resources"][].values.csr' -r | grep -v null > csr/Test_Org_v1_ICA1_v1.csr

Sign ICA1 CSR with the offline Root CA.

$ certstrap sign \
     --expires "3 year" \
     --csr csr/Test_Org_v1_ICA1_v1.csr \
     --cert out/Intermediate_CA1_v1.crt \
     --intermediate \
     --path-length "1" \
     --CA "Testing Root" \
     "Intermediate CA1 v1"

Output:

 Enter passphrase for CA key (empty for no passphrase):
 Building intermediate
 Created out/Intermediate_CA1_v1.crt from out/Intermediate_CA1_v1.csr signed by out/Testing_Root.key

The output displays out/Intermediate_CA1_v1.csr but it is a tooling print error as Intermediate_CA1_v1.csr is passed from under the csr and not out folder.

Create the cacerts folder to store the CA chain files that will be set on the PKI endpoints in Vault.
```
$ mkdir cacerts
```
Append offline Root CA at the end of ICA1 cert to create a CA chain under cacerts folder. You will use this to set the signed ICA1 in Vault.
```
$ cat out/Intermediate_CA1_v1.crt out/Testing_Root.crt > cacerts/test_org_v1_ica1_v1.crt
```

Update the Terraform code to set the signed cert for ICA1 in Vault.

$ cat >> test_org_ica1.tf << EOF

resource "vault_pki_secret_backend_intermediate_set_signed" "test_org_v1_ica1_v1_signed_cert" {
 depends_on   = [vault_mount.test_org_v1_ica1_v1]
 backend      = vault_mount.test_org_v1_ica1_v1.path

 certificate = file("\${path.module}/cacerts/test_org_v1_ica1_v1.crt")
}

EOF

Apply the Terraform changes to set the signed ICA1 in Vault.

$ terraform apply

Output:

...truncated...
Enter a value: yes

vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica1_v1_signed_cert: Creating...
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica1_v1_signed_cert: Creation complete after 0s [id=test-org/v1/ica1/v1/intermediate/set-signed]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Verify that the Terraform output displays the new ICA1 set

Verify the ICA1 cert in Vault.

$ curl -s $VAULT_ADDR/v1/test-org/v1/ica1/v1/ca/pem | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin  | openssl pkcs7 -print_certs -noout

Output:

subject=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA1 v1
issuer=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root

Verify the ICA1 CA chain in Vault.

$ curl -s $VAULT_ADDR/v1/test-org/v1/ica1/v1/ca_chain | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin  | openssl pkcs7 -print_certs -noout

Output:

subject=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA1 v1
issuer=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root

subject=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root
issuer=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root

Your directory structure should now resemble the example output.

$ tree
.
├── cacerts
│   └── test_org_v1_ica1_v1.crt
├── csr
│   └── Test_Org_v1_ICA1_v1.csr
├── main.tf
├── out
│   ├── Intermediate_CA1_v1.crt
│   ├── Testing_Root.crl
│   ├── Testing_Root.crt
│   └── Testing_Root.key
├── terraform.tfstate
├── terraform.tfstate.backup
└── test_org_ica1.tf

4 directories, 10 files

Step 4: Generate ICA2 in vault

Next, generate Terraform code to create and manage ICA2 in Vault.

In your second terminal, create a Terraform file named, "test_org_ica2.tf" to manage ICA2 resources in Vault.

$ cat > test_org_ica2.tf << EOF
resource "vault_mount" "test_org_v1_ica2_v1" {
 path                      = "test-org/v1/ica2/v1"
 type                      = "pki"
 description               = "PKI engine hosting intermediate CA2 v1 for test org"
 default_lease_ttl_seconds = local.default_1hr_in_sec
 max_lease_ttl_seconds     = local.default_1y_in_sec
}

resource "vault_pki_secret_backend_intermediate_cert_request" "test_org_v1_ica2_v1" {
 depends_on   = [vault_mount.test_org_v1_ica2_v1]
 backend      = vault_mount.test_org_v1_ica2_v1.path
 type         = "internal"
 common_name  = "Intermediate CA2 v1 "
 key_type     = "rsa"
 key_bits     = "2048"
 ou           = "test org"
 organization = "test"
 country      = "US"
 locality     = "Bethesda"
 province     = "MD"
}

resource "vault_pki_secret_backend_root_sign_intermediate" "test_org_v1_sign_ica2_v1_by_ica1_v1" {
 depends_on = [
   vault_mount.test_org_v1_ica1_v1,
   vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1,
 ]
 backend              = vault_mount.test_org_v1_ica1_v1.path
 csr                  = vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1.csr
 common_name          = "Intermediate CA2 v1.1"
 exclude_cn_from_sans = true
 ou                   = "test org"
 organization         = "test"
 country              = "US"
 locality             = "Bethesda"
 province             = "MD"
 max_path_length      = 1
 ttl                  = local.default_1y_in_sec
}

resource "vault_pki_secret_backend_intermediate_set_signed" "test_org_v1_ica2_v1_signed_cert" {
 depends_on  = [vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1]
 backend     = vault_mount.test_org_v1_ica2_v1.path
 certificate = format("%s\n%s", vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1.certificate, file("\${path.module}/cacerts/test_org_v1_ica1_v1.crt"))
}
EOF

Apply Terraform changes for ICA2.

$ terraform apply

Output:

vault_mount.test_org_v1_ica1_v1: Refreshing state... [id=test-org/v1/ica1/v1]
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica1_v1_signed_cert: Refreshing state... [id=test-org/v1/ica1/v1/intermediate/set-signed]
vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica1_v1: Refreshing state... [id=test-org/v1/ica1/v1/intermediate/generate/internal]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # vault_mount.test_org_v1_ica2_v1 will be created
  + resource "vault_mount" "test_org_v1_ica2_v1" {
      + accessor                  = (known after apply)
      + default_lease_ttl_seconds = 3600
      + description               = "PKI engine hosting intermediate CA2 v1 for test org"
      + external_entropy_access   = false
      + id                        = (known after apply)
      + max_lease_ttl_seconds     = 31536000
      + path                      = "test-org/v1/ica2/v1"
      + seal_wrap                 = (known after apply)
      + type                      = "pki"
    }

  # vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1 will be created
  + resource "vault_pki_secret_backend_intermediate_cert_request" "test_org_v1_ica2_v1" {
      + backend            = "test-org/v1/ica2/v1"
      + common_name        = "Intermediate CA2 v1 "
      + country            = "US"
      + csr                = (known after apply)
      + format             = "pem"
      + id                 = (known after apply)
      + key_bits           = 2048
      + key_type           = "rsa"
      + locality           = "Bethesda"
      + organization       = "test"
      + ou                 = "test org"
      + private_key        = (sensitive value)
      + private_key_format = "der"
      + private_key_type   = (known after apply)
      + province           = "MD"
      + type               = "internal"
    }

  # vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica2_v1_signed_cert will be created
  + resource "vault_pki_secret_backend_intermediate_set_signed" "test_org_v1_ica2_v1_signed_cert" {
      + backend     = "test-org/v1/ica2/v1"
      + certificate = (known after apply)
      + id          = (known after apply)
    }

  # vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1 will be created
  + resource "vault_pki_secret_backend_root_sign_intermediate" "test_org_v1_sign_ica2_v1_by_ica1_v1" {
      + backend              = "test-org/v1/ica1/v1"
      + ca_chain             = (known after apply)
      + certificate          = (known after apply)
      + common_name          = "Intermediate CA2 v1.1"
      + country              = "US"
      + csr                  = (known after apply)
      + exclude_cn_from_sans = true
      + format               = "pem"
      + id                   = (known after apply)
      + issuing_ca           = (known after apply)
      + locality             = "Bethesda"
      + max_path_length      = 1
      + organization         = "test"
      + ou                   = "test org"
      + province             = "MD"
      + serial               = (known after apply)
      + ttl                  = "31536000"
      + use_csr_values       = false
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

When prompted, enter yes to accept the plan and proceed with Vault configuration.

vault_mount.test_org_v1_ica2_v1: Creating...
vault_mount.test_org_v1_ica2_v1: Creation complete after 0s [id=test-org/v1/ica2/v1]
vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1: Creating...
vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1: Creation complete after 0s [id=test-org/v1/ica2/v1/intermediate/generate/internal]
vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1: Creating...
vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1: Creation complete after 0s [id=test-org/v1/ica1/v1/Intermediate CA2 v1.1]
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica2_v1_signed_cert: Creating...
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica2_v1_signed_cert: Creation complete after 0s [id=test-org/v1/ica2/v1/intermediate/set-signed]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Verify that the Terraform output displays the new ICA2 set

Verify the ICA2 cert in Vault.

$ curl -s $VAULT_ADDR/v1/test-org/v1/ica2/v1/ca/pem | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin  | openssl pkcs7 -print_certs -noout

Output:

subject=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA2 v1.1
issuer=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA1 v1

Verify the ICA2 CA chain in Vault.

$ curl -s $VAULT_ADDR/v1/test-org/v1/ica2/v1/ca_chain | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin  | openssl pkcs7 -print_certs -noout

Output:

subject=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA2 v1.1
issuer=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA1 v1

subject=/C=US/ST=MD/L=Bethesda/O=test/OU=test org/CN=Intermediate CA1 v1
issuer=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root

subject=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root
issuer=/C=US/ST=MD/L=Bethesda/O=Test/OU=Test Org/CN=Testing Root

Run ICA2 x509 certificate constraint check.

$ curl -s $VAULT_ADDR/v1/test-org/v1/ica2/v1/ca/pem | openssl x509 -in /dev/stdin -noout -text | grep "X509v3 extensions"  -A 13

Output example:

       X509v3 extensions:
           X509v3 Key Usage: critical
               Certificate Sign, CRL Sign
           X509v3 Basic Constraints: critical
               CA:TRUE, pathlen:1
           X509v3 Subject Key Identifier:
               AE:88:D1:D6:3D:3D:AD:BD:AF:6F:D0:9D:CB:5A:E6:A6:B3:71:94:CB
           X509v3 Authority Key Identifier:
               keyid:11:12:F5:E4:85:6C:E4:ED:75:37:FB:C3:CD:14:D6:B8:81:14:F6:44

           X509v3 Name Constraints: critical
               Permitted:
                 DNS:test.com

Notice that the X509v3 Basic Constraints value pathlen:1, ensures that ICA2 can only sign CSR requests, but not create more intermediate certificate authorities.

Congratulations, you successfully established your intermediate CAs in Vault with an offline Root.

Step 5: Issue a client x509 cert rooted in ICA2

Create a new PKI role rooted in ICA2 to issue client x509 certificates for test.com subdomains.

$ cat > test_org_ica2_role_test_dot_com.tf << EOF
resource "vault_pki_secret_backend_role" "role" {
 backend            = vault_mount.test_org_v1_ica2_v1.path
 name               = "test-dot-com-subdomain"
 ttl                = local.default_1hr_in_sec
 allow_ip_sans      = true
 key_type           = "rsa"
 key_bits           = 2048
 key_usage          = [ "DigitalSignature"]
 allow_any_name     = false
 allow_localhost    = false
 allowed_domains    = ["test.com"]
 allow_bare_domains = false
 allow_subdomains   = true
 server_flag        = false
 client_flag        = true
 no_store           = true
 country            = ["US"]
 locality           = ["Bethesda"]
 province           = ["MD"]
}

EOF

Apply Terraform changes for ICA2.

$ terraform apply

Output:

vault_mount.test_org_v1_ica2_v1: Refreshing state... [id=test-org/v1/ica2/v1]
vault_mount.test_org_v1_ica1_v1: Refreshing state... [id=test-org/v1/ica1/v1]
vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica1_v1: Refreshing state... [id=test-org/v1/ica1/v1/intermediate/generate/internal]
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica1_v1_signed_cert: Refreshing state... [id=test-org/v1/ica1/v1/intermediate/set-signed]
vault_pki_secret_backend_intermediate_cert_request.test_org_v1_ica2_v1: Refreshing state... [id=test-org/v1/ica2/v1/intermediate/generate/internal]
vault_pki_secret_backend_root_sign_intermediate.test_org_v1_sign_ica2_v1_by_ica1_v1: Refreshing state... [id=test-org/v1/ica1/v1/Intermediate CA2 v1.1]
vault_pki_secret_backend_intermediate_set_signed.test_org_v1_ica2_v1_signed_cert: Refreshing state... [id=test-org/v1/ica2/v1/intermediate/set-signed]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # vault_mount.test_org_v1_ica2_v1 has been changed
  ~ resource "vault_mount" "test_org_v1_ica2_v1" {
        id                        = "test-org/v1/ica2/v1"
      + options                   = {}
        # (9 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # vault_pki_secret_backend_role.role will be created
  + resource "vault_pki_secret_backend_role" "role" {
      + allow_any_name                     = false
      + allow_bare_domains                 = false
      + allow_glob_domains                 = false
      + allow_ip_sans                      = true
      + allow_localhost                    = false
      + allow_subdomains                   = true
      + allowed_domains                    = [
          + "test.com",
        ]
      + allowed_domains_template           = false
      + backend                            = "test-org/v1/ica2/v1"
      + basic_constraints_valid_for_non_ca = false
      + client_flag                        = true
      + code_signing_flag                  = false
      + country                            = [
          + "US",
        ]
      + email_protection_flag              = false
      + enforce_hostnames                  = true
      + generate_lease                     = false
      + id                                 = (known after apply)
      + key_bits                           = 2048
      + key_type                           = "rsa"
      + key_usage                          = [
          + "DigitalSignature",
        ]
      + locality                           = [
          + "Bethesda",
        ]
      + name                               = "test-dot-com-subdomain"
      + no_store                           = true
      + not_before_duration                = (known after apply)
      + province                           = [
          + "MD",
        ]
      + require_cn                         = true
      + server_flag                        = false
      + ttl                                = "3600"
      + use_csr_common_name                = true
      + use_csr_sans                       = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

When prompted, enter yes to accept the plan and proceed with Vault configuration.

vault_pki_secret_backend_role.role: Creating...
vault_pki_secret_backend_role.role: Creation complete after 0s [id=test-org/v1/ica2/v1/roles/test-dot-com-subdomain]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Next, create a certificate rooted in ICA2 using Vault CLI.

$ vault write -format=json test-org/v1/ica2/v1/issue/test-dot-com-subdomain \
   common_name=1.test.com | jq .data.certificate -r | openssl x509 -in /dev/stdin -text -noout

Output example:

Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number:
           7f:79:7f:87:92:26:81:37:1c:64:de:40:14:44:19:5d:b3:e6:64:16
   Signature Algorithm: sha256WithRSAEncryption
       Issuer: C=US, ST=MD, L=Bethesda, O=test, OU=test org, CN=Intermediate CA2 v1.1
       Validity
           Not Before: Sep 25 23:26:16 2021 GMT
           Not After : Sep 26 00:26:45 2021 GMT
       Subject: C=US, ST=MD, L=Bethesda, CN=1.test.com
       Subject Public Key Info:
           Public Key Algorithm: rsaEncryption
               Public-Key: (2048 bit)
               Modulus:
                   00:c2:9b:7c:91:55:da:70:5e:72:5e:3c:aa:20:9f:
                   2d:c7:3c:ed:4f:5e:a7:cf:50:3b:0b:e4:4f:df:69:
                   34:1a:40:0d:bf:1a:6c:53:1f:f2:0d:3a:4a:83:3f:
                   47:16:8e:88:4c:a9:bc:be:20:22:04:c0:9b:76:52:
                   b1:96:a8:8e:0f:4e:36:b8:aa:4f:da:3b:1b:3b:64:
                   34:c6:c8:e5:c2:2a:da:a1:e4:3c:0d:13:f3:e5:8c:
                   a4:b1:d5:37:11:d0:99:70:b0:37:5a:f1:4a:e9:5a:
                   e3:09:ba:db:d9:ee:59:a3:94:ee:b4:97:85:3b:a4:
                   0a:0d:31:c9:91:09:7e:66:de:01:10:fd:1e:dd:89:
                   b9:65:e9:9d:c6:aa:c6:11:b8:c4:eb:06:e6:c3:ac:
                   6c:b4:1b:65:e9:29:bb:c7:26:84:fb:52:0e:07:bd:
                   da:c2:37:5d:d2:21:4c:a9:7f:51:c4:61:03:bc:06:
                   a8:9e:18:0c:2b:8b:7b:9a:0d:30:d0:7e:2e:14:72:
                   6b:0b:55:c3:68:51:6c:3b:9d:87:46:0a:3d:7b:77:
                   a9:48:06:96:ed:af:2d:25:f4:28:eb:5b:13:ac:61:
                   22:8a:c8:a7:99:eb:d1:1c:41:3d:4b:e6:2e:16:f3:
                   56:7e:b6:38:ac:07:b1:66:4c:5f:f2:65:2c:36:1b:
                   91:1d
               Exponent: 65537 (0x10001)
       X509v3 extensions:
           X509v3 Key Usage: critical
               Digital Signature
           X509v3 Extended Key Usage:
               TLS Web Client Authentication
           X509v3 Subject Key Identifier:
               1B:A5:58:99:2E:0E:42:48:B3:8F:C4:D4:6D:E5:91:C2:5B:48:13:57
           X509v3 Authority Key Identifier:
               keyid:7A:AF:49:89:C6:BC:F1:5D:F4:A9:3A:79:66:CD:CA:E4:36:7E:78:64

           X509v3 Subject Alternative Name:
               DNS:1.test.com
   Signature Algorithm: sha256WithRSAEncryption
        7a:c2:1d:47:8e:5b:92:c6:6b:5b:28:8e:a6:29:64:1a:ea:bd:
        c3:0f:4b:c1:37:8c:61:78:b8:1c:d9:2a:9c:aa:33:00:40:f2:
        14:3a:ae:b6:ec:9e:d3:aa:ea:9f:50:15:9b:03:0c:30:9b:6d:
        f1:79:47:77:d2:1c:ff:bf:ee:8c:70:b1:c6:15:65:4c:e4:76:
        d1:b8:cd:27:68:ec:e8:dd:41:b1:50:bd:0a:1a:16:ab:8e:aa:
        6b:1f:8e:15:28:e6:51:ee:5d:c6:62:b0:cd:d7:87:cb:9b:18:
        6c:c3:82:44:4a:71:f8:bb:fa:c2:88:39:d4:7b:08:e5:7e:8a:
        47:ad:7b:d4:06:d7:e6:f3:d8:c9:79:83:22:b6:0a:db:10:44:
        9d:1b:5c:19:73:90:f4:cf:0f:72:52:14:1d:e6:10:ef:29:91:
        55:10:2d:17:f5:4a:d2:ed:85:be:37:6c:f1:75:b2:61:70:cd:
        b2:ed:43:ec:bc:69:60:87:99:57:ca:69:2c:76:b8:8e:26:46:
        c2:a2:ad:ff:a7:99:04:8a:d4:58:dd:99:53:6f:3f:10:1f:52:
        24:6d:af:e0:a8:a8:eb:62:10:f8:1c:5c:99:13:43:55:d9:a1:
        53:34:62:60:e6:fd:aa:49:30:58:18:5e:aa:c4:51:50:50:ac:
        d6:df:4f:b1

Congratulations, you have successfully created intermediate CAs (ICA1/ICA2) in Vault rooted in an offline Root CA and have issued a client certificate rooted in intermediate CA(ICA2).

Use the tree command on your current folder to display the folder hierarchy.

$ tree
.
├── cacerts
│   └── test_org_v1_ica1_v1.crt
├── csr
│   └── Test_Org_v1_ICA1_v1.csr
├── main.tf
├── out
│   ├── Intermediate_CA1_v1.crt
│   ├── Testing_Root.crl
│   ├── Testing_Root.crt
│   └── Testing_Root.key
├── terraform.tfstate
├── terraform.tfstate.backup
├── test_org_ica1.tf
├── test_org_ica2.tf
└── test_org_ica2_role_test_dot_com.tf

4 directories, 12 files

For production workflow, you can decide to not check in the csr folder as the information under it can be obtained from the Terraform state file.

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Run Terraform to undo all the changes.

$ terraform apply -destroy -auto-approve

...snip...
Apply complete! Resources: 0 added, 0 changed, 8 destroyed.

Remove the terraform state files and .tf files.
```
$ rm *tfstate* *.tf
```
Remove the csr, cacerts, and out directories.
```
$ rm -rf csr cacerts out
```
Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```
You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```

Help and reference

Build your own certificate authority (CA)

Enable ACME with PKI secrets engine