HashiCorp Developer AI Private beta now available Sign up today
Consul
Automate security with Vault

Generate mTLS Certificates for Consul with Vault

  • 16min
  • |
  • Consul
  • Vault

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates with consul-template to rotate your certificates. This method enables each agent in the Consul datacenter to have a unique certificate with a relatively short time-to-live (ttl) that is automatically rotated, which allows you to safely and securely scale your datacenter while using mutual TLS (mTLS).

In this tutorial, your will secure your Consul datacenter with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use consul-template to fetch, renew, and periodically rotate your mTLS certificates on your Consul nodes.

Prerequisites

  • Consul: to complete this tutorial, you can use a local Consul dev agent or an existing Consul deployment.
  • Vault: the tutorial assumes you already have a running Vault cluster in your network. You can use a local Vault dev agent or an existing Vault deployment.
  • consul-template: to interact with your Consul agent you will need to install the consul-template binary on the node. To rotate TLS certificates you need the binary to be installed on every node in the datacenter. You can check Service Configuration with Consul Template to learn how to install and use consul-template.

The diagram below shows the minimal architecture needed to demonstrate the functionality.

Architectural diagram showing a Client server and a Vault server with an operator issuing a command to start an automation

Configure Vault's PKI secrets engine

In order to communicate with the Vault server, you will need to set the address and token.

$ export VAULT_ADDR='http://127.0.0.1:8200'

$ export VAULT_TOKEN="root"

Enable Vault's PKI secrets engine at the pki path.

$ vault secrets enable pki

Success! Enabled the pki secrets engine at: pki/

Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.

$ vault secrets tune -max-lease-ttl=87600h pki

Success! Tuned the secrets engine at: pki/

Note

This tutorial uses a common and recommended pattern which is to have one mount act as the root CA and to use this CA only to sign intermediate CA CSRs from other PKI secrets engines (which you will create in the next few steps). For tighter security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA.

Configure Vault as Consul's CA

Consul requires that all servers and clients have key pairs that are generated by a single Certificate Authority (CA).

In this lab you will use the PKI secrets engine to generate the necessary CA and certificates.

1. Generate the root CA

Generate the root certificate and save the certificate in CA_cert.crt.

$ vault write -field=certificate pki/root/generate/internal \
        common_name="dc1.consul" \
        ttl=87600h > CA_cert.crt

This generates a new self-signed CA certificate and private key. Vault will automatically revoke the generated root at the end of its lease period (TTL); the CA certificate will sign its own Certificate Revocation List (CRL).

Tuning configuration

You can adapt the TTL to comply with your internal policies on certificate lifecycle.

You can inspect the certificate created using openssl x509 -text -noout -in CA_cert.crt

Configure the CA and CRL URLs.

$ vault write pki/config/urls \
        issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
        crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"

Example output:

Success! Data written to: pki/config/urls

2. Generate an intermediate CA

First, enable the PKI secrets engine at the pki_int path.

$ vault secrets enable -path=pki_int pki

Success! Enabled the pki secrets engine at: pki_int/

Tune the pki_int secrets engine to issue certificates with a maximum time-to-live (TTL) of 43800 hours.

$ vault secrets tune -max-lease-ttl=43800h pki_int

Success! Tuned the secrets engine at: pki_int/

Tuning configuration

You can adapt the TTL to comply with your internal policies on certificate lifecycle.

Request an intermediate certificate signing request (CSR) and save request as pki_intermediate.csr.

$ vault write -format=json pki_int/intermediate/generate/internal \
        common_name="dc1.consul Intermediate Authority" \
        | jq -r '.data.csr' > pki_intermediate.csr

The command has no output.

3. Sign the CSR and import the certificate into Vault

$ vault write -format=json pki/root/sign-intermediate csr=@pki_intermediate.csr \
        format=pem_bundle ttl="43800h" \
        | jq -r '.data.certificate' > intermediate.cert.pem

The command has no output.

Once the CSR is signed, and the root CA returns a certificate, it can be imported back into Vault.

$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Success! Data written to: pki_int/intermediate/set-signed

4. Create a Vault role

A role is a logical name that maps to a policy used to generate credentials.

$ vault write pki_int/roles/consul-dc1 \
  allowed_domains="dc1.consul" \
  allow_subdomains=true \
  generate_lease=true \
  max_ttl="720h"

Example output:

Success! Data written to: pki_int/roles/consul-dc1

For this tutorial, you are using the following options for the role:

  • allowed_domains : Specifies the domains of the role. The command uses dc1.consul as the domain, which is the default configuration you are going to use for Consul.
  • allow_subdomains : Specifies if clients can request certificates with CNs that are subdomains of the CNs allowed by the other role options (NOTE: This includes wildcard subdomains.)
  • generate_lease : Specifies if certificates issued/signed against this role will have Vault leases attached to them. Certificates can be added to the CRL by Vault revoke <lease_id> when certificates are associated with leases.

This completes the Vault configuration as a CA. Move to next step to generate certificates.

Generate a server certificate

You can test the pki engine is configured correctly by generating your first certificate.

$ vault write pki_int/issue/consul-dc1 \
  common_name="server.dc1.consul" \
  ttl="24h" | tee certs.txt

Certificate TTL tuning

The TTL for the certificate is being set to 24 hours in this tutorial, meaning that this certificate will be valid only for 24 hours before expiring. You can try using a shorter TTL on a test environment to ensure certificates are revoked properly after TTL is expired.

Example output:

Key                 Value
---                 -----
lease_id            pki_int/issue/consul-dc1/lFfKfpxtM0xY0AHDlr9pJ2GM
lease_duration      23h59m59s
lease_renewable     false
ca_chain            [-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----]
certificate         -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
expiration          1599645187
issuing_ca          -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       3f:ec:bd:ea:01:a6:35:49:a7:6d:17:ba:13:88:c1:b8:35:b4:fc:4c

Configure Consul

Configure Consul TLS using the following configuration:

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/opt/consul/agent-certs/ca.crt"
cert_file = "/opt/consul/agent-certs/agent.crt"
key_file = "/opt/consul/agent-certs/agent.key"
auto_encrypt {
  allow_tls = true
}

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "/opt/consul/agent-certs/ca.crt",
  "cert_file": "/opt/consul/agent-certs/agent.crt",
  "key_file": "/opt/consul/agent-certs/agent.key",
  "auto_encrypt": {
    "allow_tls": true
  }
}

To configure TLS encryption for Consul servers, three files are required:

  • ca_file - CA (or intermediate) certificate to verify the identity of the other nodes.
  • cert_file - Consul agent public certificate
  • key_file - Consul agent private key

For the first Consul startup, you will use the certificate generated earlier.

Use the following commands to extract the two certificates and private key from the certs.txt and place them into the right file and location.

$ mkdir -p /opt/consul/agent-certs

$ grep -Pzo "(?s)(?<=certificate)[^\-]*.*?END CERTIFICATE[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/agent.crt

$ grep -Pzo "(?s)(?<=issuing_ca)[^\-]*.*?END CERTIFICATE[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/ca.crt

$ grep -Pzo "(?s)(?<=private_key)[^\-]*.*?END RSA PRIVATE KEY[^\n]*\n" certs.txt | sed 's/^\s*-/-/g' > /opt/consul/agent-certs/agent.key

Configure consul-template

When you created the certificate you used as a parameter ttl="24h" meaning that this certificates will be valid only for 24 hours before expiring.

Deciding the right trade-off for certificate lifespan is always a compromise between security and agility. A possible third way that does not require you to lower your security is to use consul-template to automate certificate renewal for Consul when the TTL is expired.

Create template files

You can instruct consul-template to generate and retrieve those files from Vault using the following templates:

agent.crt.tpl (server only)
{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.certificate }}
{{ end }}

The template will use the pki_int/issue/consul-dc1 endpoint that Vault exposes to generate new certificates. It also mentions the common name and alternate names for the certificate.

agent.key.tpl (server only)
{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}
{{ .Data.private_key }}
{{ end }}
ca.crt.tpl
{{ with secret "pki_int/issue/consul-dc1" "common_name=server.dc1.consul" "ttl=24h"}}
{{ .Data.issuing_ca }}
{{ end }}

Once you have created the templates for consul-template, you can copy the files into /opt/consul/templates.

$ cp *.tpl /opt/consul/templates/

Create consul-template configuration

For this tutorial, you are going to create a configuration file, consul_template.hcl, that will instruct consul-template to retrieve the files needed for the agent, client and server, to configure TLS encryption.

consul_template.hcl
# This denotes the start of the configuration section for Vault. All values
# contained in this section pertain to Vault.
vault {
  # This is the address of the Vault leader. The protocol (http(s)) portion
  # of the address is required.
  address      = "http://localhost:8200"

  # This value can also be specified via the environment variable VAULT_TOKEN.
  token        = "root"

  unwrap_token = false

  renew_token  = false
}

# This block defines the configuration for a template. Unlike other blocks,
# this block may be specified multiple times to configure multiple templates.
template {
  # This is the source file on disk to use as the input template. This is often
  # called the "consul-template template".
  source      = "agent.crt.tpl"

  # This is the destination path on disk where the source template will render.
  # If the parent directories do not exist, consul-template will attempt to
  # create them, unless create_dest_dirs is false.
  destination = "/opt/consul/agent-certs/agent.crt"

  # This is the permission to render the file. If this option is left
  # unspecified, consul-template will attempt to match the permissions of the
  # file that already exists at the destination path. If no file exists at that
  # path, the permissions are 0644.
  perms       = 0700

  # This is the optional command to run when the template is rendered. The
  # command will only run if the resulting template changes.
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "agent.key.tpl"
  destination = "/opt/consul/agent-certs/agent.key"
  perms       = 0700
  command     = "sh -c 'date && consul reload'"
}

template {
  source      = "ca.crt.tpl"
  destination = "/opt/consul/agent-certs/ca.crt"
  command     = "sh -c 'date && consul reload'"
}

The configuration file for the server contains the information to retrieve the CA certificate as well as the certificate/key pair for the server agent.

The following parameters are needed in order to allow consul-template to communicate with Vault:

  • address : the address of your Vault server. In this tutorial, Vault runs on the same node as Consul so you can use http://localhost:8200.

  • token : a valid Vault ACL token with appropriate permissions. You will use Vault root token for this tutorial.

Production check

Earlier in the lab you used a root token to log in to Vault. You will use that token in the next steps to generate the TLS certs. This is not recommended for production use; the recommended security approach is to create a new token based on a specific policy with limited privileges.

Start consul-template

After configuration is completed, you can start consul-template. You must provide the file with the -config parameter.

$ consul-template -config "consul_template.hcl"

Configuration reload triggered

Verify certificate rotation

The certificate you created manually for the Consul server had a TTL of 24 hours.

This means that after the certificate expires Vault will renew it and consul-template will update the files on your agent it reload Consul configuration automatically to make it pick up the new files.

You can verify the rotation by checking that consul-template keeps listing, every 24 hours, a timestamp and the log line:

Configuration reload triggered

You can also use openssl to verify the certificate content:

$ openssl x509 -text -noout -in /opt/consul/agent-certs/agent.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1b:2d:d6:5d:63:9b:aa:05:84:7b:be:3b:6f:e1:95:bb:1c:36:8c:a4
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=dc1.consul Intermediate Authority
        Validity
            Not Before: Sep 16 16:03:45 2020 GMT
            Not After : Sep 16 16:06:15 2020 GMT
        Subject: CN=server.dc1.consul
...

and verify that the Not Before and Not After values are being updated to reflect the new certificate.

Next steps

In this tutorial you learned how to configure Consul TLS encryption using Vault's PKI secrets engine to generate short lived certificates and using consul-template to automatically rotate them at the end of their lease time.

To learn how to configure gossip encryption for Consul using Vault as a secret storage management you can follow Secure Consul Gossip Communication with Vault tutorial.

To learn how to create Consul tokens using Vault you can follow Generate Consul Tokens with HashiCorp Vault tutorial.

Previous

Vault as Consul service mesh CA

 Next

Build your own certificate authority (CA)

This tutorial also appears in:

  • 11 tutorials
    Operate Consul
    Ensure Consul resilience with disaster recovery best practices.
    • Consul
  • 11 tutorials
    HashiCorp product integrations
    Vault can manage secrets associated with other HashiCorp products.
    • Vault