Manage agent mTLS encryption

This topic describes mutual TLS (mTLS) encryption between agents in a Consul datacenter.

This mTLS encryption is distinct from Consul service mesh mTLS encryption. To learn about encrypted mTLS communication between services, refer to service mesh certificate authority.

Benefits of TLS encryption

Consul supports TLS certificates that verify the authenticity of servers and clients to secure RCP, gRPC, and HTTP traffic.

Implementing TLS encryption in Consul datacenters improves your deployment's security in the following ways:

Encrypt data in transit: TLS encrypts data transmitted between Consul datacenter nodes and user interfaces such as the UI or CLI to ensure that sensitive information is not exposed to unauthorized parties.
Agent authentication: TLS ensures that only verified parties can communicate with each other using certificates. This security measure prevents unauthorized nodes, services, and users from interacting with your Consul datacenter.
Prevent man-in-the-middle (MITM) attacks: Without TLS, attackers may intercept and change communications within your Consul deployments. TLS mitigates this risk by encrypting data and verifying the identity of the communication participants.
Comply with security regulations and standards: Compliance frameworks and regulations like PCI-DSS and HIPAA mandate the encryption of data in transit, which makes TLS a requirement for Consul deployments in regulated environments.

Mutual TLS (mTLS) requires that all clients and servers have key pairs that are generated by a single Certification Authority (CA). We recommend using a private CA that is not shared with other applications.

The following parameters in agent configuration files define the agent verification behavior:

Workflow

The process to enable TLS encryption in Consul deployments consists of the following steps:

Initialize the built-in CA used to sign all certificates.
Create server certificates to secure Consul servers.
Configure server agents for TLS encryption.
Start server agents.
Configure Consul client agents.
Start client agents.

Initialize the built-in CA

The first step to configure mTLS for Consul is to initialize the certificate authority (CA) that signs the certificates. To prevent unauthorized datacenter access, Consul requires that all certificates are signed by the same CA. We recommend using a private CA because any certificate it signs will be allowed to communicate with the Consul datacenter.

Consul supports a variety of tools for creating and managing your own CA, like the PKI secrets engine in Vault or the Terraform TLS provider. Consul also ships with built-in TLS tools to create and manage certificates.

If you have the Consul binary installed on your path, you can create the CA and certificates, even before you start a Consul server agent.

$ consul tls ca create -domain=consul
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

The command generates two files, <domain>-agent-ca.pem and <domain>-agent-ca-key.pem. In this example, the domain used to generate the certificates is the default one, consul.

The CA certificate, consul-agent-ca.pem, contains the public key necessary to validate Consul certificates. You must distribute this certificate to every node where a Consul agent runs.

The CA key, consul-agent-ca-key.pem, signs certificates for Consul nodes. You must keep this key private. Possession of this key allows anyone to run Consul as a trusted server or generate new valid certificates for the datacenter. Malicious actors may use this key to obtain access to all Consul data, including ACL tokens.

Create server certificates

To authenticate Consul servers, servers are provided with a special certificate that lists server.<domain>.<datacenter> in the Common Name and Subject Alternative Name fields. When you enable tls.defaults.verify_server_hostname, only agents that provide this certificate are allowed to boot as a server.

Without tls.defaults.verify_server_hostname = true, an attacker who compromises a Consul client agent can restart the agent as a server to get access to all the data in your datacenter. You must keep the server key private to protect your Consul data.

The following example creates a server certificate for datacenter dc1 in the consul domain. If your datacenter or domain is different, modify the command to use the appropriate flag values.

$ consul tls cert create -server -dc=dc1 -domain=consul
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

Repeat this process on the same node where you created the CA until the number of certificates is equal to the number of servers in the datacenter. You can run the command multiple times in a row, and it automatically increases the certificate and key numbers each time.

Federated Consul datacenter

A federated Consul environment requires the server certificate to include the names of all Consul datacenters that are within the federated environment.

Use the -additional-dnsname flag to provide an additional DNS names for Subject Alternative Names. localhost is always included. You can provide this flag multiple times in a single command.

The following example creates a certificate for a federated environment containing two Consul datacenters named dc1 and dc2.

$ consul tls cert create -server -dc dc1 -domain=consul -additional-dnsname="server.dc2.consul"
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

Configure server agents

After you generate the server certificates, distribute them to the Consul servers and modify the agent configuration file to include their local location.

Copy the following files to each Consul server:

consul-agent-ca.pem: CA public certificate.
dc1-server-consul-0.pem: Consul server node public certificate for first server node in the dc1 datacenter in the consul domain.
dc1-server-consul-0-key.pem: Consul server node private key for first server node in the dc1 datacenter in the consul domain.

There are two methods for configuring Consul server agents depending on the way you want to distribute certificates to the client agents:

The auto encryption method uses the Consul Connect CA to generate client certificates and then automatically distributes them to all Consul clients.
The operator method requires you to manually generate client certificates and distribute them to each client agent individually.

We recommend using the auto-encryption method with the built-in CA because Consul can then automatically rotate certificates without requiring operator intervention.

Use the operator method if you need to use a third-party CA or need more fine-grained control over certificate management.

addresses = {
  https = "0.0.0.0"
}
ports {
  https = 8501
}
tls {
  defaults {
    ca_file = "consul-agent-ca.pem"
    cert_file = "dc1-server-consul-0.pem"
    key_file = "dc1-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
    verify_server_hostname = true
  }
}
auto_encrypt {
  allow_tls = true
}

addresses = {
  https = "0.0.0.0"
}
ports {
  https = 8501
}
tls {
  defaults {
    ca_file = "consul-agent-ca.pem"
    cert_file = "dc1-server-consul-0.pem"
    key_file = "dc1-server-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
    verify_server_hostname = true
  }
}

Consul does not enable TLS for HTTP unless the https port is assigned a port number greater than 0. We recommend using 8501, the default number for the https port, because this default is designed to work automatically with other tools.

Start Consul servers

After you configure your servers, start the Consul process.

$ systemctl start consul

Your Consul servers can now communicate with TLS encryption for RPC and consensus.

Configure client agents

Next, configure your client agents with the same method you used to configure the server agent

The only file you need on the local disk to configure the Consul client agents using auto-encryption is the CA certificate consul-agent-ca-.pem.

addresses = {
  https = "0.0.0.0"
}
ports {
  https = 8501
}
tls {
  defaults {
    ca_file = "consul-agent-ca.pem"
    verify_incoming = true
    verify_outgoing = true
    verify_server_hostname = true
  }
}
auto_encrypt = {
  tls = true
}

On the node where you created the CA and server certificates, generate a client certificate with consul tls cert create -client command.

$ consul tls cert create -client -dc=dc1 -domain=consul
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem

The client certificate is also signed by the same CA used for the server certificates, but it does not contain server.<domain>.<datacenter> in the Common Name and in the Subject Alternative Name. Because verify_server_hostname is enabled, a compromised client cannot use this certificate to start as a server and access datacenter data.

Distribute the client certificates and the CA certificate consul-agent-ca.pem to every Consul client in the datacenter. Then add them to the client agent configuration.

addresses = {
  https = "0.0.0.0"
}
ports {
  https = 8501
}
tls {
  defaults {
    ca_file = "consul-agent-ca.pem"
    cert_file = "dc1-client-consul-0.pem"
    key_file = "dc1-client-consul-0-key.pem"
    verify_incoming = true
    verify_outgoing = true
    verify_server_hostname = true
  }
}

Start Consul clients

After you configure each client, start the Consul process on the node.

$ systemctl start consul

Your client agents now communicate using mutual TLS encryption.

Rotate TLS certificates

To maintain the security offered by TLS encryption, we recommend that you rotate TLS certificates often.

TLS certificates are part of Consul's reloadable configuration, so you do not need to restart the Consul agents when you renew certificates. As a result, there is no risk of downtime.

To rotate certificates for Consul server agents complete the following steps:

Generate new certificates for all server agents to replace the old ones.
Distribute the new certificates to the server nodes.
Reload Consul configuration on each server with the consul reload command.

To rotate certificates for Consul client agents complete the following steps:

When using the auto-encryption method, Consul automatically rotates the client certificates without operator intervention.

API, CLI, and UI interactions

The configuration snippets provided in this page are valid to configure complete mTLS for your Consul datacenter. This means that all interfaces require the client to provide a valid certificate in order to communicate with the Consul agent. This is valid for all requests, API, CLI, and UI.

Since Consul v1.12 it is possible to have different settings for:

the HTTP protocol, used for the Consul's REST API, the CLI integration, and the UI
the RPC protocol, used for internal communications between the Consul agents.

Interact with Consul without a client certificate

If you want to avoid the need to present a valid client certificate every time you interact with Consul using the HTTP API, CLI, or UI, configure Consul to trust all incoming HTTPS connections by setting tls.https.verify_incoming to false. RPC communications are still mTLS encrypted.

addresses = {
  https = "0.0.0.0"
}
ports {
  https = 8501
  http  = -1
}
tls {
  https {
    ca_file   = "/etc/consul.d/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/consul-agent.pem"
    key_file  = "/etc/consul.d/consul-agent-key.pem"
    verify_incoming = false
    verify_outgoing = true
  }
  internal_rpc {
    ca_file   = "/etc/consul.d/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/consul-agent.pem"
    key_file  = "/etc/consul.d/consul-agent-key.pem"
    verify_incoming        = true
    verify_outgoing        = true
    verify_server_hostname = true
  }
}

Use HTTP for local client interaction

If you want to avoid the need to present a valid client certificate or to use the CA certificate every time you interact with the local Consul agent, configure Consul to keep the HTTP listener active on the localhost interface only and set tls.https.verify_incoming to false. External requests to the API or UI are still protected by TLS encryption, but requests that originate locally do not need to present a client certificate. RPC communications are still mTLS encrypted.

addresses = {
  https = "0.0.0.0"
  http  = "127.0.0.1"
}
ports {
  https = 8501
  http  = 8500
}
tls {
  https {
    ca_file   = "/etc/consul.d/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/consul-agent.pem"
    key_file  = "/etc/consul.d/consul-agent-key.pem"
    verify_incoming = false
    verify_outgoing = true
  }
  internal_rpc {
    ca_file   = "/etc/consul.d/consul-agent-ca.pem"
    cert_file = "/etc/consul.d/consul-agent.pem"
    key_file  = "/etc/consul.d/consul-agent-key.pem"
    verify_incoming        = true
    verify_outgoing        = true
    verify_server_hostname = true
  }
}

Next steps

Your agents are now configured with mTLS for encrypted communication. With the auto encryption method, your client certificates are managed by the server. With the operator method, you distributed all the certificates manually, but have a more flexible configuration.

Documentation for the commands used in this topic is available at Consul agent configuration - TLS configuration reference and the consul tls CLI command reference.

To learn how to automate TLS certificate generation and rotation, refer to the Generate mTLS Certificates for Consul with Vault tutorial.

To continue securing your Consul deployment, add gossip encryption and enable Access Control Lists (ACLs) with the default deny policy.