Service Mesh Certificate Authority Overview

Service mesh certificate management is done centrally through the Consul servers using the configured service mesh CA (Certificate Authority) provider. A CA provider manages root and intermediate certificates and performs certificate signing operations. The Consul leader orchestrates CA provider operations as necessary, such as when a service needs a new certificate or during CA rotation events.

The CA provider abstraction enables Consul to support multiple systems for storing and signing certificates. Consul ships with a built-in CA which generates and stores the root certificate and private key on the Consul servers. Consul also has support for using Vault as a CA. With Vault, the root certificate and private key material remain with the Vault cluster.

CA and Certificate relationship

This diagram shows the relationship between the CA certificates in a Consul primary datacenter and a secondary Consul datacenter.

CA relationship

Leaf certificates are created for two purposes:

the Leaf Cert Service is used by envoy proxies in the mesh to perform mTLS with other services.
the Leaf Cert Client Agent is created by auto-encrypt and auto-config. It is used by client agents for HTTP API TLS, and for mTLS for RPC requests to servers.

Any secondary datacenters use their CA provider to generate an intermediate certificate signing request (CSR) to be signed by the primary root CA. They receive an intermediate CA certificate, which is used to sign leaf certificates in the secondary datacenter.

You can use different providers across primary and secondary datacenters. For example, an operator may use a Vault CA provider for extra security in the primary datacenter but choose to use the built-in CA provider in the secondary datacenter, which may not have a reachable Vault cluster. The following table compares the built-in and Vault providers.

CA Provider Comparison

	Consul built-in	Vault
Security	CA private keys are stored on disk	CA private keys are stored in Vault and are never exposed to Consul server agents
Resiliency	No dependency on external systems. If Consul is available, it can sign certificates	Dependent on Vault availability
Latency	Consul signs certificates locally	A network call to Vault is required to sign certificates

CA Bootstrapping

CA initialization happens automatically when a new Consul leader is elected as long as service mesh is enabled, and the CA system has not already been initialized. This initialization process will generate the initial root certificates and setup the internal Consul server state.

For the initial bootstrap, the CA provider can be configured through the Agent configuration. After initialization, the CA can only be updated through the Update CA Configuration API endpoint. If a CA is already initialized, any changes to the CA configuration in the agent configuration file (including removing the configuration completely) will have no effect.

If no specific provider is configured when service mesh is enabled, the built-in Consul CA provider will be used and a private key and root certificate will be generated automatically.

Viewing Root Certificates

Root certificates can be queried with the list CA Roots endpoint. With this endpoint, you can see the list of currently trusted root certificates. When a cluster first initializes, this will only list one trusted root. Multiple roots may appear as part of rotation.

$ curl http://localhost:8500/v1/connect/ca/roots
{
    "ActiveRootID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
    "TrustDomain": "36cb52cd-4058-f811-0432-6798a240c5d3.consul",
    "Roots": [
        {
            "ID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
            "Name": "Consul CA Root Cert",
            "SerialNumber": 7,
            "SigningKeyID": "19:45:8b:30:a1:45:84:ae:23:52:db:8d:1b:ff:a9:09:db:fc:2a:72:39:ae:da:11:53:f4:37:5c:de:d1:68:d8",
            "ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
            "NotBefore": "2018-06-06T17:35:25Z",
            "NotAfter": "2028-06-03T17:35:25Z",
            "RootCert": "-----BEGIN CERTIFICATE-----\nMIICmDCCAj6gAwIBAgIBBzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM1MjVaFw0yODA2MDMxNzM1MjVaMBYxFDASBgNVBAMT\nC0NvbnN1bCBDQSA3MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgo09lpx63bHw\ncSXeeoSpHpHgyzX1Q8ewJ3RUg6Ie8Howbs/QBz1y/kGxsF35HXij3YrqhgQyPPx4\nbQ8FH2YR4aOCAXswggF3MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/\nMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1MjpkYjo4ZDox\nYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDozNzo1Yzpk\nZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToy\nMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1\nMzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6Ly8zNmNi\nNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMD0GA1UdHgEB\n/wQzMDGgLzAtgiszNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMu\nY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIHl6UDdouw8Fzn/oDHputAxt3UFbVg/U\nvC6jWPuqqMwmAiEAkvMadtwjtNU7m/AQRJrj1LeG3eXw7dWO8SlI2fEs0yY=\n-----END CERTIFICATE-----\n",
            "IntermediateCerts": null,
            "Active": true,
            "PrivateKeyType": "",
            "PrivateKeyBits": 0,
            "CreateIndex": 8,
            "ModifyIndex": 8
        }
    ]
}

CA Configuration

After initialization, the CA provider configuration can be viewed with the Get CA Configuration API endpoint. Consul will filter sensitive values from this endpoint depending on the provider in use, so the configuration may not be complete.

$ curl http://localhost:8500/v1/connect/ca/configuration
{
    "Provider": "consul",
    "Config": {
        "LeafCertTTL": "72h",
        "IntermediateCertTTL": "8760h"
    },
    "CreateIndex": 5,
    "ModifyIndex": 5
}

The CA provider can be reconfigured using the Update CA Configuration API endpoint. Specific options for reconfiguration can be found in the specific CA provider documentation in the sidebar to the left.

Root Certificate Rotation

Whenever the CA's configuration is updated in a way that causes the root key to change, a special rotation process will be triggered in order to smoothly transition to the new certificate. This rotation is automatically orchestrated by Consul.

If the current CA Provider doesn't support cross-signing, this process can't be followed. See Forced Rotation Without Cross-Signing.

This also automatically occurs when a completely different CA provider is configured (since this changes the root key). Therefore, this automatic rotation process can also be used to cleanly transition between CA providers. For example, updating the service mesh to use Vault instead of the built-in CA.

During rotation, an intermediate CA certificate is requested from the new root, which is then cross-signed by the old root. This cross-signed certificate is then distributed alongside any newly-generated leaf certificates used by the proxies once the new root becomes active, and provides a chain of trust back to the old root certificate in the event that a certificate signed by the new root is presented to a proxy that has not yet updated its bundle of trusted root CA certificates to include the new root.

After the cross-signed certificate has been successfully generated and the new root certificate or CA provider has been set up, the new root becomes the active one and is immediately used for signing any new incoming certificate requests.

If we check the list CA roots endpoint after updating the configuration with a new root certificate, we can see both the old and new root certificates are present, and the currently active root has an intermediate certificate which has been generated and cross-signed automatically by the old root during the rotation process:

$ curl localhost:8500/v1/connect/ca/roots
{
    "ActiveRootID": "d2:2c:41:94:1e:50:04:ea:86:fc:08:d6:b0:45:a4:af:8a:eb:76:a0",
    "TrustDomain": "36cb52cd-4058-f811-0432-6798a240c5d3.consul",
    "Roots": [
        {
            "ID": "31:6c:06:fb:49:94:42:d5:e4:55:cc:2e:27:b3:b2:2e:96:67:3e:7e",
            "Name": "Consul CA Root Cert",
            "SerialNumber": 7,
            "SigningKeyID": "19:45:8b:30:a1:45:84:ae:23:52:db:8d:1b:ff:a9:09:db:fc:2a:72:39:ae:da:11:53:f4:37:5c:de:d1:68:d8",
            "ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
            "NotBefore": "2018-06-06T17:35:25Z",
            "NotAfter": "2028-06-03T17:35:25Z",
            "RootCert": "-----BEGIN CERTIFICATE-----\nMIICmDCCAj6gAwIBAgIBBzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM1MjVaFw0yODA2MDMxNzM1MjVaMBYxFDASBgNVBAMT\nC0NvbnN1bCBDQSA3MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgo09lpx63bHw\ncSXeeoSpHpHgyzX1Q8ewJ3RUg6Ie8Howbs/QBz1y/kGxsF35HXij3YrqhgQyPPx4\nbQ8FH2YR4aOCAXswggF3MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/\nMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1MjpkYjo4ZDox\nYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDozNzo1Yzpk\nZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToy\nMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1\nMzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6Ly8zNmNi\nNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMD0GA1UdHgEB\n/wQzMDGgLzAtgiszNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMu\nY29uc3VsMAoGCCqGSM49BAMCA0gAMEUCIHl6UDdouw8Fzn/oDHputAxt3UFbVg/U\nvC6jWPuqqMwmAiEAkvMadtwjtNU7m/AQRJrj1LeG3eXw7dWO8SlI2fEs0yY=\n-----END CERTIFICATE-----\n",
            "IntermediateCerts": null,
            "Active": false,
            "PrivateKeyType": "",
            "PrivateKeyBits": 0,
            "CreateIndex": 8,
            "ModifyIndex": 24
        },
        {
            "ID": "d2:2c:41:94:1e:50:04:ea:86:fc:08:d6:b0:45:a4:af:8a:eb:76:a0",
            "Name": "Consul CA Root Cert",
            "SerialNumber": 16238269036752183483,
            "SigningKeyID": "",
            "ExternalTrustDomain": "a1499528-fbf6-df7b-05e5-ae81e1873fc4",
            "NotBefore": "2018-06-06T17:37:03Z",
            "NotAfter": "2028-06-03T17:37:03Z",
            "RootCert": "-----BEGIN CERTIFICATE-----\nMIIDijCCAnKgAwIBAgIJAOFZ66em1qC7MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV\nBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMRIwEAYDVQQKDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0x\nODA2MDYxNzM3MDNaFw0yODA2MDMxNzM3MDNaMGIxCzAJBgNVBAYTAlVTMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRIwEAYDVQQK\nDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAK6ostXN6W093EpI3RDNQDwC1Gq3lPNoodL5XRaVVIBU\n3X5iC+Ttk02p67cHUguh4ZrWr3o3Dzxm+gKK0lfZLW0nNYNPAIGZWQD9zVSx1Lqt\n8X0pd+fhMV5coQrh3YIG/vy17IBTSBuRUX0mXOKjOeJJlrw1HQZ8pfm7WX6LFul2\nXszvgn5K1XR+9nhPy6K2bv99qsY0sm7AqCS2BjYBW8QmNngJOdLPdhyFh7invyXe\nPqgujc/KoA3P6e3/G7bJZ9+qoQMK8uwD7PxtA2hdQ9t0JGPsyWgzhwfBxWdBWRzV\nRvVi6Yu2tvw3QrjdeKQ5Ouw9FUb46VnTU7jTO974HjkCAwEAAaNDMEEwPwYDVR0R\nBDgwNoY0c3BpZmZlOi8vMzZjYjUyY2QtNDA1OC1mODExLTA0MzItNjc5OGEyNDBj\nNWQzLmNvbnN1bDANBgkqhkiG9w0BAQsFAAOCAQEATHgCro9VXj7JbH/tlB6f/KWf\n7r98+rlUE684ZRW9XcA9uUA6y265VPnemsC/EykPsririoh8My1jVPuEfgMksR39\n9eMDJKfutvSpLD1uQqZE8hu/hcYyrmQTFKjW71CfGIl/FKiAg7wXEw2ljLN9bxNv\nGG118wrJyMZrRvFjC2QKY025QQSJ6joNLFMpftsZrJlELtRV+nx3gMabpiDRXhIw\nJM6ti26P1PyVgGRPCOG10v+OuUtwe0IZoOqWpPJN8jzSuqZWf99uolkG0xuqLNz6\nd8qvTp1YF9tTmysgvdeGALez/02HTF035RVTsQfH9tM/+4yG1UnmjLpz3p4Fow==\n-----END CERTIFICATE-----",
            "IntermediateCerts": [
                "-----BEGIN CERTIFICATE-----\nMIIDTzCCAvWgAwIBAgIBFzAKBggqhkjOPQQDAjAWMRQwEgYDVQQDEwtDb25zdWwg\nQ0EgNzAeFw0xODA2MDYxNzM3MDNaFw0yODA2MDMxNzM3MDNaMGIxCzAJBgNVBAYT\nAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2Nv\nMRIwEAYDVQQKDAlIYXNoaUNvcnAxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJ\nKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK6ostXN6W093EpI3RDNQDwC1Gq3lPNo\nodL5XRaVVIBU3X5iC+Ttk02p67cHUguh4ZrWr3o3Dzxm+gKK0lfZLW0nNYNPAIGZ\nWQD9zVSx1Lqt8X0pd+fhMV5coQrh3YIG/vy17IBTSBuRUX0mXOKjOeJJlrw1HQZ8\npfm7WX6LFul2Xszvgn5K1XR+9nhPy6K2bv99qsY0sm7AqCS2BjYBW8QmNngJOdLP\ndhyFh7invyXePqgujc/KoA3P6e3/G7bJZ9+qoQMK8uwD7PxtA2hdQ9t0JGPsyWgz\nhwfBxWdBWRzVRvVi6Yu2tvw3QrjdeKQ5Ouw9FUb46VnTU7jTO974HjkCAwEAAaOC\nARswggEXMGgGA1UdDgRhBF8xOTo0NTo4YjozMDphMTo0NTo4NDphZToyMzo1Mjpk\nYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpkYToxMTo1MzpmNDoz\nNzo1YzpkZTpkMTo2ODpkODBqBgNVHSMEYzBhgF8xOTo0NTo4YjozMDphMTo0NTo4\nNDphZToyMzo1MjpkYjo4ZDoxYjpmZjphOTowOTpkYjpmYzoyYTo3MjozOTphZTpk\nYToxMTo1MzpmNDozNzo1YzpkZTpkMTo2ODpkODA/BgNVHREEODA2hjRzcGlmZmU6\nLy8zNmNiNTJjZC00MDU4LWY4MTEtMDQzMi02Nzk4YTI0MGM1ZDMuY29uc3VsMAoG\nCCqGSM49BAMCA0gAMEUCIBp46tRDot7GFyDXu7egq7lXBvn+UUHD5MmlFvdWmtnm\nAiEAwKBzEMcLd5kCBgFHNGyksRAMh/AGdEW859aL6z0u4gM=\n-----END CERTIFICATE-----\n"
            ],
            "Active": true,
            "PrivateKeyType": "",
            "PrivateKeyBits": 0,
            "CreateIndex": 24,
            "ModifyIndex": 24
        }
    ]
}

The old root certificate will be automatically removed once enough time has elapsed for any leaf certificates signed by it to expire.

Forced Rotation Without Cross-Signing

If the CA provider that is currently in use does not support cross-signing, then attempts to change the root key or CA provider will fail. This is to ensure operators don't make the change without understanding that there is additional risk involved.

It is possible to force the change to happen anyway by setting the ForceWithoutCrossSigning field in the CA configuration to true.

The downside is that all new certificates will immediately start being signed with the new root key, but it will take some time for agents throughout the cluster to observe the root CA change and reconfigure applications and proxies to accept certificates signed by this new root. This will mean connections made with a new certificate may fail for a short period after the CA change.

Typically all connected agents will have observed the new roots within seconds even in a large deployment so the impact should be contained. But it is possible for a disconnected, overloaded or misconfigured agent to not see the new root for an unbounded amount of time during which new connections to services on that host will fail. The issue will resolve as soon as the agent can reconnect to servers.

Currently both Consul and Vault CA providers do support cross signing. As more providers are added this documentation will list any that this section applies to.

Recovering From Expired Certificates

If the built-in CA provider is misconfigured or unavailable, Consul service mesh requests eventually stop functioning due to expiration of intermediate and root certificates. To recover manually, use the CLI helper to generate CA certificates.

Example - Regenerating the built in CA

$ consul tls ca create -cluster-id test -common-name "Consul Agent CA" -days=365 -domain consul
  ==> Saved consul-agent-ca.pem
  ==> Saved consul-agent-ca-key.pem

The example above generates a new CA with a validity of 365 days. The cluster-id argument is specific to each cluster and can be looked up by examining the TrustDomain field in the List CA Roots endpoint.

The contents of the generated cert and private key files from the above step should then be used with the Update CA Configuration endpoint. Once the CA configuration is updated on the primary datacenter, all secondary datacenters will pick up the changes and regenerate their intermediate and leaf certificates, after which any new requests that require certificate verification will succeed.