Consul on Kubernetes reference architecture

Preparing your Kubernetes environment to successfully deploy and run Consul is an important first step in your production deployment process. In this tutorial, you will prepare your Kubernetes cluster, which can be running on any platform (AKS, EKS, GKE, etc). However, we will call out cloud specific differences when applicable. Before starting this tutorial you should have experience with Kubernetes, and have kubectl and Helm configured locally.

By the end of this tutorial, you will be able to select the right resource limits for Consul pods, select the Consul datacenter design that meets your use case, and understand the minimum networking requirements.

Infrastructure requirements

Consul server agents are responsible for the datacenter state, responding to RPC queries, and processing all write operations. Since the Consul servers are highly active, and are responsible for maintaining the datacenter state, server sizing is critical for the overall performance, efficiency, and health of the Consul datacenter. Review the Consul Reference Architecture tutorial for sizing recommendations for small and large Consul datacenters.

The CPU and memory recommendations can be used when you select the resources limits for the Consul pods. The disk recommendations can also be used when selecting the resources limits and configuring persistent volumes. You will need to set both limits and requests in the Helm chart. Below is an example snippet of Helm configuration for a Consul server in a large environment.

server:
  resources: |
    requests:
      memory: "32Gi"
      cpu: "4"
    limits:
      memory: "32Gi"
      cpu: "4"

  storage: 50Gi
# ...

You should also set resource limits for Consul clients, so that the client pods do not unexpectedly consume more resources than anticipated.

Persistent volumes (PV) allow you to have a fixed disk location for your Consul data. This ensures that if a Consul server is lost, the data will still be available. This is an important feature of Kubernetes, but may take some additional configuration. If you are running Kubernetes on one of the major cloud platforms, persistent volumes should already be configured for you; be sure to read their documentation for more details. If you are setting up the persistent volumes resource in Kubernetes, you may need to map the Consul server to that volume with the storage class parameter.

Finally, you will need to enable RBAC on your Kubernetes cluster. Review the Kubernetes RBAC documentation. You should also review RBAC and authentication documentation if your Kubernetes cluster is running on a major cloud platform.

• AWS.
• GCP.
• Azure. In Azure, RBAC is enabled by default.

Datacenter design

There are many possible configurations for running Consul with Kubernetes. In this tutorial we will cover three of the most common.

1. Consul server and client agents can be deployed entirely within Kubernetes.
2. Consul servers can be deployed outside of Kubernetes and clients inside of Kubernetes.
3. Multiple Consul datacenters with agents inside and outside of Kubernetes.

Review the Consul Kubernetes-specific documentation for additional use case information.

Since all three use cases also use catalog sync, you should review the implementation details for catalog sync.

Consul datacenter deployed in Kubernetes

Deploying a Consul datacenter, which consists of both Consul servers and clients, in Kubernetes is done with the official Helm chart. This configuration is useful for managing services within Kubernetes, and is common for users who do not already have a production Consul datacenter.

A Consul datacenter in Kubernetes functions the same as a platform independent Consul datacenter, such as Consul datacenters deployed on bare metal servers or virtual machines. Agents communicate over LAN gossip, servers participate in the Raft consensus, and client requests are forwarded to the servers via RPCs.

Consul datacenter with a Kubernetes cluster

To use an existing Consul datacenter to manage services in Kubernetes, Consul clients are deployed within the Kubernetes cluster. This enables Kubernetes-defined services to be synced to Consul. This design allows Consul tools such as envconsul, consul-template, and more to work on Kubernetes.

This type of deployment in Kubernetes is also facilitated by using the official Helm chart.

Multiple Consul datacenters with a Kubernetes cluster

Multiple Consul datacenters can be joined by WAN links. The datacenters can operate independently and only communicate over the WAN. This type of datacenter design is detailed in the Reference Architecture tutorial. In this setup, you have a Consul datacenter running outside of Kubernetes and a Consul datacenter running inside of Kubernetes.

Catalog sync

To use catalog sync, you must enable it in the Helm chart. Catalog sync allows you to sync services between Consul and Kubernetes. The sync can be unidirectional in either direction or bidirectional. Read the documentation to learn more about the configuration options.

Services synced from Kubernetes to Consul are discoverable, like any other service within the Consul datacenter. Read more in the network connectivity section to learn more about related Kubernetes configuration. Services synced from Consul to Kubernetes are discoverable with the built-in Kubernetes DNS once a Consul stub domain is deployed. When bidirectional catalog sync is enabled, it behaves like the two unidirectional setups.

Network connectivity

When running Consul as a pod inside of Kubernetes, the Consul servers are automatically configured with the appropriate addresses. However, when running Consul servers outside the Kubernetes cluster and clients inside Kubernetes as pods, there are additional networking considerations.

Network connectivity for services

When using Consul catalog sync, to sync Kubernetes services to Consul, you must ensure the Kubernetes services are supported service types and configured correctly in Kubernetes. If the service is configured correctly, it is discoverable by Consul like any other service in the datacenter.

Network security

Finally, you should consider securing your Consul datacenter with ACLs, gossip and TLS encryption.

Next steps

You are now prepared to deploy Consul with Kubernetes. In this tutorial, you were introduced to several datacenter designs for a variety of use cases. This tutorial also outlined the Kubernetes prerequisites, resource requirements for Consul, and networking considerations. Continue onto the Deploying Consul with Kubernetes tutorial for information on deploying Consul with the official Helm chart.