Migrate your services to service mesh with permissive mTLS

Applications typically consist of many interdependent services spread across independent teams. Traditionally, adopting Consul service mesh required teams to coordinate and migrate all the services to the mesh at once. Otherwise, with transparent proxy enabled, services outside of the mesh will not have the mTLS certificates required to securely communicate with services inside the mesh.

Permissive mTLS mode makes it easier for teams to onboard and migrate their services to Consul. Instead of deploying and configuring ingress gateways and API gateways, operators can temporarily let sidecar proxies allow both mTLS and non-mTLS traffic. This essentially allows any non-mesh services to communicate with services inside the mesh.

In this tutorial, you will:

• Deploy a Kubernetes cluster with Terraform
• Deploy Consul and a demo application on the cluster
• Connect non-mesh services to Consul services using permissive mTLS
• Migrate services to Consul and configuring intentions
• Re-secure the mesh by restricting permissive mTLS

Scenario overview

HashiCups is a coffee shop demo application. It has a microservices architecture and uses Consul service mesh to securely connect the services. At the beginning of this tutorial, the HashiCups backend (products-api and postgres) will be on Consul service mesh. First, you will use permissive mTLS to enable non-mesh traffic (public-api) to seamlessly access mesh services (products-api). Then, you will migrate the non-mesh services to Consul then disable permissive mTLS to harden your security posture.

HashiCups uses the following microservices:

• The nginx service is an NGINX instance that routes requests to the frontend service and serves as a reverse proxy to the public-api service.
• The frontend service provides a React-based UI.
• The public-api service is a GraphQL public API that communicates with the products-api and the payments services.
• The product-api service stores the core HashiCups application logic, including authentication, coffee (product) information, and orders.
• The postgres service is a Postgres database instance that stores user, product, and order information.
• The payments service is a gRPC-based Java application service that handles customer payments.

Prerequisites

If you are not familiar with Consul's core functionality, refer to the Consul Getting Started tutorials collection (VMs, Kubernetes) first.

For this tutorial, you will need:

• An AWS account configured for use with Terraform
• aws-cli v2.0 or later
• kubectl v1.21 or later
• git v2.0 or later
• terraform v1.2 or later
• consul-k8s v1.2.0 or later

Confirm you have the consul-k8s v1.2.0 or later.

$ consul-k8s version
  consul-k8s v1.2.0-rc1 (7f6b0ee)

Clone example repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-permissive-mtls.git

Change into the directory with the newly cloned repository.

$ cd learn-consul-permissive-mtls

This repository has the following:

• The Terraform configuration files in the root directory deploys an EKS cluster in us-east-2. You will deploy Consul and HashiCups (the demo application) onto this Kubernetes cluster.
• The k8s-yamls directory contains YAML configuration files that support this tutorial.
• The hashicups directory contains YAML configuration files for deploying HashiCups.

Deploy Consul and demo application

Initialize your Terraform configuration to download the necessary providers and modules.

$ terraform init
Initializing the backend...
## ...
Initializing provider plugins...
## ...
Terraform has been successfully initialized!
## ...

Then, create the infrastructure. Confirm the run by entering yes. This will take about 15 minutes to deploy your infrastructure.

$ terraform apply
## ...
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.

 Enter a value: yes

## ...

Apply complete! Resources: 59 added, 0 changed, 0 destroyed.

Outputs:

cluster_endpoint = "https://17AAEDA2B5B51F37CB79E4AD63A10BC5.gr7.us-east-2.eks.amazonaws.com"
cluster_id = "education-eks-z8SyM53Q"
cluster_name = "education-eks-z8SyM53Q"
cluster_security_group_id = "sg-034b0e35b4079283e"
region = "us-east-2"

Configure kubectl

Now that you have deployed the Kubernetes cluster, configure kubectl to interact with it.

$ aws eks update-kubeconfig \
  --region $(terraform output -raw region) \
  --name $(terraform output -raw cluster_name)

Deploy Consul

You will now deploy Consul on your Kubernetes cluster with consul-k8s. By default, Consul deploys into its own dedicated namespace (consul). The Consul installation will use the Consul Helm chart file in the k8s-yaml directory. Permissive mTLS requires Consul v1.16.0+. Deploying Consul on each Kubernetes cluster should only take a few minutes.

global:
  image: "hashicorp/consul:1.16.0-rc1"
  imageK8S: "hashicorp/consul-k8s-control-plane:1.2.0-rc1"

Deploy Consul and confirm the installation with a y.

$ consul-k8s install -config-file k8s-yamls/values.yaml
## ...
==> Checking if Consul can be installed
## ...
    Proceed with installation? (Y/n) Y
## ...
==> Installing Consul
 ✓ Downloaded charts.
## ...
 ✓ Consul installed in namespace "consul".

Verify that you have installed Consul by inspecting the Kubernetes pods in the consul namespace.

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-connect-injector-59b5b4fccd-j9hj8       1/1     Running   0          8m36s
consul-mesh-gateway-7b86b77d99-hhgpr           1/1     Running   0          8m36s
consul-server-0                                1/1     Running   0          8m36s
consul-webhook-cert-manager-57c5bb695c-6b4c5   1/1     Running   0          8m36s

Confirm that the Helm chart version is 1.2.0 or later.

$ helm list --namespace consul
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
consul  consul          1               2023-06-12 16:43:41.409139 -0700 PDT    deployed        consul-1.2.0-rc1        1.16.0-rc1

Deploy HashiCups

Deploy the HashiCups services.

$ for service in {frontend,nginx,public-api,payments,products-api,postgres,intentions-api-db}; do kubectl apply -f hashicups/$service.yaml; done
service/frontend created
serviceaccount/frontend created
servicedefaults.consul.hashicorp.com/frontend created
deployment.apps/frontend created
service/nginx created
serviceaccount/nginx created
servicedefaults.consul.hashicorp.com/nginx created
configmap/nginx-configmap created
deployment.apps/nginx created
service/public-api created
serviceaccount/public-api created
servicedefaults.consul.hashicorp.com/public-api created
deployment.apps/public-api created
service/payments created
serviceaccount/payments created
servicedefaults.consul.hashicorp.com/payments created
deployment.apps/payments created
service/products-api created
serviceaccount/products-api created
servicedefaults.consul.hashicorp.com/products-api created
configmap/db-configmap created
deployment.apps/products-api created
service/postgres created
serviceaccount/postgres created
servicedefaults.consul.hashicorp.com/postgres created
deployment.apps/postgres created
serviceintentions.consul.hashicorp.com/postgres created
serviceintentions.consul.hashicorp.com/deny-all created

Verify that you have deployed the services on Kubernetes.

$ kubectl get services
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
frontend       ClusterIP   172.20.170.161   <none>        3000/TCP   42s
kubernetes     ClusterIP   172.20.0.1       <none>        443/TCP    15m
nginx          ClusterIP   172.20.255.205   <none>        80/TCP     41s
payments       ClusterIP   172.20.222.231   <none>        1800/TCP   37s
postgres       ClusterIP   172.20.213.162   <none>        5432/TCP   34s
products-api   ClusterIP   172.20.19.151    <none>        9090/TCP   36s
public-api     ClusterIP   172.20.62.239    <none>        8080/TCP   39s

List the Consul services. Notice that only the backend services products-api and postgres are registered to Consul since they are currently the only HashiCups services within the service mesh.

$ kubectl exec --namespace consul -it consul-server-0 -- consul catalog services
Defaulted container "consul" out of: consul, locality-init (init)
consul
mesh-gateway
postgres
postgres-sidecar-proxy
products-api
products-api-sidecar-proxy

Open HashiCups in your browser. In a new terminal, port-forward the nginx service to port 8080.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. Notice that it displays no products, since the public-api cannot connect to the products-api.

Connect non-mesh services to Consul services

By default, Consul's sidecar proxies reject all non-mTLS traffic. This prevents services outside Consul from connecting to services inside the mesh unless you set up ingress or API gateways. However, setting up and configuring ingress and API gateways increase service mesh onboarding complexity. Permissive mTLS configures sidecar proxies to temporarily accept both incoming mTLS and incoming non-mTLS connections to simplify service mesh onboarding for your services.

In order to enable your sidecar proxies to accept both mTLS and non-mTLS traffic, you need to:

• Allow permissive mTLS at the mesh level: Configure the service mesh to allow the use of permissive mTLS with the mesh configuration entry.
• Enable permissive mTLS at the service level: Once you have configured your mesh to allow the use of permissive mTLS, set mTLS mode in the service-defaults configuration entry. This overrides the proxy-defaults setting.

Allow permissive mTLS at mesh level

The following file configures the Consul datacenter (partition) to allow permissive mTLS:

apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
  name: mesh
spec:
  allowEnablingPermissiveMutualTLS: true

Enable permissive mTLS by applying the mesh configuration entry.

$ kubectl apply -f k8s-yamls/mesh-config-entry.yaml --namespace consul
mesh.consul.hashicorp.com/mesh created

Set permissive mTLS at service level

The following file configures the products-api service to accept both mTLS and non-mTLS traffic:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: products-api
spec:
  protocol: http
  mutualTLSMode: "permissive"

Enable products-api to allow non-mTLS traffic.

$ kubectl apply -f k8s-yamls/service-defaults-products-api.yaml --namespace consul

You may need to re-forward the port of the nginx service.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. It should show a list of coffees.

Notice that even though there is a deny-all intention on the Consul datacenter, public-api was able to connect to products-api. Intentions only take effect for mTLS connections.

Migrate non-mesh services into Consul service mesh

Now, you will migrate the remaining HashiCups services to Consul service mesh.

First, the each service deployment definition, update the consul.hashicorp.com/connect-inject annotation from false to true. You will need to do this for four files:

1. hashicups/nginx.yaml
2. hashicups/frontend.yaml
3. hashicups/public-api.yaml
4. hashicups/payments.yaml

apiVersion: apps/v1
kind: Deployment
## ...
      annotations:
        consul.hashicorp.com/connect-inject: "true"

Once you have done this to all four files, apply the changes. In addition, you will apply a file that creates intentions between these services.

$ for service in {frontend,nginx,public-api,payments,intentions-new-services}; do kubectl apply -f hashicups/$service.yaml; done
service/frontend unchanged
serviceaccount/frontend unchanged
servicedefaults.consul.hashicorp.com/frontend unchanged
deployment.apps/frontend configured
service/nginx unchanged
serviceaccount/nginx unchanged
servicedefaults.consul.hashicorp.com/nginx unchanged
configmap/nginx-configmap unchanged
deployment.apps/nginx configured
service/public-api unchanged
serviceaccount/public-api unchanged
servicedefaults.consul.hashicorp.com/public-api unchanged
deployment.apps/public-api configured
service/payments unchanged
serviceaccount/payments unchanged
servicedefaults.consul.hashicorp.com/payments unchanged
deployment.apps/payments configured
serviceintentions.consul.hashicorp.com/public-api created
serviceintentions.consul.hashicorp.com/payments created
serviceintentions.consul.hashicorp.com/frontend created

Verify the services appear in Consul.

$ kubectl exec --namespace consul -it consul-server-0 -- consul catalog services
Defaulted container "consul" out of: consul, locality-init (init)
consul
frontend
frontend-sidecar-proxy
mesh-gateway
nginx
nginx-sidecar-proxy
payments
payments-sidecar-proxy
postgres
postgres-sidecar-proxy
products-api
products-api-sidecar-proxy
public-api
public-api-sidecar-proxy

Set up intentions

Create intentions between the public-api and products-api to allow traffic communication between the services.

$ kubectl apply -f hashicups/intentions-public-products-api.yaml --namespace consul
serviceintentions.consul.hashicorp.com/public-api created

Restrict permissive mTLS at service level

Restrict products-api to only accept mTLS traffic.

In k8s-yamls/service-defaults-products-api.yaml, update mutualTLSMode to strict.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: products-api
spec:
  protocol: http
  mutualTLSMode: "strict"

Then, apply the changes.

$ kubectl apply -f k8s-yamls/service-defaults-products-api.yaml --namespace consul
servicedefaults.consul.hashicorp.com/products-api configured

Restrict permissive mTLS at mesh level

In k8s-yamls/mesh-config-entry.yaml, update allowEnablingPermissiveMutualTLS to false.

apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
  name: mesh
spec:
  allowEnablingPermissiveMutualTLS: false

Disallowing permissive mTLS at the mesh level does not impact permissive mTLS settings at the service level (proxy-defaults and service-defaults, MutalTLSMode=permissive). This lets organizations disallow permissive mTLS moving forward, while services currently in permissive mode can eventually migrate to the mesh at their own pace.

Then, apply the changes.

$ kubectl apply -f k8s-yamls/mesh-config-entry.yaml --namespace consul
mesh.consul.hashicorp.com/mesh configured

Confirm that the HashiCups services can still connect to each other. You may need to re-forward the port of the nginx service.

$ kubectl port-forward deploy/nginx 8080:80

Open localhost:8080 in your browser to view the HashiCups UI. It should show a list of coffees.

Clean up environment

To destroy the environment, first uninstall Consul from your Kubernetes cluster. Confirm with a y.

$ consul-k8s uninstall
##...
    Proceed with uninstall? (y/N) y
##...
   Only approve if all data from this installation can be deleted. (y/N) y
##...

Then, destroy the supporting infrastructure Enter yes to confirm the destroy operation.

$ terraform destroy
##...
Plan: 0 to add, 0 to change, 59 to destroy.
##...
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

##...
Destroy complete! Resources: 59 destroyed.

Next steps

In this tutorial, you used the Consul permissive mTLS to let Consul services accept both non-mTLS and mTLS traffic. In the process, you learned how to seamlessly connect non-service mesh traffic to Consul service mesh and migrate services to Consul.

Feel free to explore these tutorials and collections to learn more about Consul service mesh, microservices, and Kubernetes security.

• Onboard services with permissive mTLS mode enabled documentation
• Consul Kubernetes Deployment Guide
• Migrate to Microservices collection
• Consul Kubernetes Security