HashiCorp Developer AI Private beta now available Sign up today
Consul
Service Mesh Security

Control traffic communication between services with intentions

  • 26min
  • |
  • Consul
  • Interactive

Workflow

In a Consul service mesh scenario you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally.

These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.

This removes the need to generate ad-hoc firewall rules for each service instance and moves the focus over service identity.

Consul uses service intentions to further limit the traffic only to the services explicitly allowed to communicate. This limits the surface exposed by the single service and constitutes one extra step towards zero-trust security.

In this tutorial, you will learn how to tune service access by using intentions.

Specifically, you will:

  • Understand the ACL permissions required to manipulate intentions
  • List the current intentions in your Consul service mesh
  • Change a service definition to add a new upstream
  • Apply intentions to your Consul datacenter to prevent unwanted traffic

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh, with Envoy sidecar proxies running alongside each service, and an API Gateway configured to permit access to the NGINX service.

The Consul service mesh is initially configured with an allow-all intention that enables every service to communicate with each other.

The only requirement for a service to service communication, in this scenario, is the presence of an upstream in the source service definition.

Architecture Initial scenario Architecture Initial scenario

While this can be acceptable in the onboarding phase of your service mesh, in a production environment you want to transition to a deny-all default intention, where only the necessary communications are explicitly allowed.

In this scenario, the service communications needed for HashiCups to work properly are the following:

  • The NGINX service needs to communicate with Frontend and API services
  • The API service needs to communicate with the Database service
  • The API Gateway needs to communicate with the NGINX service

By the end of this tutorial, you will have configured your service mesh to deny all communications except the ones explicitly allowed by the intentions you defined.

Architecture Final scenario Architecture Final scenario

Prerequisites

This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.

This tutorial uses an interactive lab to guide you through the process of setting up a service mesh on your VM workloads. The lab environment includes all required binaries and sample configurations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Once the lab is loaded, Open the Bastion Host tab to login to the bastion host.

Configure CLI to interact with Consul

Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.

$ source assets/scenario/env-consul.env && \
 source assets/scenario/env-scenario.env

After loading the needed variables, verify you can connect to your Consul datacenter.

$ consul members
Node                Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0     10.0.4.207:8301  alive   server  1.16.3  2         dc1  default    <all>
gateway-api         10.0.4.27:8301   alive   client  1.16.3  2         dc1  default    <default>
hashicups-api       10.0.4.83:8301   alive   client  1.16.3  2         dc1  default    <default>
hashicups-db        10.0.4.65:8301   alive   client  1.16.3  2         dc1  default    <default>
hashicups-frontend  10.0.4.33:8301   alive   client  1.16.3  2         dc1  default    <default>
hashicups-nginx     10.0.4.130:8301  alive   client  1.16.3  2         dc1  default    <default>

Create ACL token for intentions management

To configure intentions in a Consul datacenter, when ACLs are enabled, you need a valid token with the proper permissions.

To change intentions for a service, Consul includes the intentions scope for the ACL rules of a service. Since in this tutorial you will configure services with names starting with hashicups-, the policy will need the intentions = "write" for all those services.

The rules to grant intentions permissions to all and only HashiCups services will be the following.

First define the output folder where to store the files generated.

$ export OUTPUT_FOLDER=assets/scenario/conf/

Then, create a policy file for the ACL rules you want to grant.

$ tee ${OUTPUT_FOLDER}/acl-policy-intentions-hashicups.hcl > /dev/null << EOF
# --------------------------------------+
# acl-policy-intentions-hashicups.hcl   |
# --------------------------------------+

service_prefix "hashicups-" {
  policy = "read"
  intentions = "write"
}
EOF

Tip

Consul grants permissions for creating and managing intentions based on the destination, not the source. When ACLs are enabled, services and operators must present a token linked to a policy that grants the necessary permissions to the destination service.

Create the Consul policy.

$ consul acl policy create \
    -name "HashiCups-intentions-policy" \
    -rules @${OUTPUT_FOLDER}/acl-policy-intentions-hashicups.hcl

The output should look like the following.

ID:           f495b8f2-5e5e-1a0f-78ab-27fc884305d7
Name:         HashiCups-intentions-policy
Description:  
Datacenters:  
Rules:
# --------------------------------------+
# acl-policy-intentions-hashicups.hcl   |
# --------------------------------------+

service_prefix "hashicups-" {
  policy = "read"
  intentions = "write"
}

Then, create a token associated with the policy.

$ consul acl token create \
    -description 'HashiCups Intentions Management token' \
    -policy-name HashiCups-intentions-policy --format json > ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json

Extended permissions

Since the Consul datacenter contains a allow-all intention that refers to all services, the rules above are not sufficient to manipulate that intention.

To solve that you will create another policy that grants intentions:write for all services.

$ tee ${OUTPUT_FOLDER}/acl-policy-intentions-all.hcl > /dev/null << EOF
# --------------------------------------+
# acl-policy-intentions-all.hcl   |
# --------------------------------------+

service_prefix "" {
  policy = "read"
  intentions = "write"
}
EOF

With the new rules file, create the policy.

$ consul acl policy create \
    -name "all-intentions-policy" \
    -rules @${OUTPUT_FOLDER}/acl-policy-intentions-all.hcl

The output should look like the following.

ID:           e9e65da3-1616-5ac8-0287-3e60da8fca2f
Name:         all-intentions-policy
Description:  
Datacenters:  
Rules:
# --------------------------------------+
# acl-policy-intentions-all.hcl   |
# --------------------------------------+

service_prefix "" {
  policy = "read"
  intentions = "write"
}

Then, create a token associated with the policy.

$ consul acl token create \
    -description 'All Intentions Management token' \
    -policy-name all-intentions-policy --format json > ${OUTPUT_FOLDER}secrets/acl-token-intentions-all.json

Note

The second token grants access to intentions for all the services present in the datacenter. The use of the token should be limited to datacenter operators, while the first token can be assigned to application developers to offload some of the maintenance for the single application.

To continue with the tutorial, using minimal permissions, export the first token as an environment variable. 

$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json | jq -r ".SecretID"`

Check intentions for a Consul datacenter

Check existing intentions in your Consul datacenter.

Use the consul intention command to check existing intentions.

$ consul intention list
ID  Source  Action  Destination  Precedence
    *       allow   *            5

From the output you can verify that all (*) service-to-service connections are allowed.

Check HashiCups UI

In this section you will verify that the HashiCups application is working properly.

Click on the HashiCups tab.

Confirm that HashiCups is reachable from the API Gateway. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.

HashiCups UI HashiCups UI

Enable a new service-to-service communication

In the present scenario, upstreams are already defined for the HashiCups' services to permit the intended traffic flows.

The allow-all intention permits to define new service-to-service communication flows by adding upstream definitions in the service configuration files.

You will now learn how to add a new service-to-service communication flow by adding an upstream that permits the Database service to communicate with the API service.

This is the intended sequence of steps that you will perform in your environment when a new communication flow is identified as necessary for a service.

Check initial status

First, you will check that, in the initial state, the Database service is not configured to communicate with the API service.

Click on the API tab.

Check existing listening processes.

$ netstat -natp | grep LISTEN

The output will be similar to the following, the order of listeners might be different in your scenario.

tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:21000           0.0.0.0:*               LISTEN      6422/envoy
tcp        0      0 10.0.4.7:8301           0.0.0.0:*               LISTEN      6359/consul
tcp        0      0 127.0.0.1:9103          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:8081          0.0.0.0:*               LISTEN      6702/public-api
tcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      6359/consul
tcp        0      0 127.0.0.1:8502          0.0.0.0:*               LISTEN      6359/consul
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      6422/envoy
tcp        0      0 127.0.0.1:19000         0.0.0.0:*               LISTEN      6422/envoy
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      6359/consul
tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN      6751/grafana-agent
tcp6       0      0 :::22                   :::*                    LISTEN      -

The service you want to reach, public-api, is configured to listen on the loopback interface, on port `8081``, and is not accessible externally.

From the output, you can notice that the VM only exposes three listeners externally:

  • the SSH server, on port 22
  • the Consul client gossip interface, on port 8301
  • the Envoy process, on port 21000

The Envoy process is used by Consul service mesh to route traffic inside the service mesh services.

Among the listeners, you can also notice one Envoy process listening on port 5432. That represents the available connection from the API service (source) and the Database service (destination/upstream).

Add a new upstream to a service definition

In this section, you will add new communication channels across services by changing the service definition and adding upstreams to the services.

Click on the Database tab.

Check existing listening processes, the order of listeners might be different in your scenario.

$ netstat -natp | grep LISTEN
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:19000         0.0.0.0:*               LISTEN      6257/envoy
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN      6503/grafana-agent
tcp        0      0 127.0.0.1:12346         0.0.0.0:*               LISTEN      6503/grafana-agent
tcp        0      0 0.0.0.0:21000           0.0.0.0:*               LISTEN      6257/envoy
tcp        0      0 10.0.4.30:8301          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:8502          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

Among the listeners, the process listening on port 5432 represents the Database service.

You can notice that the only processes exposed externally are the SSH server, Consul gossip, and Envoy.

In this case, there is no Envoy process that represents an upstream service.

You want to add the possibility to connect to the API service from the Database host, to do so modify the hashicups-db service definition to add the following section.

proxy {
  upstreams = [
    {
      destination_name = "hashicups-api"
      local_bind_port = 8081
    }
  ]
}

The section describes an upstreams array for the hashicups-db service including one upstream for the hashicups-api service on port 8081.

Note

The port represents the port that will be used on the local listener and not the original port exposed by the hashicups-api service on its node.

First, populate CONSUL_HTTP_TOKEN variable using the token used by the Consul client agent.

$ export CONSUL_HTTP_TOKEN=`cat /etc/consul.d/agent-acl-tokens.hcl | grep agent | awk '{print $3}' | tr -d \"`

Then, generate the new service definition file to include the upstream definition.

$ tee /etc/consul.d/svc-hashicups-db.hcl > /dev/null << EOF
## svc-hashicups-db.hcl
service {
  name = "hashicups-db"
  id = "hashicups-db-1"
  port = 5432
  token = "${CONSUL_HTTP_TOKEN}"
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "hashicups-api"
            local_bind_port = 8081
          }
        ]
      }
    }
  }
  check
  {
    id =  "check-hashicups-db",
    name = "hashicups-db status check",
    service_id = "hashicups-db-1",
    tcp  = "localhost:5432",
    interval = "5s",
    timeout = "5s"
  }
}
EOF

Reload Consul to apply the new service configuration.

$ consul reload
Configuration reload triggered

After the Consul agent reloaded the configuration, it will automatically update the Envoy sidecar proxy configuration to include the new upstream on the local node.

$ netstat -natp | grep LISTEN

The output will be similar to the following, the order of listeners might be different in your scenario.

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:19000         0.0.0.0:*               LISTEN      6257/envoy
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:12345         0.0.0.0:*               LISTEN      6503/grafana-agent
tcp        0      0 127.0.0.1:12346         0.0.0.0:*               LISTEN      6503/grafana-agent
tcp        0      0 0.0.0.0:21000           0.0.0.0:*               LISTEN      6257/envoy
tcp        0      0 10.0.4.30:8301          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:8081          0.0.0.0:*               LISTEN      6257/envoy
tcp        0      0 127.0.0.1:8500          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 127.0.0.1:8502          0.0.0.0:*               LISTEN      6194/consul
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

Now, the API service is exposed on the Database node loopback interface, on the port you specified in the service definition, 8081.

Check service connection

Verify you can now connect to the API service from the Database node.

$ curl --silent 'http://localhost:8081/api' \
      -H 'Accept-Encoding: gzip, deflate, br' \
      -H 'Content-Type: application/json' \
      -H 'Accept: application/json' \
      -H 'Connection: keep-alive' \
      -H 'DNT: 1' \
      -H 'Origin: http://localhost:8081' \
      --data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\",    cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed | jq

A successful API connection will return an output similar to the following:

{
  "data": {
    "pay": {
      "id": "b7912087-80c8-4520-99bb-79a89f1b611e",
      "card_plaintext": "1234123-0123123",
      "card_ciphertext": "Encryption Disabled",
      "message": "Payment processed successfully, card details returned for demo purposes, not for production"
    }
  }
}

By changing the service definition upstreams and reloading the Consul configuration to include the change, you added a new service-to-service communication flow to your service mesh. The fact you have an allow-all intention in place, is the other factor that allows the communication.

This is a very powerful method to configure services in your network, where the application developer can define the dependencies for each service as upstreams directly in the service definition, and have Consul service mesh make the upstreams available on the service node, directly on the loopback interface, on the specified port.

Tune intentions for your environment

Having the traffic dependencies defined as upstreams, makes easy for developers to deploy their applications from their local environment to production. The services will keep the localhost configuration and Consul will route their requests to the right service.

In a production environment, this simplicity can also bring unwanted behaviors.

A developer adding un-necessary upstreams to their service definition might open communication channels that are not intended.

Consul permits an operator to prevent unwanted traffic using intentions.

For this reason, before deploying a new application in your Consul service mesh, you should prepare an intention diagram that defines intended traffic permissions. This will be used by the operators to identify the necessary intentions your application will need to operate properly.

Understand HashiCups services' upstreams

The following intentions are required for HashiCups:

Tip

Notice these descriptions define traffic starting from the destination, Consul intentions are defined in the same way, the intention is defined for the destination service and defines all the source services that need to communicate wit it.

  1. The hashicups-db service needs to be reached by the hashicups-api service.
  2. The hashicups-api service needs to be reached by the hashicups-nginx services.
  3. The hashicups-frontend service needs to be reached by the hashicups-nginx service.
  4. The hashicups-nginx service needs to be reached by the gateway-api service.

Define and apply intentions

Select the Bastion Host tab.

Use the provided script to generate service intentions.

$ export OUTPUT_FOLDER=assets/scenario/conf/

$ bash ops/scenarios/99_supporting_scripts/generate_consul_service_intentions.sh
Parameter Check
Create global proxy configuration
Create intention configuration files

The script creates service-intentions configuration files both in json and hcl format.

$ tree ${OUTPUT_FOLDER}global
assets/scenario/conf/global
|-- config-global-proxy-default.hcl
|-- config-global-proxy-default.json
|-- intention-allow-all.hcl
|-- intention-allow-all.json
|-- intention-api.hcl
|-- intention-api.json
|-- intention-db.hcl
|-- intention-db.json
|-- intention-frontend.hcl
|-- intention-frontend.json
|-- intention-nginx.hcl
`-- intention-nginx.json

0 directories, 12 files

Hera one example of a service-intentions configuration file generated by the script.

intention-api.hcl
Kind = "service-intentions"
Name = "hashicups-api"
Sources = [
  {
    Name   = "hashicups-nginx"
    Action = "allow"
  }
]
intention-api.json
{
  "Kind": "service-intentions",
  "Name": "hashicups-api",
  "Sources": [
    {
      "Action": "allow",
      "Name": "hashicups-nginx"
    }
  ]
}

You can notice the intention is defined for the destination service, in this case hashicups-api, specifying the source services that the intention regulates, in this case hashicups-nginx, and a directive, in this case allow, that defines the permission for the connection.

Apply specific intentions

Apply the intentions to your Consul datacenter.

Use config write to create the following intentions.

Create the intentions for the hashicups-db service.

$ consul config write ${OUTPUT_FOLDER}global/intention-db.hcl  
Config entry written: service-intentions/hashicups-db

Create the intentions for the hashicups-api service.

$ consul config write ${OUTPUT_FOLDER}global/intention-api.hcl 
Config entry written: service-intentions/hashicups-api

Create the intentions for the hashicups-frontend service.

$ consul config write ${OUTPUT_FOLDER}global/intention-frontend.hcl 
Config entry written: service-intentions/hashicups-frontend

Create the intentions for the hashicups-nginx service to allow access from the API gateway.

$ consul config write ${OUTPUT_FOLDER}global/intention-nginx.hcl 
Config entry written: service-intentions/hashicups-nginx

Now the Consul datacenter contains, alongside the allow-all intention that permits traffic across all services, he four intentions you just defined.

These four intentions define the HashiCups traffic flow.

Architecture Intermediate scenario Architecture Intermediate scenario

Remove default intention

Now that all connections are explicitly specified, you can remove the allow-all intention without affecting the application uptime.

Tip

By removing the *>* intention, Consul will use as default intention the same behavior specified by the ACL default policy. In this scenario the default policy is default_policy = "deny", therefore if no intention is explicitly defined for the communication across two services Consul will deny the connection.

$ consul config delete -kind service-intentions -name "*"

If the token you are using does not include necessary permissions to manipulate intentions for all services, the request is expected to fail with a 403 error.

Error deleting config entry service-intentions/*: Unexpected response code: 403 (Permission denied: token with AccessorID '01bcbd3a-ce74-a615-143f-8780d8391dff' lacks permission 'intention:write' on "*")

To complete the operation assign the correct token to the environment variable and try again.

$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-all.json | jq -r ".SecretID"`

The new attempt should be successful.

$ consul config delete -kind service-intentions -name "*"
Config entry deleted: service-intentions/*

Once you completed the intention deletion, you should revert back to a less privileged token.

$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json | jq -r ".SecretID"`

Check intentions in your Consul datacenter

After applying the desired intention, check the new intentions configuration for your Consul datacenter.

Use the consul intention command to check existing intentions.

$ consul intention list
ID  Source           Action  Destination         Precedence
    gateway-api      allow   hashicups-nginx     9
    hashicups-api    allow   hashicups-db        9
    hashicups-nginx  allow   hashicups-api       9
    hashicups-nginx  allow   hashicups-frontend  9

Verify connections are now secured

After applying the intentions and removing the allow-all*` directive you reached the final configuration for the scenario.

Architecture Final scenario Architecture Final scenario

Last steps are to check if the HashiCups application is still working and if the connection between Database and API is now blocked.

Check HashiCups UI

Click on the HashiCups tab.

Confirm that HashiCups still works despite its services being configured to communicate on localhost. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.

HashiCups UI HashiCups UI

Test service to service connection

Click on the Database tab.

Verify the connection to the API service from the Database node.

$ curl 'http://localhost:8081/api' \
      -H 'Accept-Encoding: gzip, deflate, br' \
      -H 'Content-Type: application/json' \
      -H 'Accept: application/json' \
      -H 'Connection: keep-alive' \
      -H 'DNT: 1' \
      -H 'Origin: http://localhost:8081' \
      --data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\",    cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed

If the intentions are being enforced properly, the command will return an output similar to the following:

curl: (52) Empty reply from server

Using intentions you removed the risk of unwanted connections from the Database service to the API service without the need of ad-hoc firewall rules, and enforcing a global set of permissions even over a specific configuration made on the service definition itself.

Exit the ssh session to return to the bastion host and complete the tutorial.

$ exit
logout
Connection to hashicups-db closed.
admin@bastion:~$

Next steps

In this tutorial you learned how to configure upstreams for existing services and to open service-to-service communications inside Consul service mesh.

You then learned how to secure communication by limiting connections only across specific services.

In the next tutorial, you will learn how use application aware intentions to tune permissions at a finer level of detail.

For more information about the topics covered in this tutorial, refer to the following resources:

Collection Overview

Service Mesh Security

 Next

Service requests with L7 intentions

This tutorial also appears in:

  • 4 tutorials
    Secure services
    Secure service to service traffic in Consul service mesh using service intentions, both at L4 and L7 level, and Access Control Lists (ACLs) to avoid unidentified traffic. Include new services in Consul service mesh gradually using permissive mTLS.
    • Consul

Workflow