Control traffic communication between services with intentions
In a Consul service mesh scenario you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally.
These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.
This removes the need to generate ad-hoc firewall rules for each service instance and moves the focus over service identity.
Consul uses service intentions to further limit the traffic only to the services explicitly allowed to communicate. This limits the surface exposed by the single service and constitutes one extra step towards zero-trust security.
In this tutorial, you will learn how to tune service access by using intentions.
Specifically, you will:
- Understand the ACL permissions required to manipulate intentions
- List the current intentions in your Consul service mesh
- Change a service definition to add a new upstream
- Apply intentions to your Consul datacenter to prevent unwanted traffic
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul service mesh, with Envoy sidecar proxies running alongside each service, and an API Gateway configured to permit access to the NGINX service.
The Consul service mesh is initially configured with an allow-all
intention that enables every service to communicate with each other.
The only requirement for a service to service communication, in this scenario, is the presence of an upstream in the source service definition.
While this can be acceptable in the onboarding phase of your service mesh, in a production environment you want to transition to a deny-all
default intention, where only the necessary communications are explicitly allowed.
In this scenario, the service communications needed for HashiCups to work properly are the following:
- The NGINX service needs to communicate with Frontend and API services
- The API service needs to communicate with the Database service
- The API Gateway needs to communicate with the NGINX service
By the end of this tutorial, you will have configured your service mesh to deny all communications except the ones explicitly allowed by the intentions you defined.
Prerequisites
This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.
If you want to follow along with this tutorial and you do not already have the required infrastructure in place, the following steps guide you through the process to deploy a demo application and a configured Consul service mesh on AWS automatically using Terraform.
To create a Consul deployment on AWS using terraform, you need the following:
Clone GitHub repository
Clone the GitHub repository containing the configuration files and resources.
Enter the directory that contains the configuration files for this tutorial.
Create infrastructure
With these Terraform configuration files, you are ready to deploy your infrastructure.
Issue the terraform init
command from your working directory to download the
necessary providers and initialize the backend.
Then, deploy the resources. Confirm the run by entering yes
.
Tip
The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization or learn more about the Raft protocol in a fun and interactive way.
After the deployment is complete, Terraform returns a list of outputs you can use to interact with the newly created environment.
The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:
- The
ip_bastion
provides IP address of the bastion host you use to run the rest of the commands in this tutorial. - The
remote_ops
lists the bastion host IP, which you can use access the bastion host. - The
retry_join
output lists Consul'sretry_join
configuration parameter. The next tutorial uses this information to generate Consul server and client configuration. - The
ui_consul
output lists the Consul UI address. The Consul UI is not currently running. You will use the Consul UI in a later tutorial to verify that Consul started correctly. - The
ui_grafana
output lists the Grafana UI address. You will use this address in a future tutorial. - The
ui_hashicups
output lists the HashiCups UI address. You can open this address in a web browser to verify the HashiCups demo application is running properly.
Login into the bastion host VM
Login to the bastion host using ssh
.
Configure CLI to interact with Consul
Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.
After loading the needed variables, verify you can connect to your Consul datacenter.
Create ACL token for intentions management
To configure intentions in a Consul datacenter, when ACLs are enabled, you need a valid token with the proper permissions.
To change intentions for a service, Consul includes the intentions
scope for the ACL rules of a service.
Since in this tutorial you will configure services with names starting with hashicups-
, the policy will need the intentions = "write"
for all those services.
The rules to grant intentions permissions to all and only HashiCups services will be the following.
First define the output folder where to store the files generated.
Then, create a policy file for the ACL rules you want to grant.
Tip
Consul grants permissions for creating and managing intentions based on the destination, not the source. When ACLs are enabled, services and operators must present a token linked to a policy that grants the necessary permissions to the destination service.
Create the Consul policy.
The output should look like the following.
Then, create a token associated with the policy.
Extended permissions
Since the Consul datacenter contains a allow-all
intention that refers to all services, the rules above are not sufficient to manipulate that intention.
To solve that you will create another policy that grants intentions:write
for all services.
With the new rules file, create the policy.
The output should look like the following.
Then, create a token associated with the policy.
Note
The second token grants access to intentions for all the services present in the datacenter. The use of the token should be limited to datacenter operators, while the first token can be assigned to application developers to offload some of the maintenance for the single application.
To continue with the tutorial, using minimal permissions, export the first token as an environment variable.
Check intentions for a Consul datacenter
Check existing intentions in your Consul datacenter.
Use the consul intention
command to check existing intentions.
From the output you can verify that all (*
) service-to-service connections are allowed.
Check HashiCups UI
In this section you will verify that the HashiCups application is working properly.
Retrieve the API Gateway public IP from the bastion host.
The output will be similar to the following:
Then, open the address in a browser.
Confirm that HashiCups is reachable from the API Gateway. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.
Enable a new service-to-service communication
In the present scenario, upstreams are already defined for the HashiCups' services to permit the intended traffic flows.
The allow-all
intention permits to define new service-to-service communication flows by adding upstream definitions in the service configuration files.
You will now learn how to add a new service-to-service communication flow by adding an upstream that permits the Database service to communicate with the API service.
This is the intended sequence of steps that you will perform in your environment when a new communication flow is identified as necessary for a service.
Check initial status
First, you will check that, in the initial state, the Database service is not configured to communicate with the API service.
Login to hashicups-api
from the bastion host.
Check existing listening processes.
The output will be similar to the following, the order of listeners might be different in your scenario.
The service you want to reach, public-api
, is configured to listen on the loopback interface, on port `8081``, and is not accessible externally.
From the output, you can notice that the VM only exposes three listeners externally:
- the SSH server, on port 22
- the Consul client gossip interface, on port 8301
- the Envoy process, on port 21000
The Envoy process is used by Consul service mesh to route traffic inside the service mesh services.
Among the listeners, you can also notice one Envoy process listening on port 5432
. That represents the available connection from the API service (source) and the Database service (destination/upstream).
To continue with the tutorial, exit the ssh session to return to the bastion host.
Add a new upstream to a service definition
In this section, you will add new communication channels across services by changing the service definition and adding upstreams to the services.
Login to hashicups-db
from the bastion host.
Check existing listening processes, the order of listeners might be different in your scenario.
Among the listeners, the process listening on port 5432
represents the Database service.
You can notice that the only processes exposed externally are the SSH server, Consul gossip, and Envoy.
In this case, there is no Envoy process that represents an upstream service.
You want to add the possibility to connect to the API service from the Database host, to do so modify the hashicups-db
service definition to add the following section.
The section describes an upstreams
array for the hashicups-db
service including one upstream for the hashicups-api
service on port 8081
.
Note
The port represents the port that will be used on the local listener and not the original port exposed by the hashicups-api
service on its node.
First, populate CONSUL_HTTP_TOKEN
variable using the token used by the Consul client agent.
Then, generate the new service definition file to include the upstream definition.
Reload Consul to apply the new service configuration.
After the Consul agent reloaded the configuration, it will automatically update the Envoy sidecar proxy configuration to include the new upstream on the local node.
The output will be similar to the following, the order of listeners might be different in your scenario.
Now, the API service is exposed on the Database node loopback interface, on the port you specified in the service definition, 8081
.
Check service connection
Verify you can now connect to the API service from the Database node.
A successful API connection will return an output similar to the following:
By changing the service definition upstreams and reloading the Consul configuration to include the change, you added a new service-to-service communication flow to your service mesh. The fact you have an allow-all
intention in place, is the other factor that allows the communication.
This is a very powerful method to configure services in your network, where the application developer can define the dependencies for each service as upstreams directly in the service definition, and have Consul service mesh make the upstreams available on the service node, directly on the loopback interface, on the specified port.
To continue with the tutorial, exit the ssh session to return to the bastion host.
Tune intentions for your environment
Having the traffic dependencies defined as upstreams, makes easy for developers to deploy their applications from their local environment to production. The services will keep the localhost
configuration and Consul will route their requests to the right service.
In a production environment, this simplicity can also bring unwanted behaviors.
A developer adding un-necessary upstreams to their service definition might open communication channels that are not intended.
Consul permits an operator to prevent unwanted traffic using intentions.
For this reason, before deploying a new application in your Consul service mesh, you should prepare an intention diagram that defines intended traffic permissions. This will be used by the operators to identify the necessary intentions your application will need to operate properly.
Understand HashiCups services' upstreams
The following intentions are required for HashiCups:
Tip
Notice these descriptions define traffic starting from the destination, Consul intentions are defined in the same way, the intention is defined for the destination service and defines all the source services that need to communicate wit it.
- The
hashicups-db
service needs to be reached by thehashicups-api
service. - The
hashicups-api
service needs to be reached by thehashicups-nginx
services. - The
hashicups-frontend
service needs to be reached by thehashicups-nginx
service. - The
hashicups-nginx
service needs to be reached by thegateway-api
service.
Define and apply intentions
Use the provided script to generate service intentions.
The script creates service-intentions
configuration files both in json
and hcl
format.
Hera one example of a service-intentions
configuration file generated by the script.
You can notice the intention is defined for the destination service, in this case hashicups-api
, specifying the source services that the intention regulates, in this case hashicups-nginx
, and a directive, in this case allow
, that defines the permission for the connection.
Apply specific intentions
Apply the intentions to your Consul datacenter.
Use config write
to create the following intentions.
Create the intentions for the hashicups-db
service.
Create the intentions for the hashicups-api
service.
Create the intentions for the hashicups-frontend
service.
Create the intentions for the hashicups-nginx
service to allow access from the API
gateway.
Now the Consul datacenter contains, alongside the allow-all
intention that permits traffic across all services, he four intentions you just defined.
These four intentions define the HashiCups traffic flow.
Remove default intention
Now that all connections are explicitly specified, you can remove the allow-all
intention without affecting the application uptime.
Tip
By removing the *>*
intention, Consul will use as default intention the same behavior specified by the ACL default policy. In this scenario the default policy is default_policy = "deny"
, therefore if no intention is explicitly defined for the communication across two services Consul will deny the connection.
If the token you are using does not include necessary permissions to manipulate intentions for all services, the request is expected to fail with a 403 error.
To complete the operation assign the correct token to the environment variable and try again.
The new attempt should be successful.
Once you completed the intention deletion, you should revert back to a less privileged token.
Check intentions in your Consul datacenter
After applying the desired intention, check the new intentions configuration for your Consul datacenter.
Use the consul intention
command to check existing intentions.
Verify connections are now secured
After applying the intentions and removing the allow-all
*` directive you reached the final configuration for the scenario.
Last steps are to check if the HashiCups application is still working and if the connection between Database and API is now blocked.
Check HashiCups UI
First, retrieve the API Gateway address.
For this scenario, you can get the API Gateway public IP directly from the bastion host.
The output will be similar to the following:
Then, open the address in a browser.
Confirm that HashiCups still works despite its services being configured to
communicate on localhost
. The Envoy sidecar proxies route each service's local
traffic to the relevant upstream.
Test service to service connection
Login to hashicups-db
from the bastion host.
Verify the connection to the API service from the Database node.
If the intentions are being enforced properly, the command will return an output similar to the following:
Using intentions you removed the risk of unwanted connections from the Database service to the API service without the need of ad-hoc firewall rules, and enforcing a global set of permissions even over a specific configuration made on the service definition itself.
Exit the ssh session to return to the bastion host and complete the tutorial.
Next steps
In this tutorial you learned how to configure upstreams for existing services and to open service-to-service communications inside Consul service mesh.
You then learned how to secure communication by limiting connections only across specific services.
If you want to stop at this tutorial, you can destroy the infrastructure now.
From the ./self-managed/infrastruture/aws
folder of the repository, use terraform
to destroy the infrastructure.
In the next tutorial, you will learn how use application aware intentions to tune permissions at a finer level of detail.
For more information about the topics covered in this tutorial, refer to the following resources: