HashiDays One conference. Three cities. Find a city near you
Consul
Secure services

Manage permissions with access control lists (ACLs) on VMs

  • 21min
  • |
  • Consul
  • Interactive

Workflow

Consul service mesh provides secure communication across your services within and across your infrastructure, including on-premises and cloud environments. In a service mesh scenario, you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally. These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.

Consul provides one extra layer of security, Access Control Lists (ACLs), to prevent malicious actors to register services and hijack communications in your datacenter. A Consul datacenter with ACLs enabled and with default-deny policy, will require a valid token for every operation, including service registration.

In this tutorial you will learn how to create credentials required to join an existing Consul datacenter and register a service to the Consul catalog. After an introductory section about the different tokens available for Consul agents' configuration, you will generate tokens for the Consul node using the node-identiy template policy, and for a Consul service using the service-identity policy. You will also learn how to reuse an existing policy to generate a token for the Consul DNS interface, to allow service resolution locally.

You will use the generated credentials to create a service configuration for the API service, and to add a new Consul agent to Consul service mesh. After the agent joins the Consul service mesh, you will verify the service is correctly registered in the mesh using all the interfaces provided by Consul: CLI, API, DNS, and web UI.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you will deploy Consul service mesh, with Envoy sidecar proxies running alongside each service and an API Gateway configured to permit access to the NGINX service.

In addition, you will deploy an extra node that hosts another HashiCups API instance that you want to add to your datacenter.

Architecture Initial scenario Architecture Initial scenario

In this tutorial, you will use ACLs to add the API service to the Consul service mesh.

Architecture Final scenario Architecture Final scenario

Once you have registered the service in Consul, it will automatically appear in Consul DNS results and Consul will automatically load balance traffic across the different instances of the service.

Note

All operations in this tutorial, after the scenario deployment, will be performed from an additional node, a bastion host, that will be deployed as part of the scenario. The tutorial provides instructions on how to connect to the bastion host at the end of the deployment.

Prerequisites

This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to the Consul Getting Started tutorials collection.

For this tutorial, you will need:

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-acls-vms

Enter the directory that contains the configuration files for this tutorial.

$ cd learn-consul-acls-vms/self-managed/infrastructure/aws

Create infrastructure

With these Terraform configuration files, you are ready to deploy your infrastructure.

Issue the terraform init command from your working directory to download the necessary providers and initialize the backend.

$ terraform initInitializing the backend...Initializing provider plugins......Terraform has been successfully initialized!...

Then, deploy the resources. Confirm the run by entering yes.

$ terraform apply -var-file=../../ops/conf/manage_permissions_with_acls.tfvars## ...Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes## ...Apply complete! Resources: 50 added, 0 changed, 0 destroyed.

Tip

The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization.

After the deployment is complete, Terraform returns a list of outputs you can use to interact with the newly created environment.

Outputs:connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"ip_bastion = "<redacted-output>"remote_ops = "export BASTION_HOST=<redacted-output>"retry_join = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-hcoc"ui_consul = "https://<redacted-output>:8443"ui_grafana = "http://<redacted-output>:3000/d/hashicups/hashicups"ui_hashicups = "http://<redacted-output>"

The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:

  • The ip_bastion provides IP address of the bastion host you use to run the rest of the commands in this tutorial.
  • The remote_ops lists the bastion host IP, which you can use access the bastion host.
  • The retry_join output lists Consul's retry_join configuration parameter. The next tutorial uses this information to generate Consul server and client configuration.
  • The ui_consul output lists the Consul UI address. The Consul UI is not currently running. You will use the Consul UI in a later tutorial to verify that Consul started correctly.
  • The ui_grafana output lists the Grafana UI address. You will use this address in a future tutorial.
  • The ui_hashicups output lists the HashiCups UI address. You can open this address in a web browser to verify the HashiCups demo application is running properly.

List AWS instances

The scenario deploys eight virtual machines.

$ terraform state list## ...aws_instance.api[0]aws_instance.api[1]aws_instance.bastionaws_instance.consul_server[0]aws_instance.database[0]aws_instance.frontend[0]aws_instance.gateway-api[0]aws_instance.nginx[0]## ...

After deployment six virtual machines, consul_server[0], database[0], frontend[0], api[0], nginx[0], and gateway-api[0] are configured in a Consul datacenter with service mesh enabled.

The two remaining nodes are:

  • bastion - the bastion host that you will use to perform the tutorial steps.
  • api[1] - a virtual machine to host a second instance of the hashicups-api service.

Login into the bastion host VM

Login to the bastion host using ssh.

$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`#...admin@bastion:~$

Configure CLI to interact with Consul

Configure your bastion host to communicate with your Consul environment using the two, dynamically generated, environment variable files.

$ source "/home/admin/assets/scenario/env-scenario.env" && \  source "/home/admin/assets/scenario/env-consul.env"

After loading the needed variables, verify you can connect to your Consul datacenter.

$ consul membersNode                  Address             Status  Type    Build   Protocol  DC   Partition  Segmentconsul-server-0       192.168.80.7:8301   alive   server  1.17.0  2         dc1  default    <all>gateway-api-0         192.168.80.4:8301   alive   client  1.17.0  2         dc1  default    <default>hashicups-api-0       192.168.80.8:8301   alive   client  1.17.0  2         dc1  default    <default>hashicups-db-0        192.168.80.2:8301   alive   client  1.17.0  2         dc1  default    <default>hashicups-frontend-0  192.168.80.12:8301  alive   client  1.17.0  2         dc1  default    <default>hashicups-nginx-0     192.168.80.11:8301  alive   client  1.17.0  2         dc1  default    <default>

Notice that bastion and hashicups-api-1 are not part of the datacenter.

Create ACL token for ACL management

A management token is required to manage Consul's ACLs system. We recommend creating a new Consul management token for this purpose instead of using the ACL bootstrap token.

$ consul acl token create \  -description="SVC HashiCups API token" \  --format json \  -policy-name="global-management" | tee /home/admin/assets/scenario/conf/secrets/acl-token-management.json

That will produce an output similar to the following.

{    "CreateIndex": 115,    "ModifyIndex": 115,    "AccessorID": "9cdf707d-8847-9bb2-606c-d4f0849692d6",    "SecretID": "e4d09a88-f3dc-0191-7ce6-f7a35a7097ae",    "Description": "SVC HashiCups API token",    "Policies": [        {            "ID": "00000000-0000-0000-0000-000000000001",            "Name": "global-management"        }    ],    "Local": false,    "CreateTime": "2023-11-18T13:27:42.735373628Z",    "Hash": "UfTdHatx/OxAw8z+/8suSKQrY6MgpC3CutkuZZAzPC4="}

Set your newly generated management token as the CONSUL_HTTP_TOKEN environment variable. This will allow you to perform ACL system operations in the next steps using the new token.

$ export CONSUL_HTTP_TOKEN=`cat /home/admin/assets/scenario/conf/secrets/acl-token-management.json | jq -r ".SecretID"`

Create ACL configuration for Consul agent

Consul provides a fine grained set of permissions to manage your Consul nodes and services. In this section you will create the ACL tokens required to register a new Consul client node in a Consul datacenter. You will then register a service in the Consul service mesh with these ACL tokens.

The initial tutorial environment includes an empty VM named hashicups-api-1 for you. You will create ACLs token to register this Consul client node to the Consul datacenter, and register the service in Consul.

Consul has a section in the configuration to specify all the ACL tokens to be used by the agent.

agent-acl-tokens.hcl

...acl {  tokens {    agent  = "<Consul agent token>"    default  = "<Consul default token>"    config_file_service_registration = "<Consul service registration token>"  }}...

The different tokens are used for different operation performed by the Consul agent:

  • agent - Used for clients and servers to perform internal operations. This token must have write access to the node name it will register as in order to set any of the node-level information in the catalog.
  • default - Used for the default operations when agent token is not provided and for the endpoints that do not permit a token being passed. This will be used for the DNS interface of the Consul agent.
  • config_file_service_registration - Specifies the ACL token the agent uses to register services and checks. This token needs write permission to register all services and checks defined in this agent's configuration.

You will generate the ACL configuration section after these three tokens are generated.

Generate default token

Each token must be associated with a policy providing the desired permissions. The scenario already contains a policy used to generate the default token for the other nodes that is already suitable for the node default token. The policy is named acl-policy-dns.

Read the policy using the consul acl policy command.

$ consul acl policy read -name acl-policy-dnsID:           f77b2144-190f-00ee-46b5-cb769d018e8eName:         acl-policy-dnsDescription:  Policy for DNS endpointsDatacenters:  Rules:# -----------------------------+# acl-policy-dns.hcl           |# -----------------------------+node_prefix "" {  policy = "read"}service_prefix "" {  policy = "read"}# only needed if using prepared queriesquery_prefix "" {  policy = "read"}# needed for prometheus metricsagent_prefix ""{  policy = "read"}

The acl-policy-dns policy, used to generate the default token, grants read access to all services and nodes so DNS query can work. It also grants read permissions for prepared queries and agent metrics endpoints.

Generate a new token using the acl-policy-dns policy.

$ consul acl token create \  -description="hashicups-api-1 default token" \  --format json \  -policy-name="acl-policy-dns" | tee /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.json

That will produce an output similar to the following.

{    "CreateIndex": 116,    "ModifyIndex": 116,    "AccessorID": "ba4533ad-6e44-7279-41d3-08bca235306f",    "SecretID": "391bc1b8-2837-db2c-6466-613e16051050",    "Description": "hashicups-api-1 default token",    "Policies": [        {            "ID": "f77b2144-190f-00ee-46b5-cb769d018e8e",            "Name": "acl-policy-dns"        }    ],    "Local": false,    "CreateTime": "2023-11-18T13:27:42.871333836Z",    "Hash": "xPN3IX5yLnk7SDzHYXIBCPKpQtAWGP/VoLfZBmiK9kA="}

Generate agent token

In order for the agent to set any node-level information in Consul catalog, the agent token must have write access to the node name it will register as. One minimal example for such a policy is the following.

node-identity-policy-template.hcl

# Allow the agent to register its own node in the Catalog and update its network coordinatesnode "<node name>" {  policy = "write"}# Allows the agent to detect and diff services registered to itself. This is used during# anti-entropy to reconcile difference between the agents knowledge of registered# services and checks in comparison with what is known in the Catalog.service_prefix "" {  policy = "read"}

You can use node-identity to help you generate agent tokens for Consul nodes. Node identities let you to quickly construct policies for nodes, rather than manually creating identical polices for each node. Generate a new token for hashicups-api-1 using node-identity.

$ consul acl token create \  -description="hashicups-api-1 agent token" \  --format json \  -node-identity="hashicups-api-1:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.json

That will produce an output similar to the following.

{    "CreateIndex": 117,    "ModifyIndex": 117,    "AccessorID": "715cad9a-ebc5-21cd-0919-ac08effdbf3d",    "SecretID": "3c470114-1214-d4fe-e3c5-3dd1f9515373",    "Description": "hashicups-api-1 agent token",    "NodeIdentities": [        {            "NodeName": "hashicups-api-1",            "Datacenter": "dc1"        }    ],    "Local": false,    "CreateTime": "2023-11-18T13:27:42.918717795Z",    "Hash": "dn0S8locXJeqcWcmqHURzQDKSO4UZKwV49QO0YCCbv8="}

Generate config_file_service_registration token

The config_file_service_registration token needs write permission to register all services and checks defined in this agent's configuration. For Consul service mesh, the token also needs permissions to register the Envoy sidecar proxy for the service as an independent service in Consul catalog. One minimal example for such a policy is the following.

service-identity-policy-template.hcl

# Allow the service and its sidecar proxy to register into the catalog.service "<service name>" {    policy = "write"}service "<service name>-sidecar-proxy" {    policy = "write"}# Allow for any potential upstreams to be resolved.service_prefix "" {    policy = "read"}node_prefix "" {    policy = "read"}

You can use service-identity to help you generate config_file_service_registration tokens for Consul nodes. Service identities let you to quickly construct policies for services, rather than manually creating identical polices for each service instance and for their companion sidecar proxy. Generate a new token for hashicups-api-1 using service-identity.

$ consul acl token create \  -description="hashicups-api-1 config_file_service_registration token" \  --format json \  -service-identity="hashicups-api:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json

That will produce an output similar to the following.

{    "CreateIndex": 118,    "ModifyIndex": 118,    "AccessorID": "8cf68272-25ce-0d87-8dec-713890e8a0ba",    "SecretID": "75906ad5-2162-efcb-cf95-63a2b6396840",    "Description": "hashicups-api-1 config_file_service_registration token",    "ServiceIdentities": [        {            "ServiceName": "hashicups-api",            "Datacenters": [                "dc1"            ]        }    ],    "Local": false,    "CreateTime": "2023-11-18T13:27:42.963791211Z",    "Hash": "WCtk+YLUNIm5VR0zLykM9oo5TG4xdBbwAIBb3fSZ2qA="}

Verify you have created the three tokens and the files containing the tokens are present in the assets/ folder.

$ ls ~/assets/scenario/conf/secrets/ | grep hashicups-api-1acl-token-agent-hashicups-api-1.jsonacl-token-default-hashicups-api-1.jsonacl-token-svc-hashicups-api-1.json

Generate Consul agent ACL configuration section

With the three tokens you created, generate the Consul agent ACL configuration.

$ tee /home/admin/assets/scenario/conf/hashicups-api-1/agent-acl-tokens.hcl > /dev/null << EOFacl {  tokens {    agent  = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.json | jq -r ".SecretID"`"    default  = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.json | jq -r ".SecretID"`"    config_file_service_registration = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`"  }}EOF

Generate hashicups-api service configuration

Generate hashicups-api service configuration.

$ tee /home/admin/assets/scenario/conf/hashicups-api-1/svc-hashicups-api.hcl > /dev/null << EOF## -----------------------------## svc-hashicups-api.hcl## -----------------------------service {  name = "hashicups-api"  id = "hashicups-api-1"  tags = [ "inst_1" ]  port = 8081  token = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`"  connect {    sidecar_service {        proxy {          upstreams = [            {              destination_name = "hashicups-db"              local_bind_port = 5432            }          ]        }     }  }  checks =[  {  id =  "check-hashicups-api.public.http",  name = "hashicups-api.public  HTTP status check",  service_id = "hashicups-api-1",  http  = "http://localhost:8081/health",  interval = "5s",  timeout = "5s"  },  {    id =  "check-hashicups-api.public",    name = "hashicups-api.public status check",    service_id = "hashicups-api-1",    tcp  = "localhost:8081",    interval = "5s",    timeout = "5s"  },  {    id =  "check-hashicups-api.product",    name = "hashicups-api.product status check",    service_id = "hashicups-api-1",    tcp  = "localhost:9090",    interval = "5s",    timeout = "5s"  },  {    id =  "check-hashicups-api.payments",    name = "hashicups-api.payments status check",    service_id = "hashicups-api-1",    tcp  = "localhost:8080",    interval = "5s",    timeout = "5s"  }]}EOF

The tutorial shows an example of how to configure a service in Consul service mesh. The same concepts apply to a Consul service discovery. If you are trying to register a new service for service discovery, remove the connect section in the service configuration file.

Add new service to Consul service mesh with ACLs

In the previous paragraphs, you created agent-acl-tokens.hcl, containing token configuration for the Consul agent, and svc-hashicups-api.hcl, containing service configuration for the hashicups-api service instance. The bastion host already contains the remaining configuration needed to configure Consul on the hashicups-api-1 node. Review the full configuration.

$ ls /home/admin/assets/scenario/conf/hashicups-api-1agent-acl-tokens.hclagent-gossip-encryption.hclconsul-agent-ca.pemconsul.hclsvc-hashicups-api.hcl

Copy the configuration on the remote node.

$ scp -r -i /home/admin/certs/id_rsa /home/admin/assets/scenario/conf/hashicups-api-1/* admin@hashicups-api-1:/etc/consul.d/agent-acl-tokens.hcl                  ...agent-gossip-encryption.hcl           ...consul-agent-ca.pem                   ...consul.hcl                            ...svc-hashicups-api.hcl                 ...

Start Consul process

Login to hashicups-api-1 from the bastion host.

$ ssh -i certs/id_rsa hashicups-api-1#..admin@hashicups-api-1:~

Verify the VM contains the files required for Consul configuration.

$ ls /etc/consul.dagent-acl-tokens.hclagent-gossip-encryption.hclconsul-agent-ca.pemconsul.hclsvc-hashicups-api.hcl

Start Consul agent.

$ /usr/bin/consul agent \  -retry-join=consul \  -log-file=/tmp/consul-client \  -config-dir=/etc/consul.d > /tmp/consul-client.log 2>&1 &

The command starts the Consul server in the background to not lock the terminal. You can access the Consul server log through the /tmp/consul-server.log file.

Start Envoy sidecar proxy

To make the API service instance join Consul service mesh, your node requires an Envoy proxy acting as a sidecar for the service. You need a valid token with the permission to register the sidecar as a service in Consul catalog to start the proxy process.

Use the token created previously using the service-identity policy.

$ export _agent_token=`cat /etc/consul.d/agent-acl-tokens.hcl | grep -Po "(?<=registration = \")[^\"]+(?=\")"` && echo ${_agent_token}

That will produce an output similar to the following.

75906ad5-2162-efcb-cf95-63a2b6396840

Start Envoy sidecar proxy for hashicups-api service.

$ /usr/bin/consul connect envoy \  -token=${_agent_token} \  -envoy-binary /usr/bin/envoy \  -sidecar-for hashicups-api-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Start the API service instance

Once the Consul node joined the Consul datacenter and the Envoy sidecar is configured and ready to route traffic to and from your service, start the hashicups-api service to listen on the local network interface.

$ ~/start_service.sh local

The output of the command will show some Docker output while the Docker images that compose the services are being pulled on the node.

To continue with the tutorial, exit the ssh session to return to the bastion host.

$ exitlogoutConnection to hashicups-api-1 closed.admin@bastion:~$

Verify service registration

After you registered the service, it will start appearing among the results when Consul catalog is queried.

Tip

To simplify tests all services already present in the Consul service mesh had a tag assigned, inst_0 while the new instance got registered using a different tag, inst_1.

Verify it using the different interfaces exposed by Consul.

Use the consul catalog command to query the Consul catalog and show all available services.

$ consul catalog services -tags

The command will produce an output similar to the following.

consul                                gateway-api                           hashicups-api                         inst_0,inst_1hashicups-api-sidecar-proxy           inst_0,inst_1hashicups-db                          inst_0hashicups-db-sidecar-proxy            inst_0hashicups-frontend                    inst_0hashicups-frontend-sidecar-proxy      inst_0hashicups-nginx                       inst_0hashicups-nginx-sidecar-proxy         inst_0

Notice that the hashicups-api and hashicups-api-sidecar-proxy instances present two different tags, inst_0 and inst_1, that indicate that both instances are registered within Consul.

Destroy the infrastructure

Now that the tutorial is complete, clean up the infrastructure you created.

From the ./self-managed/infrastruture/aws folder of the repository, use terraform to destroy the infrastructure.

$ terraform destroy --auto-approve

Next steps

In this tutorial you learned the necessary steps to generate Access Control Lists' tokens to add a new Consul agent to your Consul datacenter and to register a new service to your Consul service mesh.

Specifically you:

  • Generated a new management token used the built-in global-management policy.
  • Generated a new token from a pre-existing policy
  • Generated a new token using the node-identity policy
  • Generated a new token using the service-identity policy
  • Verified the successful node and service registration using all the different interfaces provided by Consul: CLI, API, DNS, and UI

For more information about the topics covered in this tutorial, refer to the following resources:

To learn more about other security features provided by Consul service mesh refer to:

Previous

Service requests with L7 intentions

 Next

Permissive mTLS

Workflow