Service ACL token used to perform blocked queries on behalf of a connect service is not refreshed

[dhiaayachi]

Overview of the Issue

When a connect service is created on a cluster with ACL activated, The token used to create the service is stored in the agent and used for all the blocking queries related to that service. This token cannot be changed and if the token is deleted the service need to be recreated with a new token to continue to function properly.

Also a token is configured in envoy and provided as part of the envoy XDS requests, this token is authenticated and proper access privilege is checked for it but is not used for any agent to server operations related to that service.

Reproduction Steps

The issue that could happen is as follow:

• a service is created with a token A
• an envoy is bootstrapped with the same token A
• At this point everything work correctly
• after a while, the service operator decide to rotate the token and create token B with the same policies
• envoy config for the service is updated with token B
• token A is deleted

At this point, when consul agent receive an XDS request from envoy it will authenticate token B (which is present and have the correct privileges). When the agent create blocking queries to update the service (for example leaf certificates) it will use token A which will be denied by the server (because token A is deleted). This lead to a service that never receive updates.

This is also true in the use case where customers use Nomad and Consul, the Nomad/Consul integration permit to Nomad to handle the lifecycle of Consul ACL tokens (Create, inject to envoy and delete), a discrepancy between what is injected in envoy and what the service have as a token could lead to the same issue.

Proposed Fix

Every time, consul agent receive a request from envoy with an ACL token the agent will refresh the service token with it to ensure that the same token is used to perform blocking queries on behalf of the service.