Service ACL token used to perform blocked queries on behalf of a connect service is not refreshed #11949
Labels
theme/connect
Anything related to Consul Connect, Service Mesh, Side Car Proxies
Overview of the Issue
When a connect service is created on a cluster with ACL activated, The token used to create the service is stored in the agent and used for all the blocking queries related to that service. This token cannot be changed and if the token is deleted the service need to be recreated with a new token to continue to function properly.
Also a token is configured in envoy and provided as part of the envoy XDS requests, this token is authenticated and proper access privilege is checked for it but is not used for any agent to server operations related to that service.
Reproduction Steps
The issue that could happen is as follow:
At this point, when consul agent receive an XDS request from envoy it will authenticate token B (which is present and have the correct privileges). When the agent create blocking queries to update the service (for example leaf certificates) it will use token A which will be denied by the server (because token A is deleted). This lead to a service that never receive updates.
This is also true in the use case where customers use Nomad and Consul, the Nomad/Consul integration permit to Nomad to handle the lifecycle of Consul ACL tokens (Create, inject to envoy and delete), a discrepancy between what is injected in envoy and what the service have as a token could lead to the same issue.
Proposed Fix
Every time, consul agent receive a request from envoy with an ACL token the agent will refresh the service token with it to ensure that the same token is used to perform blocking queries on behalf of the service.
The text was updated successfully, but these errors were encountered: