-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
connect proxy: x-forwarded-client-cert contains duplicate certificates in chain #15704
Comments
Hi @t-davies, It looks like you're using the Vault CA provider for Consul. Which Vault version is in use? If using Vault 1.11+, review this knowledge base article ASAP, including the recommended workaround. You can also follow the related issue on Github: #15217. |
Hey @jkirschner-hashicorp, it is indeed 1.11 - thanks for the info and the quick reply. |
@jkirschner-hashicorp looks like this is resolved in Consul 1.13.4? Would it be worth us just upgrading? |
It's resolved in Consul 1.13.4 for primary datacenters, but not secondary datacenters. Do you have a multi-datacenter Consul deployment (where the datacenters are connected with WAN federation, rather than just being disconnected, independent datacenters)? More details in this comment: #15217 (comment) |
I clarified this in the main changelog, but not in the list on the releases page. This is a good reminder for me to update the information on the releases page as well. |
Thanks for the info. We're single DC, so sounds like that should be ok for us. |
Just confirming that applying the workaround from the KB article did fix this issue, the chain in The date/time that we experienced this (06/12/2022 06:09:34) aligns with ~50% of the original intermediate CA lifespan (not after: 06/06/2023 10:22:22) too. |
Excellent - I'm glad to hear that upgrading to Consul 1.13.4 resolved the issue for you, and I appreciate the additional confirmation of the ~50% of the original intermediate CA lifespan timing. The more general issue (#15217) will remain open until we've released the fix for secondary datacenters, but I'll close this particular issue for now given your current status. Feel free to re-open if needed! |
Overview of the Issue
TLDR
The
x-forwarded-client-cert
value suddenly contains multiple duplicate certificates.--
We have a Consul cluster that has been running for several months without issue. The servers were upgraded from 1.12.3 to 1.13.3 approximately 1 month ago and have been running without issue since. Consul agents running on Nomad client nodes were upgraded afterwards over a period of ~2 weeks and, again, have been running without issue since.
On 6th December at approx 06:09 numerous applications running in the cluster that have traffic go via sidecar proxies start reporting issues with large headers.
We observe that
x-forwarded-client-cert
is indeed very large, luckily we don't depend on this and so we PUT /v1/configThis stops Envoy from sending the
x-forwarded-client-cert
header to our applications and resolves the large headers issues.Upon further investigation it seems that since 6th December at 06:09, the
Chain
value withinx-forwarded-client-cert
contains multiple duplicate issuer certs.Prior to 6th December, 06:09
Chain contains 2 certs - just listing the subjects here not the full certs.
Post to 6th December, 06:09
Chain contains 10 certs, the previously present 2 plus 8 duplicates - just listing the subjects here not the full certs.
This, obviously, makes the
x-forwarded-client-cert
header extremely large. Whilst everything is working now, since we are no longer passing this header to services - I'm assuming that this is not an intended behaviour.Reproduction Steps
See issue description for recent upgrade path, nothing specific was done to trigger this - it just started occurring.
Consul info for both Client and Server
Client info
Server info
Operating system and Environment details
Deployed to AWS on Amazon Linux 2, single region - 5 Consul servers.
Agents deployed on Nomad cluster nodes, sidecar proxies running as Nomad tasks.
envoy version: edd69583372955fdfa0b8ca3820dd7312c094e46/1.23.1/Clean/RELEASE/BoringSSL
Log Fragments
Nothing stands out as being obviously wrong in the logs.
The text was updated successfully, but these errors were encountered: