fix: update TLS config to support ALPN. Fixes #14422 #14435
Merged
+2
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As explained at argoproj#14422, any attempt to access the Argo server fails with the following error: ``` $ curl -kv https://localhost:2746/api/v1/workflow-templates/argo 2>&1 | grep ALPN * ALPN: curl offers h2,http/1.1 * ALPN: server did not agree on a protocol. Uses default. {"code":14,"message":"connection error: desc = \"transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: https://github.com/grpc/grpc-go/issues/434\""} ``` Despite appearances, this isn't actually a client-side issue. The problem is that that Go's `net/http` server requires explicitly enabling HTTP/2 in the TLS configuration, otherwise it won't return any negotiated protocols during the handshake. More details: * grpc-ecosystem/grpc-gateway#220 (comment) * grpc/grpc-go#434 (comment) The fix here is the same as in philips/grpc-gateway-example@e1dfd22. Tested locally by running `make start UI=true SECURE=true`, then running the following commands to verify both HTTP/2 and HTTP/1.1 requests work: ``` $ curl --http1.1 -kv https://localhost:2746/api/v1/workflow-templates/argo 2>&1 | grep ALPN * ALPN: curl offers http/1.1 * ALPN: server accepted http/1.1 $ curl -kv https://localhost:2746/api/v1/workflow-templates/argo 2>&1 | grep ALPN * ALPN: curl offers h2,http/1.1 * ALPN: server accepted h2 ``` This reverts commit 8d7dae6. Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>
Member
|
cc @Joibel |
Member
Author
|
/retest |
Member
|
Reminder to self: cherry-pick this forward to |
blkperl
approved these changes
May 14, 2025
Joibel
approved these changes
May 22, 2025
Member
|
/cherry-pick main |
Joibel
pushed a commit
that referenced
this pull request
May 22, 2025
Joibel
added a commit
that referenced
this pull request
May 23, 2025
…14490) Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> Co-authored-by: Mason Malone <651224+MasonM@users.noreply.github.com>
Joibel
added a commit
to pipekit/argo-workflows
that referenced
this pull request
Jun 4, 2025
…rgoproj#14435)" This reverts commit 899f42b.
MasonM
added a commit
to MasonM/argo-workflows
that referenced
this pull request
Jun 15, 2025
argoproj#14435 tried to fix the bug in the release-3.6 branch where requests failed with the following error: ``` Service Unavailable: connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: grpc/grpc-go#434" on the UI ``` Unfortunately, this didn't work and broke the readiness probe, so it was reverted in argoproj@1285c11. The problem is the readiness probe makes a GET request to `/`: https://github.com/argoproj/argo-workflows/blob/1e2a87f2afdebbcd0e55069df5a945f5faca9d45/manifests/base/argo-server/argo-server-deployment.yaml#L30-L36 With ALPN enabled, that request was automatically upraded to HTTP/2, which caused it to be forwarded to the gRPC server instead of the HTTP server. Initially, I tried to fix this by changing cmux to only forward requests with `Content-Type: application/grpc` (which is guaranteed to be present per the [gRPC over HTTP2 spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md)) to the gRPC server: argoproj/argo-workflows@main...MasonM:argo-workflows:fix-http2-multiplexing but that doesn't work with TLS enabled due to the following limitation mentioned at https://github.com/soheilhy/cmux/: > TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers. Instead, this uses the solution proposed in grpc/grpc-go#555 (comment), which relies on the [h2c package](https://pkg.go.dev/golang.org/x/net/http2/h2c). I added basic tests to `argo_server_test.go` for the readiness probe, but that only tests when TLS is disabled. I wrote following script to manually test with both HTTPS and HTTP: ```shell set -euo pipefail BASEURL="$scheme://localhost:2746" docurl() { printf "Testing %s with HTTP/1.1\n" "$*" curl -kv --http1.1 "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' printf "Testing %s with ALPN\n" "$*" curl -kv --alpn "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' } printf 'Testing root path\n' docurl "$BASEURL/" printf '\nTesting grpc-gateway\n' docurl "$BASEURL/api/v1/workflow-templates/argo" printf '\nTesting grpc server\n' docurl -XPOST -H 'Content-Type: application/grpc' "$BASEURL/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates" ``` <details> <summary>Output with HTTP</summary> ``` $ scheme=http ./test_mux.sh Testing root path Testing http://localhost:2746/ with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing http://localhost:2746/ with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing grpc-gateway Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing http://localhost:2746/api/v1/workflow-templates/argo with ALPN < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 ``` </details> <details> <summary>Output with HTTPS</summary> ``` $ scheme=https ./test_mux.sh Testing root path Testing https://localhost:2746/ with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing https://localhost:2746/ with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: text/html; charset=utf-8 Testing grpc-gateway Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing https://localhost:2746/api/v1/workflow-templates/argo with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/json < grpc-metadata-argo-version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/grpc ``` </details> Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>
MasonM
added a commit
to MasonM/argo-workflows
that referenced
this pull request
Jun 15, 2025
argoproj#14435 tried to fix the bug in the release-3.6 branch where requests failed with the following error: ``` Service Unavailable: connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: grpc/grpc-go#434" on the UI ``` Unfortunately, this didn't work and broke the readiness probe, so it was reverted in argoproj@1285c11. The problem is the readiness probe makes a GET request to `/`: https://github.com/argoproj/argo-workflows/blob/1e2a87f2afdebbcd0e55069df5a945f5faca9d45/manifests/base/argo-server/argo-server-deployment.yaml#L30-L36 With ALPN enabled, that request was automatically upraded to HTTP/2, which caused it to be forwarded to the gRPC server instead of the HTTP server. Initially, I tried to fix this by changing cmux to only forward requests with `Content-Type: application/grpc` (which is guaranteed to be present per the [gRPC over HTTP2 spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md)) to the gRPC server: argoproj/argo-workflows@main...MasonM:argo-workflows:fix-http2-multiplexing but that doesn't work with TLS enabled due to the following limitation mentioned at https://github.com/soheilhy/cmux/: > TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers. Instead, this uses the solution proposed in grpc/grpc-go#555 (comment), which relies on the [h2c package](https://pkg.go.dev/golang.org/x/net/http2/h2c). I added basic tests to `argo_server_test.go` for the readiness probe, but that only tests when TLS is disabled. I wrote following script to manually test with both HTTPS and HTTP: ```shell set -euo pipefail BASEURL="$scheme://localhost:2746" docurl() { printf "Testing %s with HTTP/1.1\n" "$*" curl -kv --http1.1 "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' printf "Testing %s with ALPN\n" "$*" curl -kv --alpn "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' } printf 'Testing root path\n' docurl "$BASEURL/" printf '\nTesting grpc-gateway\n' docurl "$BASEURL/api/v1/workflow-templates/argo" printf '\nTesting grpc server\n' docurl -XPOST -H 'Content-Type: application/grpc' "$BASEURL/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates" ``` <details> <summary>Output with HTTP</summary> ``` $ scheme=http ./test_mux.sh Testing root path Testing http://localhost:2746/ with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing http://localhost:2746/ with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing grpc-gateway Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing http://localhost:2746/api/v1/workflow-templates/argo with ALPN < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 ``` </details> <details> <summary>Output with HTTPS</summary> ``` $ scheme=https ./test_mux.sh Testing root path Testing https://localhost:2746/ with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing https://localhost:2746/ with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: text/html; charset=utf-8 Testing grpc-gateway Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing https://localhost:2746/api/v1/workflow-templates/argo with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/json < grpc-metadata-argo-version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/grpc ``` </details> Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>
MasonM
added a commit
to MasonM/argo-workflows
that referenced
this pull request
Jul 6, 2025
argoproj#14435 tried to fix the bug in the release-3.6 branch where requests failed with the following error: ``` Service Unavailable: connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: grpc/grpc-go#434" on the UI ``` Unfortunately, this didn't work and broke the readiness probe, so it was reverted in argoproj@1285c11. The problem is the readiness probe makes a GET request to `/`: https://github.com/argoproj/argo-workflows/blob/1e2a87f2afdebbcd0e55069df5a945f5faca9d45/manifests/base/argo-server/argo-server-deployment.yaml#L30-L36 With ALPN enabled, that request was automatically upraded to HTTP/2, which caused it to be forwarded to the gRPC server instead of the HTTP server. Initially, I tried to fix this by changing cmux to only forward requests with `Content-Type: application/grpc` (which is guaranteed to be present per the [gRPC over HTTP2 spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md)) to the gRPC server: argoproj/argo-workflows@main...MasonM:argo-workflows:fix-http2-multiplexing but that doesn't work with TLS enabled due to the following limitation mentioned at https://github.com/soheilhy/cmux/: > TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers. Instead, this uses the solution proposed in grpc/grpc-go#555 (comment), which relies on the [h2c package](https://pkg.go.dev/golang.org/x/net/http2/h2c). I added basic tests to `argo_server_test.go` for the readiness probe, but that only tests when TLS is disabled. I wrote following script to manually test with both HTTPS and HTTP: ```shell set -euo pipefail BASEURL="$scheme://localhost:2746" docurl() { printf "Testing %s with HTTP/1.1\n" "$*" curl -kv --http1.1 "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' printf "Testing %s with ALPN\n" "$*" curl -kv --alpn "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' } printf 'Testing root path\n' docurl "$BASEURL/" printf '\nTesting grpc-gateway\n' docurl "$BASEURL/api/v1/workflow-templates/argo" printf '\nTesting grpc server\n' docurl -XPOST -H 'Content-Type: application/grpc' "$BASEURL/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates" ``` <details> <summary>Output with HTTP</summary> ``` $ scheme=http ./test_mux.sh Testing root path Testing http://localhost:2746/ with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing http://localhost:2746/ with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing grpc-gateway Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing http://localhost:2746/api/v1/workflow-templates/argo with ALPN < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 ``` </details> <details> <summary>Output with HTTPS</summary> ``` $ scheme=https ./test_mux.sh Testing root path Testing https://localhost:2746/ with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing https://localhost:2746/ with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: text/html; charset=utf-8 Testing grpc-gateway Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing https://localhost:2746/api/v1/workflow-templates/argo with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/json < grpc-metadata-argo-version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/grpc ``` </details> Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #14422
Motivation
As explained at #14422, any attempt to access the Argo server fails with an ALPN error. Example using curl:
Despite appearances, this isn't actually a client-side issue. The problem is that Go's
net/httpserver requires explicitly enabling HTTP/2 in the TLS configuration, otherwise it won't return any negotiated protocols during the handshake. Here's the code that controls this: https://github.com/golang/go/blob/1e756dc5f73dc19eb1cbf038807d18ef1cc54ebc/src/net/http/server.go#L3399-L3406More details:
Modifications
First, I reverted #14433, since that workaround is no longer necessary. The actual fix is the same as in philips/grpc-gateway-example@e1dfd22. Note that even though only
h2is in the list, HTTP/1.1 connections still work (see below).Verification
Tested locally by running
make start UI=true SECURE=true, then running the following commands to verify both HTTP/2 and HTTP/1.1 requests work:I also verified http://localhost:8080/workflows loads properly, as well as https://localhost:8080/workflows when using
make start UI_SECURE=true SECURE=true.Documentation
N/A