fix: support ALPN and H2C with gRPC. Fixes #14627 #14567

MasonM · 2025-06-15T20:36:50Z

Motivation

#14435 tried to fix the bug in the release-3.6 branch where requests failed with the following error:

Service Unavailable: connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: https://github.com/grpc/grpc-go/issues/434" on the UI

Unfortunately, this didn't work and broke the readiness probe, so it was reverted in 1285c11. The problem is the readiness probe makes a GET request to /:

argo-workflows/manifests/base/argo-server/argo-server-deployment.yaml

Lines 30 to 36 in 1e2a87f

    
           readinessProbe: 
        
             httpGet: 
        
               port: 2746 
        
               scheme: HTTPS 
        
               path: / 
        
             initialDelaySeconds: 10 
        
             periodSeconds: 20

That request should be going to the HTTP server, and it does when using HTTP/1.1, since argoserver is configured to forward all HTTP/1.1 requests to the listener for the HTTP server:

argo-workflows/server/apiserver/argoserver.go

Lines 279 to 280 in 1e2a87f

    
           httpL := tcpm.Match(cmux.HTTP1Fast()) 
        
           grpcL := tcpm.Match(cmux.Any())

but with ALPN enabled, the client (the kubelet in this case) automatically upgraded the request to HTTP/2, which caused it to be forwarded to the gRPC server instead.

The reason this wasn't caught in #14435 is because the Argo server isn't run inside Kubernetes in either the local dev environment or CI builds, so the readiness probe wasn't being used. Additionally, http://localhost:8080/ in the local dev environment points to the webpack-dev-server, which proxies requests to the Argo server, without ALPN:

argo-workflows/ui/webpack.config.js

Lines 112 to 119 in 7fd6b10

    
           proxy: [ 
        
               { 
        
                   context: ['/api/v1', '/artifact-files', '/artifacts', '/input-artifacts', '/artifacts-by-uid', '/input-artifacts-by-uid', '/oauth2'], 
        
                   target: proxyTarget, 
        
                   secure: false, 
        
                   xfwd: true // add x-forwarded-* headers to simulate real-world reverse proxy servers 
        
               } 
        
           ]

Modifications

Initially, I tried to fix this by changing cmux to only forward requests with Content-Type: application/grpc (which is guaranteed to be present per the gRPC over HTTP2 spec) to the gRPC server:
main...MasonM:argo-workflows:fix-http2-multiplexing

but that doesn't work with TLS enabled due to the following limitation mentioned at https://github.com/soheilhy/cmux/:

TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers.

Link to the code it's referencing: https://github.com/golang/go/blob/1e756dc5f73dc19eb1cbf038807d18ef1cc54ebc/src/net/http/server.go#L1959

Instead, this uses the solution proposed in grpc/grpc-go#555 (comment), which relies on the h2c package.

I had to partially revert #14588 to restore the changes in util/tls/tls.go that enabled ALPN. To be extra safe, I didn't remove the ENV GRPC_ENFORCE_ALPN_ENABLED=false workaround yet, though it should theoretically be okay to remove now.

Verification

I added basic tests to argo_server_test.go for the readiness probe, but that only tests when TLS is disabled. I wrote the following script to manually test with both HTTPS and HTTP:

#!/bin/bash

set -euo pipefail
BASEURL="$scheme://localhost:2746"
docurl() {
    printf "Testing %s with HTTP/1.1\n" "$*"
    curl -kv --http1.1 "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version'

    printf "Testing %s with HTTP/2 with prior knowledge\n" "$*"
    curl -kv --http2-prior-knowledge "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version'

    printf "Testing %s with ALPN\n" "$*"
    curl -kv --alpn "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version'
}

printf 'Testing root path\n'
docurl "$BASEURL/"

printf '\nTesting grpc-gateway\n'
docurl "$BASEURL/api/v1/workflow-templates/argo"

printf '\nTesting grpc server\n'
docurl -XPOST -H 'Content-Type: application/grpc' "$BASEURL/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates"

Output with HTTP

$ scheme=http ./test_mux.sh 
Testing root path                                                                                                                                                                                                                                    
Testing http://localhost:2746/ with HTTP/1.1                                                                                                                                                                                                         
< HTTP/1.1 200 OK                                                                                                                                                                                                                                    
< Content-Type: text/html; charset=utf-8
Testing http://localhost:2746/ with HTTP/2 with prior knowledge
< HTTP/2 200                                                                                                              
< content-type: text/html; charset=utf-8
Testing http://localhost:2746/ with ALPN     
< HTTP/1.1 200 OK           
< Content-Type: text/html; charset=utf-8                 
                                                             
Testing grpc-gateway                    
Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1
< HTTP/1.1 200 OK     
< Content-Type: application/json
< Grpc-Metadata-Argo-Version: latest+19a02a0.dirty
Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/2 with prior knowledge
< HTTP/2 200                             
< content-type: application/json
< grpc-metadata-argo-version: latest+19a02a0.dirty
Testing http://localhost:2746/api/v1/workflow-templates/argo with ALPN
< HTTP/1.1 200 OK                       
< Content-Type: application/json
< Grpc-Metadata-Argo-Version: latest+19a02a0.dirty
                                                                                                                          
Testing grpc server         
Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/2 with prior knowledge
< HTTP/2 200                                                                                                              
< content-type: application/grpc
Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8

Output with HTTPS

$ scheme=https ./test_mux.sh
Testing root path
Testing https://localhost:2746/ with HTTP/1.1
* ALPN: curl offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
Testing https://localhost:2746/ with HTTP/2 with prior knowledge
* ALPN: curl offers h2
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: text/html; charset=utf-8
Testing https://localhost:2746/ with ALPN
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: text/html; charset=utf-8

Testing grpc-gateway
Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1
* ALPN: curl offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
< HTTP/1.1 200 OK
< Content-Type: application/json
< Grpc-Metadata-Argo-Version: latest+19a02a0.dirty
Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/2 with prior knowledge
* ALPN: curl offers h2
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: application/json
< grpc-metadata-argo-version: latest+19a02a0.dirty
Testing https://localhost:2746/api/v1/workflow-templates/argo with ALPN
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: application/json
< grpc-metadata-argo-version: latest+19a02a0.dirty

Testing grpc server
Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1
* ALPN: curl offers http/1.1
* ALPN: server did not agree on a protocol. Uses default.
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/2 with prior knowledge
* ALPN: curl offers h2
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: application/grpc
Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
< HTTP/2 200 
< content-type: application/grpc

Documentation

N/A

argoproj#14435 tried to fix the bug in the release-3.6 branch where requests failed with the following error: ``` Service Unavailable: connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: grpc/grpc-go#434" on the UI ``` Unfortunately, this didn't work and broke the readiness probe, so it was reverted in argoproj@1285c11. The problem is the readiness probe makes a GET request to `/`: https://github.com/argoproj/argo-workflows/blob/1e2a87f2afdebbcd0e55069df5a945f5faca9d45/manifests/base/argo-server/argo-server-deployment.yaml#L30-L36 With ALPN enabled, that request was automatically upraded to HTTP/2, which caused it to be forwarded to the gRPC server instead of the HTTP server. Initially, I tried to fix this by changing cmux to only forward requests with `Content-Type: application/grpc` (which is guaranteed to be present per the [gRPC over HTTP2 spec](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md)) to the gRPC server: argoproj/argo-workflows@main...MasonM:argo-workflows:fix-http2-multiplexing but that doesn't work with TLS enabled due to the following limitation mentioned at https://github.com/soheilhy/cmux/: > TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers. Instead, this uses the solution proposed in grpc/grpc-go#555 (comment), which relies on the [h2c package](https://pkg.go.dev/golang.org/x/net/http2/h2c). I added basic tests to `argo_server_test.go` for the readiness probe, but that only tests when TLS is disabled. I wrote following script to manually test with both HTTPS and HTTP: ```shell set -euo pipefail BASEURL="$scheme://localhost:2746" docurl() { printf "Testing %s with HTTP/1.1\n" "$*" curl -kv --http1.1 "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' printf "Testing %s with ALPN\n" "$*" curl -kv --alpn "$@" 2>&1 | egrep -i 'ALPN|^< HTTP|< Content-Type|< Grpc-Metadata-Argo-Version' } printf 'Testing root path\n' docurl "$BASEURL/" printf '\nTesting grpc-gateway\n' docurl "$BASEURL/api/v1/workflow-templates/argo" printf '\nTesting grpc server\n' docurl -XPOST -H 'Content-Type: application/grpc' "$BASEURL/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates" ``` <details> <summary>Output with HTTP</summary> ``` $ scheme=http ./test_mux.sh Testing root path Testing http://localhost:2746/ with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing http://localhost:2746/ with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing grpc-gateway Testing http://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing http://localhost:2746/api/v1/workflow-templates/argo with ALPN < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc http://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 ``` </details> <details> <summary>Output with HTTPS</summary> ``` $ scheme=https ./test_mux.sh Testing root path Testing https://localhost:2746/ with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing https://localhost:2746/ with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: text/html; charset=utf-8 Testing grpc-gateway Testing https://localhost:2746/api/v1/workflow-templates/argo with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: application/json < Grpc-Metadata-Argo-Version: latest+94f9148.dirty Testing https://localhost:2746/api/v1/workflow-templates/argo with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/json < grpc-metadata-argo-version: latest+94f9148.dirty Testing grpc server Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with HTTP/1.1 * ALPN, offering http/1.1 * ALPN, server did not agree to a protocol < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 Testing -XPOST -H Content-Type: application/grpc https://localhost:2746/workflowtemplate.WorkflowTemplateService/ListWorkflowTemplates with ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2 < HTTP/2 200 < content-type: application/grpc ``` </details> Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM · 2025-07-14T01:20:37Z

/retest

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM · 2025-07-16T02:55:57Z

util/grpc/mux.go

+// to an HTTP/2 connection which is understandable to s.ServeConn. (s.ServeConn
+// understands HTTP/2 except for the h2c part of it.)"
+func NewMuxHandler(grpcServerHandler http.Handler, httpServerHandler http.Handler) http.Handler {
+	return h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {


One possible concern here is performance, since h2c buffers the entire request in memory, unlike cmux. I'm not sure if that's actually going to be a problem here.

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM · 2025-07-16T03:50:19Z

/retest

Joibel · 2025-07-25T10:10:56Z

This LGTM, and works for me. Should we merge the reversion of the ENV var in the Dockerfile into this PR, or do you think it wants doing separately?

MasonM · 2025-07-27T19:30:33Z

@Joibel Thanks for the review! I was thinking of doing it separately to be extra safe.

isubasinghe

Seems fine, haven't tested it so just leaving a comment, code looks good.
Thanks for the work!

Joibel · 2025-07-28T09:08:09Z

/cherry-pick release-3.6

Joibel · 2025-07-28T09:08:20Z

/cherry-pick release-3.7

gcp-cherry-pick-bot · 2025-07-28T09:08:36Z

Cherry-pick failed with Merge error 9ed9ed9972616395dd3457a47092de07b75fb5f9 into temp-cherry-pick-fd80cc-release-3.6

gcp-cherry-pick-bot · 2025-07-28T09:08:39Z

Cherry-pick failed with Merge error 9ed9ed9972616395dd3457a47092de07b75fb5f9 into temp-cherry-pick-fd80cc-release-3.7

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9)

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9) (cherry picked from commit 44754be)

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9)

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9) (cherry picked from commit 44754be)

…ck 3.7) (#14741) Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> Co-authored-by: Mason Malone <651224+MasonM@users.noreply.github.com>

…ck release-3.6) (#14742) Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> Co-authored-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM force-pushed the fix-tls-config branch from d763e69 to aac14f3 Compare July 6, 2025 00:34

MasonM added 2 commits July 5, 2025 18:44

fix: handle RST_STREAM errors due to Content-Length mismatch

19a02a0

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

fix: restore ALPN negotiation

52d2a18

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM force-pushed the fix-tls-config branch from aac14f3 to 52d2a18 Compare July 6, 2025 02:02

MasonM changed the title ~~fix: support ALPN and H2C with gRPC~~ fix: support ALPN and H2C with gRPC. Fixes #14627 Jul 6, 2025

MasonM marked this pull request as ready for review July 6, 2025 02:38

MasonM requested review from Joibel and isubasinghe July 6, 2025 02:38

Merge branch 'main' into fix-tls-config

8dd8e89

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM added 2 commits July 15, 2025 19:15

fix: additional comments

9f69a8f

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

Merge branch 'main' into fix-tls-config

51db4e5

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

MasonM commented Jul 16, 2025

View reviewed changes

chore: cleanup newHTTPServer

04bb93d

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com>

Joibel self-assigned this Jul 25, 2025

Joibel approved these changes Jul 25, 2025

View reviewed changes

isubasinghe reviewed Jul 28, 2025

View reviewed changes

Joibel merged commit 9ed9ed9 into argoproj:main Jul 28, 2025
60 of 62 checks passed

This was referenced Aug 5, 2025

Release v3.7 patch releases discussion #14509

Open

Release v3.6 patch releases discussion #14259

Open

Joibel pushed a commit that referenced this pull request Aug 5, 2025

fix: support ALPN and H2C with gRPC. Fixes #14627 (#14567)

44754be

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9)

Joibel pushed a commit that referenced this pull request Aug 5, 2025

fix: support ALPN and H2C with gRPC. Fixes #14627 (#14567)

d9a0086

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9)

Joibel pushed a commit that referenced this pull request Aug 5, 2025

fix: support ALPN and H2C with gRPC. Fixes #14627 (#14567)

0629bce

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9) (cherry picked from commit 44754be)

Joibel pushed a commit that referenced this pull request Aug 5, 2025

fix: support ALPN and H2C with gRPC. Fixes #14627 (#14567)

1a292c4

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9)

Joibel pushed a commit that referenced this pull request Aug 5, 2025

fix: support ALPN and H2C with gRPC. Fixes #14627 (#14567)

d357886

Signed-off-by: Mason Malone <651224+MasonM@users.noreply.github.com> (cherry picked from commit 9ed9ed9) (cherry picked from commit 44754be)

Joibel mentioned this pull request Aug 5, 2025

chore: revert fix: ignore ALPN by using the env variable hack (#14588) #14743

Merged

MasonM mentioned this pull request Oct 27, 2025

REQUEST: Promotion to Approver for MasonM argoproj/argoproj#399

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: support ALPN and H2C with gRPC. Fixes #14627 #14567

fix: support ALPN and H2C with gRPC. Fixes #14627 #14567

Uh oh!

MasonM commented Jun 15, 2025 •

edited

Loading

Uh oh!

MasonM commented Jul 14, 2025

Uh oh!

MasonM Jul 16, 2025

Uh oh!

MasonM commented Jul 16, 2025

Uh oh!

Joibel commented Jul 25, 2025

Uh oh!

MasonM commented Jul 27, 2025

Uh oh!

isubasinghe left a comment

Uh oh!

Joibel commented Jul 28, 2025

Uh oh!

Joibel commented Jul 28, 2025

Uh oh!

Uh oh!

gcp-cherry-pick-bot bot commented Jul 28, 2025

Uh oh!

gcp-cherry-pick-bot bot commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	readinessProbe:
	httpGet:
	port: 2746
	scheme: HTTPS
	path: /
	initialDelaySeconds: 10
	periodSeconds: 20

	httpL := tcpm.Match(cmux.HTTP1Fast())
	grpcL := tcpm.Match(cmux.Any())

	proxy: [
	{
	context: ['/api/v1', '/artifact-files', '/artifacts', '/input-artifacts', '/artifacts-by-uid', '/input-artifacts-by-uid', '/oauth2'],
	target: proxyTarget,
	secure: false,
	xfwd: true // add x-forwarded-* headers to simulate real-world reverse proxy servers
	}
	]

fix: support ALPN and H2C with gRPC. Fixes #14627 #14567

fix: support ALPN and H2C with gRPC. Fixes #14627 #14567

Uh oh!

Conversation

MasonM commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verification

Documentation

Uh oh!

MasonM commented Jul 14, 2025

Uh oh!

MasonM Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

MasonM commented Jul 16, 2025

Uh oh!

Joibel commented Jul 25, 2025

Uh oh!

MasonM commented Jul 27, 2025

Uh oh!

isubasinghe left a comment

Choose a reason for hiding this comment

Uh oh!

Joibel commented Jul 28, 2025

Uh oh!

Joibel commented Jul 28, 2025

Uh oh!

Uh oh!

gcp-cherry-pick-bot bot commented Jul 28, 2025

Uh oh!

gcp-cherry-pick-bot bot commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MasonM commented Jun 15, 2025 •

edited

Loading