Envoy 1.19 is unable to communicate with downstream Dotnet gRPC service over h2c

[bourquep]

What steps did you take and what happened:
After upgrading from Contour 1.10 to 1.18 (which ships with Envoy 1.19), my grpc-web clients are unable to talk with a Dotnet grpc service.

Envoy logs:

upstream reset: reset reason: remote reset, transport failure reason:

Dotnet logs:

HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'

This is very similar to this Istio issue: istio/istio#34448

What did you expect to happen:

Contour/envoy should still allow me to connect to downstream gRPC services over h2c.

Anything else you would like to add:

grpc-web client -- https --> Envoy -- gRPC h2c --> Dotnet 5 service

Environment:

• Contour version: 1.18
• Kubernetes version: (use kubectl version): v1.20.8-gke.900
• Kubernetes installer & version:
• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):

[bourquep]

After a bit of digging, this problem was actually introduced in Envoy 1.18:

http: fixing a standards compliance issue with :scheme. The :scheme header sent upstream is now based on the original URL scheme, rather than set based on the security of the upstream connection. This behavior can be temporarily reverted by setting envoy.reloadable_features.preserve_downstream_scheme to false.

The problem with this is that Dotnet Kestrel refuses any connections whose :scheme header doesn't match the actual underlying transport.

Because it is Envoy that terminates the public SSL connections, and it communicates to my service via h2c, it should not set the scheme header to https.

I could revert to the "old, working way" by setting envoy.reloadable_features.preserve_downstream_scheme to false, but I am not sure how to do this... There is nothing in the documentation that mentions how we can customize the Envoy config.

[youngnick]

Sorry for the delay on responding to this one @bourquep.

It seems like you would like a way to set envoy.reloadable_features.preserve_downstream_scheme to false somehow by Contour?

This sounds like it should be okay, we would need to check the details to see if there are any other implications, and we would probably want to start with a Contour-global setting rather than a host-specific one. Do you think that would be acceptable? (Obviously, pending investigation and other maintainer agreement)

[bourquep]

@youngnick Indeed, as far as I can tell by my research, setting envoy.reloadable_features.preserve_downstream_scheme to false would fix the problem with Kestrel. All I need from Contour is a way to configure this 😄.

In my personal use cases, a Contour-global setting would work as my services are all written in .Net. But I could imaging that having a per-route setting could be useful to those who have services implemented in a variety of languages.

[youngnick]

I've dropped this onto our board for prioritization.

[davidovich]

We are affected by this issue also. Since we set hc2 at Contour HTTPProxy configuration to allow clear text HTTP/2 and that TLS is terminated at envoy, is there a way to set the scheme to http ?

As this change in Envoy seems to enforce the standard, I do not understand how passing the original scheme is meant to work with h2c as clearly the upstream cannot use the https scheme if TLS is already terminated at Envoy.

[davidovich]

Note that this issue seems only to affect .NET core and version 6 should provide a setting to fix this issue.

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack