Proxy TCP connections over HTTP2 connection pools

[simonsj]

I'd like to use Envoy as something like a stunnel replacement, but leveraging long-lived HTTP2 connection pools, like this:

client <-----> E1 <-----------> E2 <-----> server
         raw       encrypted         raw
         TCP       conn. pooled      TCP
                   HTTP2 tunnel

^ where neither the client nor server speaks any HTTP.

Clients use servers' IP addresses directly, so no DNS is involved.

I was thinking it should be possible to map TCP flows like that over long-lived HTTP2 tunnels.

After reading over issues and the mailing lists and experimenting a bit locally, it's not clear to me that this kind of mode exists today as-is. But I could have missed something.

Is it possible to configure Envoy for something like this? Would it be feasible to add such a mode?

[mattklein123]

This is not supported out of the box. What does the HTTP/2 tunneling give you exactly? Are the TCP connections short lived? If this is needed a custom protocol is required. The best bet is probably some type of bi-directional gRPC streaming API that gets converted back to raw TCP on either side. You could build Envoy filters fairly easily to accomplish this.

[simonsj]

What does the HTTP/2 tunneling give you exactly?

What I'm after to is multiplex multiple streams over the single encrypted HTTP/2 connection, trying to avoid needing to set up one new connection per raw TCP flow. I'd ideally be able to reuse that connection for lots of flows over time.

Are the TCP connections short lived?

Yes, in general; at most these connections could last up to 2 hours but most would fall into the seconds range.

If this is needed a custom protocol is required. The best bet is probably some type of bi-directional gRPC streaming API that gets converted back to raw TCP on either side. You could build Envoy filters fairly easily to accomplish this.

Thanks, this is helpful, and good to know there wasn't something in-the-box that I missed. I'll study up on the filter code.

[mattklein123]

@simonsj if you potentially want to implement I can provide tips / rough overview. If this is something that the community would find useful we can upstream also.

[simonsj]

@simonsj if you potentially want to implement I can provide tips / rough overview. If this is something that the community would find useful we can upstream also.

@mattklein123 that would be awesome, I'd love to give this a try.

[mattklein123]

@simonsj here is a rough outline:

1. I would define a gRPC streaming API that basically passes a trivial message back and forth. Each message would just contain raw bytes (you might potentially want to carry some kind of routing data along with the bytes but that's separate).

2. You could follow along with tcp_proxy filter: https://github.com/envoyproxy/envoy/blob/master/source/common/filter/tcp_proxy.h. Specifically here: https://github.com/envoyproxy/envoy/blob/master/source/common/filter/tcp_proxy.cc#L207. Instead of making a raw TCP connection, you could instantiate a typed gRPC client following along with here: https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_subscription_impl.h#L24

The above would give you the egress side of the equation. Then for ingress, you would write an HTTP filter which basically does tcp_proxy filter but in reverse.

1. Parse gRPC incoming stream using grpc codec.
2. Make TCP connection to local host or wherever.
3. Feed bytes.

[mattklein123]

Switching this to a feature request. I think Google needs this also so they might implement. cc @alyssawilk

[PiotrSikora]

Please don't reinvent the wheel. This should use extended CONNECT method proposed in https://tools.ietf.org/html/draft-mcmanus-httpbis-h2-websockets, and proxy TCP stream as a HTTP/2 stream without any extra gRPC encapsulation.

client <-----> E1 <-----------------> E2 <-----> server
         raw       HTTP/2 connection       raw
         TCP       :method = CONNECT       TCP
                   :protocol = tcp

[mattklein123]

@PiotrSikora agreed. I was not aware of this proposal. Yes, let's implement this for sure.

[songole]

@PiotrSikora @mattklein123 We are also interested in this solution. However, the rfc is a draft and I guess the preferred solution may not be available anytime soon. Do you still recommend the previous approach in the mean time? Or, is there an easier solution with Websocket and avoid HTTP/2 ? The payload to tunnel for us is HTTP/1.1.

[PiotrSikora]

@songole I don't have cycles to work on it right now, but I do recommend implementing the draft, the final version won't differ much from it (if at all).