Evaluate & potentially replace https://github.com/soheilhy/cmux

[ahrtr]

What would you like to be added?

etcdserver depends on https://github.com/soheilhy/cmux to receive both gRPC and REST connections on the same TCP listener/port. But the library isn't well maintained; note it hasn't been updated for more than 3 years at the time of writing this ticket. It also has issue working with HTTP/2, please refer to soheilhy/cmux/issues.

We should evaluate & summarize the problems the library has, and

• either try to find an alternative solution.
• or completely separate gRPC and REST/HTTP connections;

I expect someone with more network expertise could drive this effort.

Why is this needed?

See above

[serathius]

cc @aojea

[fuweid]

The ServeHTTP is still experimental feature in gRPC side https://pkg.go.dev/google.golang.org/grpc#Server.ServeHTTP.
There is different between gRPC and net/http about the underlying implementation of HTTP2 transport.
As far as I know, the gRPC implementation has some auto-tune features on window-update.
If there is good performance provided gRPC, I think we can consider using separate connections.

[serathius]

I think we can consider using separate connections.

Using separate connections is much better as shown by #15402, however removing connection multiplexing would be a breaking change for users. If we look at the list of endpoints in #15402 (comment), the most impacted would be would be:

• Users of /health via health probes, so I assume most of the users.
• Users of /v3/* v3 grpc gateway, which based on the questionnaire we sent are more than we expected as not every programming language supports grpc.
• Users of /metrics that didn't separate the port via --listen-metrics-urls.

[rleungx]

We encountered this issue grpc/grpc-go#7261 caused by serverHandlerTransport.

[serathius]

@rleungx Looked into the problem. Doesn't look it affects etcd as we don't allow arbitrary as we don't use stream requests with large amounts of data.

[rleungx]

@rleungx Looked into the problem. Doesn't look it affects etcd as we don't allow arbitrary as we don't use stream requests with large amounts of data.

We are using the embedded etcd which handles both HTTP and gRPC requests.