[BUG] Cannot communicate with http2 when reactor-netty is enabled

[rithin-pullela-aws]

Describe the bug

When an OpenSearch cluster is setup up http.type as reactor-netty-secure, the curl invocation of any endpoints fails. The reason is the client(curl) and the server try to communicate via http2. When http1.1 is forced we do not see the error, indicating the server fails to support http2 when reactor netty is enabled.

Test results:

1. Failure case:

curl -XGET <URL>/_cat/nodes -u admin:[redacted pass]! --insecure -k -v
...

* ALPN: curl offers h2,http/1.1  <- curl talking to server
...
*handshake*
...
* ALPN: server accepted h2   <- Server accepting h2
...
...
* using HTTP/2 
* Server auth using Basic with user 'admin'
* [HTTP/2] [1] OPENED stream for <URL>/_cat/nodes
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: <URL>]
* [HTTP/2] [1] [:path: /_cat/nodes]
* [HTTP/2] [1] [authorization: Basic [redacted base 64]]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /_cat/nodes HTTP/2
> Host: <URL>
> Authorization: Basic [redacted base 64]
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
* Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Closing connection
curl: (16) Remote peer returned unexpected data while we expected SETTINGS frame.  
Perhaps, peer does not support HTTP/2 properly.

The same command works if we enforce http1.1

curl -XGET <URL>/_cat/nodes -u admin:[redacted pass]! --insecure -k -v --http1.1
...
* ALPN: curl offers http/1.1
...
* ALPN: server accepted http/1.1
... 
* using HTTP/1.x
...
Successful Response:
172.31.45.34  37 19 0 0.00 0.00 0.00 di data,ingest           - data1
172.31.39.223 56 53 0 0.00 0.00 0.00 m  cluster_manager       * master1
172.31.33.42  57 80 0 0.00 0.00 0.00 r  remote_cluster_client - client1
172.31.33.5   24 19 0 0.00 0.00 0.00 di data,ingest           - data2

Related component

Other

To Reproduce

• I used the latest RC for 3.1
• Install the transport reactor netty 4 plugin.
• Enable reactor netty by adding to opensearch.yml: http.type: reactor-netty4-secure
• Make a http2 call to the openSearch server.

Expected behavior

Ideally there should be no issue communicating in http2 standard.

Additional Details

Tested against 3.1 RC2

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.