Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP/2 Benchmarks on reactor netty 2 don't work #2829

Open
pderop opened this issue Jun 9, 2023 · 2 comments
Open

HTTP/2 Benchmarks on reactor netty 2 don't work #2829

pderop opened this issue Jun 9, 2023 · 2 comments
Labels
type/bug A general bug
Milestone

Comments

@pderop
Copy link
Contributor

pderop commented Jun 9, 2023

Expected Behavior

Be able to run some Gatling HTTP/2 bench using reactor netty 2 and netty 5.

Actual Behavior

HTTP/2 gatling benchmarks are failing with errors like this:

> Request timeout to 127.0.0.1/127.0.0.1:8080 after 60000 ms      15535 (62.97%)
> i.n.c.ConnectTimeoutException: connection timed out: /127.0.0.   8816 (35.74%)
1:8080
> i.n.h.s.SslHandshakeTimeoutException: handshake timed out after    319 ( 1.29%)
r 10000ms

Steps to Reproduce

java -DPROTOCOL=H2 -jar frontend-rn-20x/build/libs/frontend-rn-20x-1.0.0.jar
  • on another console, run Gatling:
rm -rf test-reports/; java -DPROTOCOL=H2 -DURATION=10 -jar gatling/build/libs/gatling-1.0.0-all.jar application “test” TextPlain

After few minutes, some errors will be displayed like "handshake timed out".

While the test is running, jstat seems to indicate that the frontend is having some heap memory leak:

    MC         MU       CCSC      CCSU         OC           OU       YGC    FGC    FGCT     CGC    CGCT       GCT   
   24704.0    24333.0    2944.0    2781.2     155648.0     106984.5     10     0     0.000     6     0.007     0.028
   24896.0    24493.3    2944.0    2781.7     172032.0     107161.0     21     0     0.000     6     0.007     0.051
   24960.0    24632.7    2944.0    2783.1     507904.0     197459.5     34     0     0.000     6     0.007     0.160
   24960.0    24636.7    2944.0    2784.0    1638400.0     754936.5     40     0     0.000     6     0.007     0.363
   24960.0    24638.4    2944.0    2784.0    2744320.0    1550304.0     45     0     0.000     6     0.007     0.631
   24960.0    24638.4    2944.0    2784.0    2670592.0    1759994.5     50     0     0.000     6     0.007     0.774
   24960.0    24638.4    2944.0    2784.0    3190784.0    2045706.0     54     0     0.000     6     0.007     0.946
   24960.0    24638.7    2944.0    2784.0    3739648.0    1920992.0     58     0     0.000     8     0.008     1.247
   25088.0    24686.6    2944.0    2784.0    3833856.0    2117195.5     62     0     0.000     8     0.008     1.473
   25088.0    24742.3    2944.0    2784.5    4947968.0    3147744.0     67     0     0.000     9     0.010     2.010
   25152.0    24803.0    2944.0    2784.5    6152192.0    4818912.0     71     0     0.000     9     0.010     2.416
   25152.0    24805.2    2944.0    2784.5    7303168.0    6281184.0     81     0     0.000    10     0.010     2.999
   25152.0    24805.2    2944.0    2784.5    7565312.0    6135776.0     90     0     0.000    11     0.013     3.440
   25152.0    24805.2    2944.0    2784.5    7393280.0    5818336.0     99     0     0.000    12     0.013     4.085
   25152.0    24805.2    2944.0    2784.5    7049216.0    5103584.0    110     0     0.000    13     0.015     4.677
   25280.0    24872.6    2944.0    2784.5    7946240.0    6973408.0    116     0     0.000    13     0.015     5.307
   25280.0    24879.1    2944.0    2784.5    8318976.0    8315002.5    135     0     0.000    13     0.015     6.234
   25280.0    24778.9    2944.0    2755.4    5996544.0    3319548.6    139     1     0.450    13     0.015     6.900
   25280.0    24838.7    2944.0    2755.4    7487488.0    5935374.1    143     1     0.450    13     0.015     7.540
   25280.0    24838.7    2944.0    2755.4    8355840.0    8354581.7    196     2     0.450    13     0.015     8.299
   25280.0    24838.7    2944.0    2755.4    7073792.0    5211586.1    198     2     1.121    13     0.015     9.261
   25280.0    24838.7    2944.0    2755.4    7946240.0    7765442.1    208     2     1.121    13     0.015     9.967
   25280.0    24849.1    2944.0    2755.4    7348224.0    5662974.5    251     3     1.970    13     0.015    11.251
   25344.0    24916.1    2944.0    2755.4    8237056.0    8235258.5    268     3     1.970    13     0.015    11.987
   25344.0    24916.1    2944.0    2755.4    7946240.0    6956097.3    290     4     2.927    13     0.015    13.330
   25344.0    24920.2    2944.0    2755.4    8364032.0    8363073.3    350     5     2.927    13     0.015    13.902
   25344.0    24966.4    2944.0    2755.4    8359936.0    8358496.2    412     6     3.999    13     0.015    15.565
   25344.0    24967.5    2944.0    2755.4    8364032.0    8363298.6    456     7     5.088    13     0.015    17.063
   25344.0    24967.5    2944.0    2755.4    8310784.0    8309361.3    458     8     6.204    13     0.015    18.271
   25344.0    24969.1    2944.0    2755.4    8318976.0    8314907.8    462    10     8.350    13     0.015    20.592
   25344.0    25008.7    2944.0    2760.2    8318976.0    8315733.3    464    11     9.458    13     0.015    21.802

Using the following Netty5 debug options don't report anything:

-Dio.netty5.leakDetection.level=paranoid 
-Dio.netty5.leakDetection.targetRecords=32 
-Dio.netty5.buffer.lifecycleTracingEnabled=true 
-Dio.netty5.buffer.leakDetectionEnabled=true 

Possible Solution

Your Environment

Java 19
Benchmarks project
Reactor Netty 2
Netty 5

@pderop pderop added type/bug A general bug status/need-triage A new issue that still need to be evaluated as a whole labels Jun 9, 2023
@pderop pderop added this to the 2.0.x Backlog milestone Jun 9, 2023
@pderop pderop changed the title HTTP/2 Benchmarks on reactor netty 2 does not work HTTP/2 Benchmarks on reactor netty 2 don't work Jun 9, 2023
@pderop
Copy link
Contributor Author

pderop commented Jun 9, 2023

Here is a screenshot of memory analyzer on the frontend.
Screenshot 2023-06-09 at 15 44 40

@pderop
Copy link
Contributor Author

pderop commented Jun 9, 2023

some updates: the frontend test server was not setting any max concurrent streams, and specify a max-concurrent-stream on the frontend server avoids the memory leak.

Now, the benchmark still fails with this exception that happens on the frontend server (using the "TextPlain" scenario:

16:35:04.683 [reactor-http-nio-10] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty5.handler.codec.http2.Http2Exception: Stream 13973 does not exist
	at io.netty5.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:99)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.verifyStreamMayHaveExisted(DefaultHttp2ConnectionDecoder.java:667)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:427)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:513)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:256)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
	at io.netty5.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
	at io.netty5.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
	at io.netty5.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
	at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
	at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at reactor.netty5.http.server.HttpServerConfig$H2ChannelMetricsHandler.channelRead(HttpServerConfig.java:717)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at io.netty5.channel.internal.DelegatingChannelHandlerContext.fireChannelRead(DelegatingChannelHandlerContext.java:113)
	at io.netty5.handler.codec.ByteToMessageDecoder$ByteToMessageDecoderContext.fireChannelRead(ByteToMessageDecoder.java:446)
	at io.netty5.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
	at io.netty5.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1081)
	at io.netty5.handler.ssl.SslHandler.decode(SslHandler.java:1128)
	at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
	at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
	at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at io.netty5.channel.ChannelHandler.channelRead(ChannelHandler.java:235)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:838)
	at io.netty5.channel.AbstractChannel$ReadSink.processRead(AbstractChannel.java:1996)
	at io.netty5.channel.nio.AbstractNioByteChannel.doReadNow(AbstractNioByteChannel.java:74)
	at io.netty5.channel.AbstractChannel$ReadSink.readLoop(AbstractChannel.java:2061)
	at io.netty5.channel.AbstractChannel.readNow(AbstractChannel.java:910)
	at io.netty5.channel.nio.AbstractNioChannel.access$100(AbstractNioChannel.java:42)
	at io.netty5.channel.nio.AbstractNioChannel$1.handle(AbstractNioChannel.java:107)
	at io.netty5.channel.nio.NioHandler.processSelectedKey(NioHandler.java:506)
	at io.netty5.channel.nio.NioHandler.processSelectedKeysOptimized(NioHandler.java:489)
	at io.netty5.channel.nio.NioHandler.processSelectedKeys(NioHandler.java:430)
	at io.netty5.channel.nio.NioHandler.run(NioHandler.java:407)
	at io.netty5.channel.SingleThreadEventLoop.runIO(SingleThreadEventLoop.java:192)
	at io.netty5.channel.SingleThreadEventLoop.run(SingleThreadEventLoop.java:176)
	at io.netty5.util.concurrent.SingleThreadEventExecutor.lambda$doStartThread$4(SingleThreadEventExecutor.java:773)
	at io.netty5.util.internal.ThreadExecutorMap.lambda$apply$1(ThreadExecutorMap.java:68)
	at io.netty5.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1589)

@violetagg violetagg removed the status/need-triage A new issue that still need to be evaluated as a whole label Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug A general bug
Projects
None yet
Development

No branches or pull requests

2 participants