-
Notifications
You must be signed in to change notification settings - Fork 647
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possible memory leak in HttpServerHandler and FluxReceive #406
Comments
@kadledav Can you provide some description of the scenario that you think causes the leak? Do you consume the whole response body, or you receive some error? Do you abandon the response body and consume only the status code/headers? For the first issue I agree that it is a bug and will be fixed. |
@kadledav Can you provide more information about the leak scenario? |
The leak scenario is quite simple. I have exposed our application to a HTTP stress test and waited until ResourceLeakDetector reported a memory leak. But I am not able to reproduce it with a simple spring-boot application. Our application has more complicated Netty pipeline because we are also using a legacy protocol on the same port as HTTP for legacy clients. I think I have found out where the problem could be. This memory leak issue has started since we have updated spring-boot from 2.0.1 to 2.0.2 (regardless I force the reactor-netty version). I think it could be related to some incompatibilities in our adjusted libraries. The problem still persist with spring-boot 2.0.4. Even though the problem is not directly caused by reactor-netty project there was an unreleased direct memory object which caused memory leak which. I think the code working with direct memory should be robust and always clean it's memory. I have encountered a similar problem before, when we have a wrong implementation of Again I am not able to reproduce it with a simple spring-boot application. So I will continue searching what is causing the the memory leak in our application but not in a simple example. |
@kadledav Can you try reproducing the issue with |
I have reproduced a memory leak and attached a log file. In the log file are all requests from server start until a memory leak occurred . memoryLeak.log Scenario: |
@kadledav What did you say the Reactor Netty version is? |
Hello, I have finally find out what is causing the memory leak. We are using our Netty ServerBootstrap and based on the communication we are modifying the pipeline to support desired protocol. Unfortunately we cannot start always with HTTP and do upgrade protocol with 101 status code due to our legacy clients, which doesn't support HTTP communication. Our problem: We need to modify our pipeline to HTTP if it's HTTP communication. So we created a solution which has some issues, because we are not able to easily build a HTTP pipeline with reactor-netty API. Basically we have looked into a reactor-netty code and used There is a class example providing
I know that our approach is ugly at least because those reasons:
Is there a more elegant way to retrieve HTTP pipeline without building and binding reactor-netty |
Hm you are able to manipulate the pipeline with the following two possibilities depending on when you want to attach the handlers: |
Yes, this helps a lot. Thank you. |
I was searching for memory leak by Netty in our application (org.springframework.boot:2.0.3.RELEASE , reactor-netty:0.7.8.RELEASE)
First:
I have discovered a suspicious code in the reactor.ipc.netty.http.server.HttpServerHandler#channelRead
in the case that a following expression is true:
msg instanceof HttpRequest AND decoderResult.isFailure()
there is no
ReferenceCountUtil.release(msg)
and msg is also not send for other processing. So I think there is a possible memory leak.Second:
Unfortunately that wasn't my case. I have found out that problem is in the reactor.ipc.netty.channel.FluxReceive . It seems that sometimes there were some messages in the
receiverQueue and release() on those messages was never called. So I have made an ugly patch (added finally block) in reactor.ipc.netty.channel.ChannelOperations#applyHandler
The code above helped me and Netty doesn't report memory leak anymore.
Unfortunately I am not able to write simple test to reproduce it.
Reactor Netty version
0.7.8.RELEASE
JVM version
jdk-10.0.1
OS version
Windows 10
The text was updated successfully, but these errors were encountered: