[🐛 Bug]: Memory leak running hub, due to HttpClient-x-SelectorManager threads never closed

[bhecquet]

What happened?

Hello

On our hub (version 4.15.0, JDK 11.0.20), running 13 nodes, memory increase continuously as tests run. Attaching a VisualVM shows that it seems to be caused by an increasing number of HttpClient-x-SelectorManager threads that are not closed (seems similar to #12206 or #11336)

Operations are normal, except when max memory is reached. In this case, hub does not respond anymore

In the graphic above, you can see that 300 more threads have been created in an hour

Is there a way (to help) to track down the creator of the staled threads ? Thread dump doesn't help much

"HttpClient-111-SelectorManager" - Thread t@756
   java.lang.Thread.State: RUNNABLE
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:357)
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:182)
        at java.base@11.0.20.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        - locked <2eebf6a6> (a sun.nio.ch.Util$2)
        - locked <4a436b2f> (a sun.nio.ch.WindowsSelectorImpl)
        at java.base@11.0.20.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at platform/java.net.http@11.0.20.1/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:867)

   Locked ownable synchronizers:
        - None

Thanks

How can we reproduce the issue?

Start a hub
Start some nodes
Execute many tests. I think the more tests are run, the easier it's to point the issue

Relevant log output

no relevant logs available on hub

Operating System

Windows / Linux

Selenium version

4.11.0

What are the browser(s) and version(s) where you see this issue?

any browser

What are the browser driver(s) and version(s) where you see this issue?

any browser

Are you using Selenium Grid?

4.15.0

[github-actions]

@bhecquet, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[diemol]

@pujagani didn't you troubleshoot this already in another issue?

[bhecquet]

I did a memory dump. Put in Eclipse Memory Analyzer:

[joerg1985]

I just have pushed a commit cdbe9c6 that might point into the direction why there are idling websocket. There was a misunderstanding in what cases the onError is called in the client, that the websocket could deliver any further messages. But when it is called the websocket is broken beyond repair and will not deliver any further messages.

[bhecquet]

We've analyzed the code in JdkHttpClient.java and the memory dump and we thought the problem might also be in the close method, where "websockets" list is not cleared, which could lead to references being kept
What do you think about that ?

[pujagani]

@pujagani didn't you troubleshoot this already in another issue?

The other issue was also related to a leak but that was pertaining to a leak happening in only a particular Java version. I have handled a similar one via #11345 earlier. But I think @joerg1985 has an idea for this one. If not, I am happy to take a look.

[joerg1985]

@bhecquet the reference to the HttpClient should be dropped after some time, so this list should not be an issue.
Could you retest with selenium 4.16.0?

[bhecquet]

Hello, I just tested the 4.16.0 version, and I observe the same behaviour as with 4.15.0

[joerg1985]

Okay one thing that should be checked, we have chain like this:

Client <---- WS ----> Grid <---- WS ----> Node <----- WS ------> Driver

Now the Driver does raise an error on the Websocket

Client <---- WS ----> Grid <---- WS ----> Node * disconnected * Driver

I don't think the other connections are also disconnected in this case.
To have an error from the driver site is alarming easy, this code will provoke it on my system:

var options = new ChromeOptions();        
options.setCapability("webSocketUrl", true); // <- bidi must be enabled to run into errors of the devtools connection
var driver = new ChromeDriver(options);

driver.close(); // <- runs into onError of the devtools connection, not the bidi connection
driver.quit();

@pujagani would you like to check this or should i give it a try at a undefined point in the future?
I think the fix for this would be to send a close message to the maybeHandler inside the WebSocketUpgradeHandler.exceptionCaught call, but this is more a educated guess.

[pujagani]

I can check this in a day or two. I am happy to help, though you are the expert in this area. I might ask questions as and when I need the guidance.

[pujagani]

I was able to reproduce the issue with Java 11 at a small scale but what I see is leftover SelectorManager threads. I am looking into triaging it further. The same does not occur with Java 21 since HttpClient implements AutoCloseable.
I am unable to see WebSocketImpl instances while profiling through. Are you running tests that use CDP heavily? @bhecquet