Thread leaks (at least in tests)

[findepi]

./mvnw -pl :trino-main test

To me this looks like an application leaking memory with leaking threads being likely the reason

eventually tests slow down progress probably due to retries

2024-09-10T09:53:19.188-0600	WARN	http-client-memoryManager-110913	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56590/v1/memory: Server refused connection: http://127.0.0.1:56590/v1/memory
2024-09-10T09:53:20.191-0600	WARN	http-client-memoryManager-110906	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56590/v1/memory: Server refused connection: http://127.0.0.1:56590/v1/memory
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.24 s -- in io.trino.sql.query.TestDistinctAggregationsNoMarkDistinct
2024-09-10T09:53:21.110-0600	WARN	http-client-memoryManager-107141	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:22.127-0600	WARN	http-client-memoryManager-107139	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:23.128-0600	WARN	http-client-memoryManager-107141	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
2024-09-10T09:53:23.434-0600	WARN	http-client-node-manager-106280	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56422/v1/info/state: Server refused connection: http://127.0.0.1:56422/v1/info/state
2024-09-10T09:53:23.662-0600	WARN	http-client-node-manager-107123	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56450/v1/info/state: Server refused connection: http://127.0.0.1:56450/v1/info/state
2024-09-10T09:53:24.032-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:24.136-0600	WARN	http-client-memoryManager-107139	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56450/v1/memory: Server refused connection: http://127.0.0.1:56450/v1/memory
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.31 s -- in io.trino.operator.scalar.TestJoniRegexpFunctions
2024-09-10T09:53:25.034-0600	WARN	http-client-memoryManager-106301	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:26.038-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:27.045-0600	WARN	http-client-memoryManager-106301	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
2024-09-10T09:53:28.049-0600	WARN	http-client-memoryManager-106311	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56422/v1/memory: Server refused connection: http://127.0.0.1:56422/v1/memory
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.73 s -- in io.trino.operator.scalar.TestVarbinaryFunctions
2024-09-10T09:53:35.822-0600	WARN	http-client-memoryManager-100157	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
2024-09-10T09:53:36.826-0600	WARN	http-client-memoryManager-100156	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
2024-09-10T09:53:37.155-0600	WARN	http-client-node-manager-100127	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56218/v1/info/state: Server refused connection: http://127.0.0.1:56218/v1/info/state
2024-09-10T09:53:37.834-0600	WARN	http-client-memoryManager-100157	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56218/v1/memory: Server refused connection: http://127.0.0.1:56218/v1/memory
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 126.6 s -- in io.trino.operator.scalar.TestStringFunctions

2024-09-10T09:53:59.571-0600	WARN	http-client-node-manager-111469	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56610/v1/info/state: Server refused connection: http://127.0.0.1:56610/v1/info/state
2024-09-10T09:53:59.939-0600	WARN	http-client-memoryManager-111482	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
2024-09-10T09:54:00.945-0600	WARN	http-client-memoryManager-111489	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
2024-09-10T09:54:01.952-0600	WARN	http-client-memoryManager-111482	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56610/v1/memory: Server refused connection: http://127.0.0.1:56610/v1/memory
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.28 s -- in io.trino.operator.scalar.timestamptz.TestOperators
^[[F2024-09-10T09:54:06.603-0600	WARN	http-client-memoryManager-104553	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:07.510-0600	WARN	http-client-node-manager-104528	io.trino.metadata.RemoteNodeState	Error fetching node state from http://127.0.0.1:56377/v1/info/state: Server refused connection: http://127.0.0.1:56377/v1/info/state
2024-09-10T09:54:07.611-0600	WARN	http-client-memoryManager-104564	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:08.613-0600	WARN	http-client-memoryManager-104553	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
2024-09-10T09:54:09.616-0600	WARN	http-client-memoryManager-104564	io.trino.memory.RemoteNodeMemory	Error fetching memory info from http://127.0.0.1:56377/v1/memory: Server refused connection: http://127.0.0.1:56377/v1/memory
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 123.3 s -- in io.trino.operator.scalar.timestamptz.TestExtract

but continue nonetheless (awesome resilience!), although very slowly

[martint]

Can you try disabling the thread-per-driver scheduler for the tests and rerun the analysis?

[findepi]

I tried that, and I did get similar results.

[wendigo]

➜  trino git:(master) ✗ grep http-client heap.dump | wc -l
     558
➜  trino git:(master) ✗ grep query-execution heap.dump | wc -l
    2661
➜  trino git:(master) ✗ grep query-execution heap.dump | wc -l
    2661

Most of the threads are in the query-execution- group

[findepi]

that's what i was seeing as well, yes