Summary
The network layer never enables TCP keepalive, and the Postgres/Redis executors deliberately set an infinite socket read timeout after authentication. A half-open connection (peer host crashed or partitioned with no FIN/RST) therefore pins a server thread and its file descriptor forever — there is no application heartbeat on those protocols either.
Where (HEAD af4f67fd2)
network/src/main/java/com/arcadedb/network/binary/Channel.java:42-47 — the constructor sets only TCP_NODELAY:
public Channel(final Socket iSocket) throws IOException {
socket = iSocket;
socket.setTcpNoDelay(true);
// THIS TIMEOUT IS CORRECT BUT CREATE SOME PROBLEM ON REMOTE, NEED CHECK BEFORE BE ENABLED
// timeout = iConfig.getValueAsLong(OGlobalConfiguration.NETWORK_REQUEST_TIMEOUT);
}grep for setKeepAlive / SO_KEEPALIVE across the whole network/ module returns none.
Live callers via ChannelBinaryServer: PostgresNetworkExecutor sets socket.setSoTimeout(0) after auth (markAuthenticated, :200) and RedisNetworkExecutor does the same (:806) — correctly, because authenticated clients hold long-lived idle connections.
Repro
A Postgres or Redis client authenticates, then its host crashes / the link is silently dropped. The server thread is blocked in in.readXXX() with SO_TIMEOUT == 0 and SO_KEEPALIVE off, so the OS never probes the dead peer and the read never returns. Thread + FD leak permanently; enough such events exhaust the pool / FD table.
Suggested fix
Enable socket.setKeepAlive(true) in the Channel constructor (the network-layer owner of socket options), ideally with a configurable probe interval on modern JDKs, so the OS can tear down dead peers even on infinite-SO_TIMEOUT idle connections.
Scope / confidence
Medium-high — the mechanism is certain (both the missing option and the deliberate infinite timeout verified at HEAD). Whether to treat it as a bug or an accepted tradeoff is a judgment call, but for wire protocols with no application-level heartbeat there is no other liveness backstop.
Related, same file (smaller)
ChannelBinary.close() (:278-297) closes in before out; closing a socket's input stream closes the socket, so the subsequent out.close() flush fails and any buffered-but-unflushed bytes are silently dropped (logged at FINE). Low impact today because the live executors flush per message. Fix: flush/close out first.ChannelBinaryClient (:31-69) leaks the socket FD on every failed connect: the cleanup catch handles only RuntimeException, while every connect failure path throws a checked exception (IOException/SocketException), and the if (socket.isConnected()) guard is wrong for cleanup (a failed connect leaves it unconnected but still holding an FD). Latent — this class has no production instantiation at HEAD (HA moved to Apache Ratis), so it is a legacy-code defect with no live impact today. Fix: catch (Throwable) and close unconditionally.
Summary
The network layer never enables TCP keepalive, and the Postgres/Redis executors deliberately set an infinite socket read timeout after authentication. A half-open connection (peer host crashed or partitioned with no FIN/RST) therefore pins a server thread and its file descriptor forever — there is no application heartbeat on those protocols either.
Where (HEAD
af4f67fd2)network/src/main/java/com/arcadedb/network/binary/Channel.java:42-47— the constructor sets onlyTCP_NODELAY:grepforsetKeepAlive/SO_KEEPALIVEacross the wholenetwork/module returns none.Live callers via
ChannelBinaryServer:PostgresNetworkExecutorsetssocket.setSoTimeout(0)after auth (markAuthenticated,:200) andRedisNetworkExecutordoes the same (:806) — correctly, because authenticated clients hold long-lived idle connections.Repro
A Postgres or Redis client authenticates, then its host crashes / the link is silently dropped. The server thread is blocked in
in.readXXX()withSO_TIMEOUT == 0andSO_KEEPALIVEoff, so the OS never probes the dead peer and the read never returns. Thread + FD leak permanently; enough such events exhaust the pool / FD table.Suggested fix
Enable
socket.setKeepAlive(true)in theChannelconstructor (the network-layer owner of socket options), ideally with a configurable probe interval on modern JDKs, so the OS can tear down dead peers even on infinite-SO_TIMEOUTidle connections.Scope / confidence
Medium-high — the mechanism is certain (both the missing option and the deliberate infinite timeout verified at HEAD). Whether to treat it as a bug or an accepted tradeoff is a judgment call, but for wire protocols with no application-level heartbeat there is no other liveness backstop.
Related, same file (smaller)
ChannelBinary.close()(:278-297) closesinbeforeout; closing a socket's input stream closes the socket, so the subsequentout.close()flush fails and any buffered-but-unflushed bytes are silently dropped (logged at FINE). Low impact today because the live executors flush per message. Fix: flush/closeoutfirst.ChannelBinaryClient(:31-69) leaks the socket FD on every failed connect: the cleanupcatchhandles onlyRuntimeException, while every connect failure path throws a checked exception (IOException/SocketException), and theif (socket.isConnected())guard is wrong for cleanup (a failed connect leaves it unconnected but still holding an FD). Latent — this class has no production instantiation at HEAD (HA moved to Apache Ratis), so it is a legacy-code defect with no live impact today. Fix:catch (Throwable)and close unconditionally.