Timed out waiting for buffer space in TcpNioConnection under moderate load

[ouharri]

I am implementing a high‑throughput TCP gateway using Spring Boot 4, Spring Framework 7, and Spring Integration IP 7.0.0. The system is designed for a long‑term target of 10,000+ concurrent persistent TCP connections, each sending telemetry frames.

However, even under very moderate load (around 1,000–3,000 concurrent connections, each sending 1 message every ~10 seconds), the server begins dropping connections with repeated:

Timed out waiting for IO
Timed out waiting for buffer space

This happens even though:

• I am using TcpNioConnectionFactory
• Direct buffers are enabled
• Connection factory has large OS buffer sizes
• A custom CompositeExecutor (IO + assembler) is configured with large pools, no queues, abort/caller policies
• A very fast deserializer is in place with zero copying

I read through the NIO note in the Spring docs:
https://docs.spring.io/spring-integration/reference/ip/note-nio.html

…and also past issues, especially this one:
#2222

But the symptoms remain identical even with the recommended setup.

I would like to confirm whether this is:

1. a misconfiguration on my side, or
2. a limitation/bug in Spring Integration NIO, or
3. something requiring special tuning at OS/JDK level.

Any guidance would be hugely appreciated.

---

Load pattern

• ~1,000–3,000 concurrent persistent TCP connections in current tests
• Each connection sends about 1 small message every ~10 seconds

---

Logs

Below is an example of the recurring error :

ERROR [     nio-asm-19] TcpNioConnection : Read exception 0:0:0:0:0:0:0:1:61393:3333:da8e7d48-65d3-484a-9045-4f888f0f82a2 IOException:null:Timed out waiting for IO

ERROR [       nio-io-2]  TcpNioConnection : Exception on Read 0:0:0:0:0:0:0:1:61393:3333:da8e7d48-65d3-484a-9045-4f888f0f82a2 Timed out waiting for buffer space

java.io.IOException: Timed out waiting for buffer space
	at org.springframework.integration.ip.tcp.connection.TcpNioConnection$ChannelInputStream.write(TcpNioConnection.java:824) ~[spring-integration-ip-7.0.0.jar:7.0.0]
	at org.springframework.integration.ip.tcp.connection.TcpNioConnection.sendToPipe(TcpNioConnection.java:494) ~[spring-integration-ip-7.0.0.jar:7.0.0]
	at org.springframework.integration.ip.tcp.connection.TcpNioConnection.doRead(TcpNioConnection.java:477) ~[spring-integration-ip-7.0.0.jar:7.0.0]
	at org.springframework.integration.ip.tcp.connection.TcpNioConnection.readPacket(TcpNioConnection.java:534) ~[spring-integration-ip-7.0.0.jar:7.0.0]
	at org.springframework.integration.ip.tcp.connection.AbstractConnectionFactory.lambda$keyReadable$0(AbstractConnectionFactory.java:774) ~[spring-integration-ip-7.0.0.jar:7.0.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1474) ~[na:na]

This eventually leads to connection closures.

---

Environment

• Spring Boot: 4.0.0
• Spring Framework: 7.x
• Spring Integration IP: 7.0.0
• JDK: Java 25
• Platform: macOS (same behavior observed on Linux)

Using:

• Tcp.inboundAdapter
• TcpNioConnectionFactory
• CompositeExecutor (IO + assembler pools)
• VirtualThreadTaskExecutor for downstream message processing
• Direct buffers enabled

---

Connection Factory Configuration

(only relevant pieces shown)

factory.setUsingDirectBuffers(true);
factory.setSoReceiveBufferSize(256 * 1024);
factory.setSoSendBufferSize(256 * 1024);
factory.setSoTcpNoDelay(true);
factory.setSoKeepAlive(true);
factory.setSingleUse(false);
factory.setReadDelay(10_000); // tested with different values
factory.setBacklog(65_535);
factory.setTaskExecutor(tcpNioCompositeExecutor())
factory.setNioHarvestInterval(5_000);

---

Custom CompositeExecutor

public CompositeExecutor tcpNioCompositeExecutor() {
    int cores = Runtime.getRuntime().availableProcessors();

    int ioCore = Math.max(4, cores);
    int ioMax  = Math.max(cores * 4, 16);

    int asmCore = Math.max(cores * 4, 16);
    int asmMax  = Math.max(cores * 16, 64);

    ThreadPoolTaskExecutor io = new ThreadPoolTaskExecutor();
    io.setThreadNamePrefix("nio-io-");
    io.setCorePoolSize(ioCore);
    io.setMaxPoolSize(ioMax);
    io.setQueueCapacity(0);
    io.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
    io.initialize();

    ThreadPoolTaskExecutor asm = new ThreadPoolTaskExecutor();
    asm.setThreadNamePrefix("nio-asm-");
    asm.setCorePoolSize(asmCore);
    asm.setMaxPoolSize(asmMax);
    asm.setQueueCapacity(0);
    asm.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
    asm.initialize();

    return new CompositeExecutor(io, asm);
}

---

Please let me know if I am missing something in my configuration or understanding.
If not, a fix or clarification would be highly appreciated.

[artembilan]

The error:

java.io.IOException: Timed out waiting for buffer space
	at org.springframework.integration.ip.tcp.connection.TcpNioConnection$ChannelInputStream.write(TcpNioConnection.java:824) ~[spring-integration-ip-7.0.0.jar:7.0.0]

comes from this logic:

if (!this.buffers.offer(buffer, TcpNioConnection.this.pipeTimeout, TimeUnit.MILLISECONDS)) {
	throw new IOException("Timed out waiting for buffer space");
}

in the:

	/**
	 * Provides an InputStream to receive data from {@link SocketChannel#read(ByteBuffer)}
	 * operations. Each new buffer is added to a BlockingQueue; when the reading thread
	 * exhausts the current buffer, it retrieves the next from the queue.
	 * Writes block for up to the pipeTimeout if 5 buffers are queued to be read.
	 *
	 */
	class ChannelInputStream extends InputStream {

So, there are more writes than readers from that connection.
It looks like we read faster from the socket than data is processed on that ChannelInputStream.
Perhaps something in downstream logic cannot cope with such a fast pace of inputs.

That TcpNioConnection.this.pipeTimeout is one minute by default, so doesn't look like the problem is on TCP level of your application.

Some sample project to play with on our side would be great help.

Thanks

[artembilan]

Another problem might come from this bug: #10666.

Could you, please, confirm your Spring Integration version?

[ouharri]

Hello @artembilan ,

After further investigation, I spent time stepping through the TcpNioConnection internals under high‑concurrency conditions and analyzing the full NIO → pipe → assembler → deserializer flow. I was able to identify the exact root cause of the intermittent timeouts and why they appear only under load.

Below is a clear summary of the findings.

Root cause: a lost‑notify race on writingLatch

Inside TcpNioConnection, message assembly involves three main actors:

1. NIO reader (doRead) – reads from the SocketChannel and sends bytes into the pipe.
2. ChannelInputStream pipe – a 5‑slot queue bridging NIO reads and the deserializer.
3. Assembler thread (run() → convert()) – waits for data and invokes the deserializer.

The reader sets writingLatch to indicate “a read cycle is in progress”. The assembler uses this latch when no data is currently available to wait up to 60 seconds for fresh bytes.

The bug

Inside ChannelInputStream.write(), the implementation currently performs:

TcpNioConnection.this.writingLatch = new CountDownLatch(1);

This line replaces the latch with a new one after every write, even when the assembler is already waiting on the previous latch.

This leads to a lost‑notification race:

1. Assembler enters convert(), captures latch A, and prepares to wait.
2. Before the assembler actually calls await(), ChannelInputStream.write() executes and replaces latch A with B.
3. NIO reader signals B, but assembler is still waiting on A.
4. A is never signaled → assembler waits the full 60s → Timed out waiting for IO.
5. While assembler is blocked, the pipe cannot be drained → queue fills → Timed out waiting for buffer space.

Both exceptions originate from this single race condition.

This behavior traces back to:

• Issue NIO Assembler Thread is Sometimes Not Released to Thread Pool Between Messages [INT-3453] #7418
• PR INT-3453 Fix NIO Connection Race Condition #1185

which introduced the latch reset logic. Unfortunately, that change also introduced the race.

---

Fix (validated extensively)

The fix is straightforward:

• Remove the latch reset inside ChannelInputStream.write()

- TcpNioConnection.this.writingLatch = new CountDownLatch(1);

• Clear the latch only at the end of the read cycle in doRead():

finally {
    this.writingToPipe = false;
    writingLatchToUse.countDown();
    this.writingLatch = null;
}

This restores the intended lifecycle:

• One latch per read cycle.
• The assembler waits only on the latch tied to the active read.
• No stale latch remains in place.

Result

After applying this fix, the behavior becomes stable:

• No more 60‑second IO timeouts.
• No pipe‑queue saturation.
• Backpressure remains correct.
• TLS and non‑TLS operate identically.
• No regressions observed.

Tested under:

• High‑concurrency scenarios with persistent TCP connections (20k+)
• 1 msg/10sec traffic rates
• Both direct and heap ByteBuffer
• Custom deserialization logic

Everything behaved consistently.

---

If this analysis aligns with the expected behavior, I can open a PR containing:

• the minimal fix,
• a regression test,
• and a short explanation of the latch lifecycle.

Happy to contribute.

[artembilan]

Your analysis sounds adamant .
Please, be my guest and open PR. We will see what to do next from there.
Thank you!

[ouharri]

Your analysis sounds adamant . Please, be my guest and open PR. We will see what to do next from there. Thank you!

Sure, I’ll open the PR shortly.