Client thread parked forever with queued ThreadlessExecutor tasks

[BhautikKoladiya]

What version of gRPC-Java are you using?

1.76.0

What is your environment?

• OS: Linux (aarch64/ARM64)
• JDK: openjdk version 21.0.4

What did you expect to see?

Client thread should not block indefinitely when NettyStream receives both MessageReceive and StreamClose events. blockingUnaryCall should return once the stream is closed and pending callbacks are processed

What did you see instead?

blockingUnaryCall hangs indefinitely. The client thread remains parked in LockSupport.park() inside ThreadlessExecutor.waitAndDrain() and never resumes execution. This occurs despite pending tasks being present in the ThreadlessExecutor queue, including a SerializingExecutor runnable, and additional queued work in SerializingExecutor.runQueue such as MessagesAvailable and StreamClosed. As a result, these callbacks are never processed and the call remains blocked forever.

Heap dump analysis:

Object	State
Client Thread	LockSupport.park() in waitAndDrain()
ThreadlessExecutor.waiter	The parked thread (correctly set)
ThreadlessExecutor queue	[null] (head) → [SerializingExecutor] (item present but not polled)
SerializingExecutor.runState	-1 (RUNNING)
SerializingExecutor.runQueue	[null] (head) → [MessagesAvailable] → [StreamClosed]

The SerializingExecutor was present in the ThreadlessExecutor queue waiting to be polled, but the parked thread never retrieved it despite unpark() having been called. This caused the StreamClosed callback to never execute, leaving the blocking call stuck forever.

Steps to reproduce the bug

Not able to reproduce this issue on demand. It appears to be a rare race condition that has only been observed a few times in production. Also after first occurrence, we upgraded gRPC from 1.64.2 to 1.76.0, but it was still observed afterward.

[panchenko]

Looks similar to #12620

[BhautikKoladiya]

@panchenko Thanks for the reply. This issue seems to be slightly different — we haven’t configured retries on the client call.

Issue on the ThreadlessExecutor side. Based on the implementation, if there are any pending tasks in the queue then waiter thread should not remain in a parked state. However, Here ThreadlessExecutor queue contains a task, and SerializingExecutor.runQueue also has pending tasks (MessagesAvailable and StreamClosed). Ideally, this should result in the waiter thread being unparked and processing those tasks, but that doesn’t appear to be happening here.