Lost-wakeup race in WorkScheduler.WaitForTask hangs requests forever (uncancellable, leaks a concurrency slot) #10051

Product

Hot Chocolate

Version

16.3.0

Link to minimal reproduction

https://github.com/mta-incom/hotchocolate_16x_lostwakeup_bug_reproduction

Steps to reproduce

The reproduction is a ~70-line console app that references only HotChocolate. It
builds an in-process request executor for a two-type schema (a mutation returning
an object with one async child field — no database, no HTTP, no DataLoader, no
custom middleware), then repeatedly fires a batch of identical mutations and uses a
wall-clock timeout as a hang detector.

1. Clone the reproduction and run it:

   dotnet run -c Release
   

2. The loop prints round N ok as each batch completes. The hang is a race, so to
   make it reproduce reliably within seconds, run with a small thread pool and
   higher concurrency (these only speed up reproduction; they are not required for
   the bug):

   MIN_THREADS=3 MAX_THREADS=6 CONCURRENCY=300 TIMEOUT_SECONDS=25 dotnet run -c Release
   

3. Within a few seconds/rounds the program prints HANG DETECTED round=… hung=…/…
   and stops making progress: one or more executor.ExecuteAsync(mutation) calls
   never complete. The program then parks (Thread.Sleep(Timeout.Infinite)) so a
   full dump can be captured with dotnet-dump collect -p <pid> --type Full.

The only two ingredients required are: (a) a mutation (so the execution loop
waits via WorkScheduler.WaitForTask), and (b) at least one resolver that completes
on a different thread (here the child field does await Task.Yield(); in a real
application this is any async, e.g. database-backed, resolver). No [BatchResolver],
batching, paging, or DataLoader is involved.

What is expected?

Running the reproduction project, every execution of the mutation should completed; the driver should print round N ok
indefinitely and never print HANG DETECTED.

What is actually happening?

Intermittently, one or more concurrent mutations never complete. The request hangs
forever and is uncancellable — neither ExecutionTimeout nor a cancelled
token recovers it, because the wait that gets stuck registers no cancellation.

A full-process dump of a hung run shows a lost wakeup in WorkScheduler (not a
blocked thread — the thread pool is idle; the execution loop is a parked async
continuation):

• The serial-task waiter is suspended at
  WorkScheduler.WaitForTask(taskId) → TryPauseAsync() → await _signal
  (a live AsyncStateMachineBox<WorkScheduler+<WaitForTask>d__23> /
  <TryPauseAsync>d__27 on the heap).
• Its frozen state: the awaited taskId is already present in _completed,
  _signal is in the reset state (_continue == false) with a stranded, non-null
  continuation, _serial and _work have nothing running, and one child task is
  left undrained on the work stack.

Mechanism (matches the source in WorkScheduler.cs / WorkScheduler.Execute.cs):

• WaitForTask checks _completed.Contains(taskId) under lock (not yet), then calls
  TryDispatchOrComplete(isWaitingForTaskCompletion: true) which does
  _signal.Reset(), then awaits the signal in TryPauseAsync.
• Complete(task) adds the id to _completed inside the _sync lock but calls
  _signal.Set() after releasing the lock.
• If that Set() lands in the window between the waiter's _completed check and
  its _signal.Reset(), the Reset() erases the wakeup. The waiter then parks in
  TryPauseAsync and never loops back to re-check _completed — even though the task
  it is waiting for is already complete. Lost wakeup → the request hangs forever.

Consequence: the hang does not block or spin a thread (the await releases it), but
each occurrence permanently leaks the request itself, a request-concurrency permit
(the stuck request never releases its slot under v16's concurrency gate), and the
memory pinned by that request. Once enough permits leak, the server stops accepting
new requests while still appearing idle. Only a process restart clears it.

Relevant log output

#output from run of reproduction code
minThreads=4 maxThreads=8 concurrency=200 timeout=30s pid=36244
HANG DETECTED round=25 hung=1/200 pid=36244

Additional context

Not fixed in any released version or on main. The three files that contain the
race are byte-identical across 16.0.13, 16.3.0, the latest 16.4.0 prerelease, and
current main:

WorkScheduler.cs            md5 95e35bc05f85ca1b7304edc18df4c1e7
WorkScheduler.Execute.cs    md5 b566aa59fdec0c30010f07d7f8a4ac63
AsyncManualResetEvent.cs    md5 3be7b8b182ccd96e48989b4e57311193

Not GC-mode specific. Reproduces with both Server GC (default in the repro) and
Workstation GC (DOTNET_gcServer=0).

Amplification, not cause. The small thread pool + high concurrency only widen the
race window so it reproduces quickly; the defect is intra-request and not caused by
load. It has also been observed in a real application on a plain paged query whose
resolver merely awaited a database call — no Task.Yield(), no DataLoader anywhere.

Relation to #9892. This is the same symptom class (an uncancellable HotChocolate
16 execution hang) but a distinct mechanism, confirmed by comparing dumps:

• [BatchResolver] hangs when parent resolver returns Task<T> with multiple parent items #9892 is a synchronous re-entrant lock deadlock — a thread blocked at
  System.Threading.Lock.EnterAndGetCurrentThreadId() inside
  WorkScheduler.ExecuteInternalAsync, on the [BatchResolver] branch completion
  path (requires [BatchResolver] + an async parent + multiple parent items;
  deterministic, single request, no concurrency).
• This report is an async lost-wakeup on _signal in WaitForTask — no thread
  blocked, no [BatchResolver], probabilistic under concurrency.
• A fix for one would not fix the other (different primitive: System.Threading.Lock
_sync vs AsyncManualResetEvent _signal).

Suggested fix. Either call _signal.Set() while still holding the _sync lock in
Complete(), or re-check _completed after the _signal.Reset() in WaitForTask
(before awaiting), so a Set() that raced the Reset() is still observed:

TryDispatchOrComplete(isWaitingForTaskCompletion: true);
lock (_sync) { if (_completed.Contains(taskId)) return; } // re-check after Reset closes the race
await TryPauseAsync().ConfigureAwait(false);