bug: uncaught exception from RPC invocation during shutdown

[dergoegge]

The new thread pool introduced in #33689 will throw an exception (which will not be caught and lead to an unclean exit) if an RPC is called during shutdown.

2026-02-11T18:56:34Z [http:debug] Received a POST request for / from 10.89.0.6:58998
2026-02-11T18:56:34Z [http:debug] Interrupting HTTP server
2026-02-11T18:56:34Z [rpc:debug] Interrupting HTTP RPC server
2026-02-11T18:56:34Z [all:info] http_pool_1 thread exit
2026-02-11T18:56:34Z [all:info] http_pool_0 thread exit
2026-02-11T18:56:34Z [rpc:debug] Interrupting RPC
terminate called after throwing an instance of 'std::runtime_error'
  what():  No active workers; cannot accept new tasks

This was found with a test running on Antithesis.

[fanquake]

cc @furszy

[pinheadmz]

I'm not familiar with Antihesis, are you able to reliably reproduce the crash? And if so, can you test this branch which is #32061 rebased on threadpool, just not cleaned up yet.

[dergoegge]

There is no reproducer testcase that I can quickly run to verify, it's more like the test running on antithesis itself is the reproducer (as it will very likely be able to re-find the issue). You can read more about their methodology here https://antithesis.com/resources/deterministic_simulation_testing/.

It would probably be fairly easy to write a functional test to reproduce this (perhaps combined with a strategically places sleep on the c++ side).

I will run the Antithesis test on your PR changes, but I'm not sure if it would be the right fix unless that PR is ready to be merged soon for v31.

[pinheadmz]

I just wonder if its the same as #31929 which is addressed in #32061

[dergoegge]

This only popped up now with the new thread pool, so I don't think so but I could be wrong of course.

[furszy]

Cool tool @dergoegge. This just uncovers an existing issue.

Before, this would have returned an incorrect error: "Request rejected because http work queue depth exceeded" during shutdown, because the previous Enqueue() method was returning false both when the queue exceeded the max size and when it was interrupted. Now we have split these two cases to handle errors better, but miss to handle the latter one properly.

We can either catch the exception or change submission to retrieve std::expected/util::Result with the error.

Reproducing it should also be straightforward:

1. Receive an http request.
2. sleep() during request processing, before pool submission.
3. Trigger shutdown, sleep() after unregistering the libevent callback and interrupting the pool.
4. Let step (2) resume. This attempts to submit a task to an interrupted pool.

Will fix it. Thanks.

[furszy]

#34577