Unnecessarily failing IO when the submitting thread exits before the IO completes

[dcoutts]

In well-typed/blockio-uring#58 we appear to observe the following behaviour:

1. thread submits IO to the ring
2. the thread exits
3. the IO completes
4. the CQE is delivered with error -EFAULT

https://github.com/torvalds/linux/blob/master/io_uring/io_uring.c#L1058

That is, if the thread that submitted the IO exits before the IO completes, then the IO is reported as failing.

This would appear to be an unnecessary failure. The IO really did complete successfully, and the ring is still there and other threads are able to pick up the CQE. Why do we need to report the IO as failing? We're not explicitly cancelling the IO, indeed we do not want to cancel the IO.

This behaviour interacts poorly with application designs that use a pool of worker threads. In this application design only one thread animates the application at once (so there's no danger of concurrent use of the SQ or CQ), but the thread that animates the application can change relatively frequently (for reasons probably not worth elaborating here), and idle worker threads can get tidied up from time to time.

[axboe]

The quoted code is trying to submit the IO again, not complete it. That's why it's being flagged with -EFAULT.

[dcoutts]

Thanks. Ok, oops, I got the wrong end of the stick with the code.

So it sounds like it'd be useful for me to make a concrete test case and attach it here.

And I take it (from the hand-wavy description) you're not saying it's obviously correct/expected behaviour? I'm very much hoping I don't have to totally re-architect this application (which is not actually an application but is a rather mature and high performance language runtime system).

[axboe]

IMHO the design doesn't sound great, if you rely on threads that spawn just to submit IO. That's not very efficient. Are they also creating their own ring for this and then exiting it? Or are you sharing a ring for submits, and each created thread will get exclusive access? Either one of those sounds highly suboptimal to me.

[dcoutts]

The Haskell RTS design looks like this:

• proper N:M threading model, with userspace green threads and green thread pre-emption
• one Haskell execution context (HEC) per core, with all the data structures for things like the per-HEC scheduler etc
• only one OS thread runs a HEC at once (holding a lock)
• when a green thread needs to execute a foreign function (e.g. C lib) the OS thread releases the HEC lock and executes the foreign call, but another OS thread from a thread pool picks up the HEC to continue execution of the other green threads
• a dynamically managed thread pool, so idle threads get killed off eventually
• per-HEC I/O manager that manages I/O multiplexing to allow green threads to do I/O, without having to use an OS thread per I/O (as would be the case if we relied on the generic FFI support)

I'm extending this with support for io_uring. This should be a perfect fit because a M:N green thread model fits nicely with 1:1 OS threads plus async I/O. It allows the green thread to act like they're performing synchronous I/O, but only the individual green thread is blocked because the I/O is async and handled by the RTS I/O manager.

The design of the RTS I/O manager for io_uring uses an io_uring per HEC, so effectively one ring per core. The thread that is running the HEC (at any given moment) will submit and collect I/O on behalf of all the green threads on the HEC. This avoids any thread context switching and lets us take advantage of io_uring's ability to do synchronous I/O synchronously with minimal overhead. So normally this is good and matches the io_uring sweet spot quite well.

However as noted, there is a thread pool to manage FFI, and the design of that is such that the thread that was running the HEC goes off to run the foreign call, while a thread from the thread pool continues running the HEC. This means that which thread is running the HEC will change from time to time. And this is where we run into the problems with io_uring because we submit I/O from whichever thread is running the HEC at that moment, but the lifetime of that thread is not related to the lifetime of the I/O that it submitted. From a user-space point of view this could work fine: there will always be a thread waiting to collect the CQEs, and they're still expected, since all the user space I/O tracking information lives with the HEC.

This aspect of how foreign calls is done may at first seem a little mad (e.g. as opposed to using the pool only to execute foreign calls while a single dedicated thread runs the HEC at all times), but other related features make it make more sense. One example is that foreign code (again think C libs) can call into Haskell, and this is done using the thread controlled by the foreign code: the thread just has to wait to take the HEC lock and then it can execute the Haskell code directly. And of course while the "foreign" thread runs the HEC then any green threads can submit more I/O, and that's on a thread that we do not control at all, at least not its lifetime once it returns to the foreign caller. And there's a similar feature where to use C libs that use thread-local state, we arrange that a specific green thread always be run only by a specific OS thread (which it can do because it just needs to be the one holding the HEC lock and running the HEC). So that's why it ended up with this design with many thread but only one thread at a time.

So I would argue that we are (or are trying to) use io_uring in a reasonably performant way (as much as we can), but the mismatch between the lifetime of I/O operations and the lifetime of the OS threads that submit them is a major problem for us. This HEC design with a thread pool has been like this for 20 years or so, so it's very hard to change a mature system that works. To use io_uring with our work-pool design we'd have to do something much slower like offload all I/O submission to another thread and incur a thread context switch for each one (with some opportunity for batching to make it not terrible). Obviously that throws out the synchronous I/O advantages.