Investigate consolidating SocketAsyncEngine into ThreadPool

[stephentoub]

We should investigate / experiment with moving SocketAsyncEngine down to be integrated with the ThreadPool. Today all of our logic for efficiently waiting on file descriptors (and specifically sockets) via epoll/kqueues is available only through System.Net.Sockets, and handles its own thread management. We should explore some kind of ThreadPool.RegisterFileDescriptorForPolling(IntPtr fd, Action<object> callback, object state) Unix-specific API that would enable:

• other I/O-related APIs to utilize this waiting support, without needing an artificial dependency on Socket to do so
• the ThreadPool to manage the threads utilized by SocketAsyncEngine today, hopefully more efficiently and potentially avoiding some of the thread hopping we incur today

(I thought we already had an issue for this, but I can't find it; someone please feel free to close this as a dup if you find it :)

cc: @geoffkizer, @kouvel, @jkotas, @davidfowl

[davidfowl]

Is the proposal missing an IntPtr in the callback as the first argument?

ThreadPool.RegisterFileDescriptorForPolling(IntPtr fd, Action<IntPtr, object> callback, object state)

I think this could also be used for windows IOCP. I wonder if we need to support more than an IntPtr so that we can potentially change the implementation behavior based on the type.

[stephentoub]

Is the proposal missing an IntPtr in the callback as the first argument?

I expect it's missing a bunch of things ;-)

[adamsitnik]

My only concern is that epoll does not support regular disk files, so this solution can't be reused for all kinds of streams.

I wonder whether we should revisit using io_uring again and adapt ThreadPool to work with io_uring instead. It has the obvious downside of being Linux-specific and supported only by new kernels, but might be more future-proof (it seems that io_uring might finally become the async IO done right on Linux).

[davidfowl]

I think we this API should work for both windows and linux and should basically abstract this async IO model for all operating systems.

[tmds]

SocketAsyncEngine deals with more than polling. It keeps track of multiple async reads/write that are performed against a SafeHandle and executes them in order. It also supports aborting all on-going operations when the SafeHandle gets disposed.

This requires per SafeHandle bookkeeping of on-going async operations.

[geoffkizer]

I think we this API should work for both windows and linux and should basically abstract this async IO model for all operating systems.

I don't think this is the appropriate level at which to do something like this. The API semantics are quite different between epoll and IOCP; that's why we have all the SocketAsyncContext stuff.

We do have something of an IO abstraction in the Socket PAL. It's not very clean currently, but I think it's more in line with what you are asking for here.

[geoffkizer]

BTW, I'm very much in favor of doing something like this.

One design issue to consider: Epoll (and kqueue) don't just signal general fd readiness; they specifically signal read readiness or write readiness (or both). And they also signal EOF for both.

So we either need a callback that indicates the specific readiness condition, or we need multiple callbacks for different conditions. My suggestion would be to have two callbacks, one for read and one for write, and allow each callback to also signal EOF.

There's some weirdness that exists in SocketAsyncEngine today around handle lifetime management. We should try to clean this up in any new API. I believe part of the problem here is determining when a fd will no longer generate readiness notifications and thus doesn't need to be tracked any more by the runtime. We should chat about the details here.