Need a solution for select / async events

[brson]

Pipes supports 'select', which allows one to receive on multiple ports at the same time. The new scheduler doesn't have a solution for this, but it needs to be more general than the old one.

Some types of events we might want to wait on:

• Port receives, both one-shot, stream and other receives
• I/O reads of all kinds
• Timers
• Socket accepts

Miscellaneous notes:

The biggest requirement is that we need one abstraction that allows for waiting on both I/O events (single-threaded) and comm events (multithreaded). uv handles and comm types have very different implementations.

select may not be the right abstraction. Maybe there are more expressive ways to do this. We could for instance have an event loop with callbacks. Also look an .NET's async I/O which I know nothing about. Needs more research.

Asynchronous I/O events should hopefully allow for all the various extension methods and decorator types that synchronous I/O allows.

The solution should additionally be workable for tasks that are not coroutines scheduled by an I/O event loop. We would like to have tasks that are backed simply by threads and don't use the I/O event loop and that makes the design of various runtime components more difficult. It's potentially ok not to solve this yet.

Some types of events need to be cancellable as well, e.g. if you wait on both a timer and an accept, if the timer goes off first you may want to cancel the accept.

[brson]

This will be greatly complicated by the fact that I/O types are not bound to their schedulers - so it won't be as simple as registering a uv callback.

[mneumann]

Wouldn't it be better if tasks would always stay in the Scheduler they are created in? Or if you migrate them, also somehow migrate the task's libuv context?

[thadguidry]

This idea comes out of the Data Architect world that I live in. It's more along the lines of that basic idea that your pitching which I call "receivership".

In Pentaho Data Integration, there is a process step called "Block this step until steps finish". What it does is Block a single step in an execution flow until a selected step or even multiple steps somewhere in the chain or process has completed it's execution and returned a True for it's "execution complete flag" where a launched listener is waiting and watching.

It's a generic wrapper if you will, for additional next steps to perform, but those next steps do not get called, do not get any input, and no variables are passed to them until the the child Blocking step has seen from (there's a listener process launched for each step or steps choosen to watch) all its parent or parents steps somewhere say that it's own execution has finished and lets the listener know. I think the "Block this step until steps finish" just waits for all those listeners it is watching to say it's OK to proceed with it's own set of next steps.

Anyways, just wanted to throw this out there... and maybe this idea should be focused on another issue, feel free to move it there... here's their Javadoc http://javadoc.pentaho.com/kettle/org/pentaho/di/trans/steps/blockuntilstepsfinish/BlockUntilStepsFinish.html

[brson]

@mneumann I believe that it is not better for tasks to stay in the scheduler that created them. Here scheduler == thread, and it's easy under such a strategy not to distribute load effectively (fwiw our current scheduler never migrates tasks and it's easy to underutilize cores. This is easy to see in long running test cases that block other test cases while leaving other cores unoccupied).

The libuv context is essentially the libuv event loop, which is deeply integrated with the scheduler - it's what drives the scheduler and allows synchronous I/O to yield to other tasks - so the event loop itself cannot change threads. It may be possible in some circumstances to transplant file handles from one libuv event loop to another, but I don't think libuv currently has such functionality, and it isn't possible in all cases.

We could make a limitation that I/O handles are not sendable, but that is fairly restrictive.

[brson]

@thadguidry I responded on the list.

Putting the list link here for posterity: https://mail.mozilla.org/pipermail/rust-dev/2013-May/004305.html

[mikedilger]

I haven't used libuv, but I understand it is for platform independence. My comments will be only about linux, as that is the only knowledge base I have worth contributing from.

In newer kernels, the different types of events have all been unified to use file descriptors and thus be "select"-able with epoll_wait(). Timers can be detected with timerfd, signals with signalfd, and network and file activity with the file descriptors you are already familiar with. Thus all these events can be handled asynchronously non-blocking in edge-triggered mode. epoll() is thread-aware in a way that multiple threads can wait on the same epoll, and the event will go to whichever one the kernel is most enamoured with ;-)

I have some very basic code (560 lines) that touches on all this if you want to see an example.

[mikedilger]

https://github.com/mikedilger/epoll_demo

[mneumann]

I think the "problem" is that libuv/libevent can only select on kernel events while we also want to wait for port receives which are handled in pure user level. To solve it, you have to use a regular pipe (known to the kernel) which can awake the event loop from waiting. This is how libuv implements async callbacks. Async in this context means that you invoke a callback in the same thread as the event loop, but triggered from another thread.

I don't know if you will use a separate I/O task, or just one thread that performs all I/O and also runs the tasks. In the latter case you'd only have to do an async cross-thread call when:

1. The port receive is from a task on another scheduler/native thread and
2. this thread is currently blocked in the event loop waiting for I/O to happen.

You can special case 1. and directly schedule to the receiving task. And 2. you can handle with an async callback. Actually you can use an async callback whenever the receiving tasks lives in another scheduler. But I think you have to take care when you run the async callback (and for example schedule another thread from there) and then another async callback is triggered, you might loose it or not.

[mneumann]

@brson: I think thread migration is a good general solution as long as you are (optionally) able to pin a task to it's originating scheduler to avoid trashing. And maybe it's good when threads are randomly distributed at creation time to limit the work stealing.

[mneumann]

Timers, socket accepts and I/O of all kind would be handled by libuv. The callbacks would just set a bit in the corresponding Rust I/O object, and determine if the corresponding task would become runnable as a result. After all callbacks have been run by libev, the scheduler would then "simply" run all runnable threads. Port receives would be triggered (as I described above) via async callbacks. An async callback is "global" (only one per event loop), i.e. you need to keep information about the ports which contain messages in another data structure, which then the async callback can inspect and perform the same action as for other callbacks (determine if corresponding task would become runnable).

[brson]

After further consideration with @bblum on IRC I think it may not make sense to conflate waiting on pipes with waiting on I/O - especially if it's going to cause complexity or extra overhead for either. (Rust) pipes and sockets are pretty easy to adapt to each other. In particular, for the important use case of waiting on I/O while also waiting on some other signal, I think we can use an I/O-based message abstraction, i.e. something that behaves like pipes but is backed by unix domain sockets.

[pnkfelix]

@brson Would signal handling support be considered part of the more general problem of asynchronous events described here? (I am wondering whether bugs that could be resolved via a program-specified signal handler should be considered blocked by this issue.)