Why does tokio::sync::oneshot implement cooperative scheduling?

[olix0r]

There's been a long-standing bug in Tower that we recently tracked down to some very surprising behavior in tokio::sync::oneshot. Basically, we have an assumption that if a oneshot::Sender has sent a value, then it must be observable by the oneshot::Receiver. But this assumption is violated by Tokio's cooperative scheduling, which can sporadically cause a oneshot::Receiver::poll to return Pending even though a value has been sent. The solution to this issue is to wrap oneshot::Receiver with tokio::task::unconstrained so that its polls are not impacted by budgeting. This behavior is subtle, implicit, and undocumented--it seems extremely likely that there are other similar bugs lurking anywhere a oneshot is used for synchronization.

In my view, this dramatically undermines the utility of oneshot as a synchronization primitive.

While I understand the goals of cooperative scheduling task budgeting, I don't understand what this approach protects against in the case of oneshot. A receiver:

• can only return ready once
• should not be prone to spurious wake-ups -- i.e. a receiver should not need to return pending more than once in most uses.

What problems would arise if we changed oneshot's behavior to stop participating in task budgeting?

[olix0r]

Here's a minimal example of the issue for those unfamiliar with this:

#[tokio::main(flavor = "current_thread")]
async fn main() {
    for i in 1..=129 {
        let (tx, rx) = tokio::sync::oneshot::channel();
        tx.send(()).unwrap();
        tokio::select! {
            biased;
            _ = rx => {}
            _ = futures::future::ready(()) => panic!("should not hit this: {} iters", i),
        }
    }
}

thread 'main' panicked at 'should not hit this: 129 iters', src/main.rs:12:47

[hawkw]

While I understand the goals of cooperative scheduling task budgeting, I don't understand what this approach protects against in the case of oneshot. A receiver:

• can only return ready once

• should not be prone to spurious wake-ups -- i.e. a receiver should not need to return pending more than once in most uses.

What problems would arise if we changed oneshot's behavior to stop participating in task budgeting?

I believe the reason that polling a oneshot::Receiver calls poll_proceed is not because a single oneshot is responsible for a large number of polls, but because a task may be polling a very large number of different completed oneshots in a loop or something.

It's certainly possible that it's not desirable for oneshots to do this, but I'm pretty sure that was the motivation.

[olix0r]

@Darksonn and @carllerche explained this a bit more on discord. The real issue is that oneshot only really guarantees that the receiver will observe values at some point in the future after a sender sends a value. There are no atomicity/visibility guarantees, even when using tokio::task::unconstrained. To put it simply: oneshot is not a synchronization primitive, it deals only in message-passing. A single task that sends a value may not necessarily be able to (immediately) observe that value.

We probably should make these constraints much clearer in the oneshot documentation.

I also suggested that it may be beneficial to have a runtime configuration for tests that enforces much more aggressive (random?) task yielding so that these behaviors are easier to observe in development.

[barafael]

I also suggested that it may be beneficial to have a runtime configuration for tests that enforces much more aggressive (random?) task yielding so that these behaviors are easier to observe in development.

https://crates.io/crates/madsim-tokio

[BratSinot]

[Darksonn]

Do not use the answer box to post questions. Post a new question instead.