Test whether self-referential futures are compatible with stacked borrows

[RalfJung]

I have no idea if an async function that borrows across a yield point is compatible with Stacked Borrows. We should find out! I just am not sure how.

@cramertj @withoutboats I could use your help. :) What is a minimal example for code that creates a self-referential future and then runs it? Ideally with no dependencies beyond libstd but that might not be possible.^^ Also, if there are different cases/"kinds" of self-referential fields in futures, we should make sure we cover all of them.

rustc must test these things somehow, right? Might be worth looking into that, too.

[Nemo157]

I'm actually working on trying to create a step-by-step example of how an async fn is first transformed into a self-referential generator then into a state machine. I should have a nice small self-contained example that uses both references into upvars and local variables ready in the next couple of days. (I already have this playground that I intended to use, but I just realised I made a mistake in expanding await! that means this doesn't actually require self-references into local variables, just upvars).

if there are different cases/"kinds" of self-referential fields in futures, we should make sure we cover all of them

There are only two different kinds that I know of, and fundamentally they should be the same. You can have references to upvars that have been moved into the generator and references to locals created while the generator is running. Once the generator transform has run these both just become references into the generator state variable, and I don't think they're distinguishable in the MIR.

[Nemo157]

Actually I might as well throw something together now, here should be a small example showing all kinds of references in an async fn. y is an upvar of the generator created in foo and z is a local of it; a reference to each are moved into the async block on line 14, inside await! the result implementation of this block is stored into a local variable (so that variable is referencing both y and z in the outer generator) and polled to completion over a yield point. a is a local of the generator which has a reference taken but is dereferenced and dropped without hitting a yield point. Finally x is kept as reference out of the generator environment, as an example of non-self-referential references.

There's also the minimal framework for running an async fn inside main, this may be changing a little bit in the near future in a couple of ways, but I'll be creating an up-to-date version of this example as it does.

[withoutboats]

@RalfJung might it be easier to focus on generators before async fn, since they have direct access to yield?

[cramertj]

Here's a self-referential generator example:

#![feature(generators, generator_trait)]

use std::ops::Generator;

fn main() {
    let mut generator = || {
        let mut x = Box::new(5);
        let y = &mut *x;
        *y = 5;
        yield ();
        *y = 10;
        *x
    };
    
    unsafe { generator.resume() };
}

[RalfJung]

@cramertj Thanks! I thought there had to be async and a Future. I clearly know nothing about how async functions work.^^

[RalfJung]

To my big surprise, this actually passes in miri :D

EDIT: Even more interestingly though, it no longer passes locally...

[RalfJung]

Oh. The generator MIR transform happens after we added the retag statements, leading to... strange results.

However, the generator MIR stuff happens extremely late -- it happens after inlining! That seems really strange to me, shouldn't we first get the MIR "complete" before doing interesting transformations?

[RalfJung]

See rust-lang/rust#56100

[RalfJung]

@withoutboats That makes sense, I wasn't aware of the layering here. This seems covered by #536 though, and it's actually working (with a small rustc fix), which is great!

However, something like @Nemo157's example also seems like a nice addition. Thanks for that @Nemo157! It also seems to pass, to my ever greater surprise.^^