Semantics of closures and structs do not exactly correspond

[nikomatsakis]

There are a few places where the semantics we enforce for closures could not be faithfully simulated by end-users. These omissions are accidental, but programs do rely on them, and removing them breaks reasonable looking programs. I identified these as part of a patch that exposed closures in a more accurate way.

The model that PR #27087 puts forward is that closures can be modeled as a structure like:

struct Closure<'l0...'li, T0...Tj, U0...Uk> {
    upvar0: U0,
    ...
    upvark: Uk
}

where 'l0...'li and T0...Tj are the lifetime and type parameters in scope on the function that defined the closure, and U0...Uk are type parameters representing the types of its upvars (borrowed, if appropriate). I'll not go into all the details here of why I chose this structure (it's explained in a comment in the code), but I do want to point out that clearly 'l0...'l1 and T0...Tj do not appear in the fields -- they are there so that when we are monormorphizing, we can easily access the "in-scope" substitutions from the enclosing fn, which may be needed for method calls and so forth.

Now, because 'li and Ti are unused, if a user were to write this, it would require a PhantomData. This would in turn make OIBIT consider those type parameters to represent reachable data, which might affect Send and so forth. However, the existing code only considered the types of a closure's upvars when computing OIBIT calculations, and I have preserved those semantics. This has a concrete impact if you have code like:

fn foo<T:Default>() {
    send(|| { let x = T::default(); ... })
}

Here, the closure is required to be sendable. This is considered true because it doesn't close over any state. However, if this were a struct with phantom data, it would be considered false, because the struct would be considered to potentially reach data of type T, which is not Send.

There is a similar case in the outlives relation:

impl<'a> Foo<'a> {
    fn method(&self) {
        let x = || something_that_does_not_use_self();
        bar(x); // requires that x: 'static
    }
}

fn bar<T: 'static>(t: T) { }

Here, we have an early-bound region 'a in-scope, and hence the type of x would correspond to C<'a> (there are no upvars and no type parameters). However, bar requires that C<'a>: 'static. This is currently considered to hold because there are no upvars. However, for a normal struct, this would require that 'a: 'static, which is false.

This last point means that this closure can escape 'a, since there is no actual reachable data with that lifetime. (This is also true for regular fns and I think true for impls since they are permitted to have extra lifetime parameters.)

Anyway, we need to consider whether and how to close these gaps -- perhaps by extending normal structs, perhaps by limiting closures.

cc @rust-lang/lang

[arielb1]

Isn't this only important from a function-local perspective? Because of the lack of HKT, extra-function closures are present only when they are substituted into other types - and in that case, the exact value of the substitutions doesn't matter (in fact, AFAIK closure signatures can even contain ReScope-s from their containing function).

Actually, that's not exactly correct - as you said, closures can essentially use InvariantType for some of their type parameters - they can reference a type parameter but not be affected by its OIBIT. If the set of OIBIT is fixed, this can be done by the equivalent of

use std::marker::PhantomData;
pub struct InvariantType<T>(PhantomData<*mut T>);
impl<T> Copy for InvariantType<T> {}
impl<T> Clone for InvariantType<T> { fn clone(&self) -> Self { *self } }
unsafe impl<T> Send for InvariantType<T> {}
unsafe impl<T> Sync for InvariantType<T> {}

It may be wise to introduce InvariantType<T>.

[eefriedman]

I can't remember who pointed this out, but it's currently possible to use PhantomData with a function pointer type:

use std::marker::PhantomData;
pub struct Invariant<T>(PhantomData<fn(&T)>);
fn send<T:Send>(_:T) {}
fn foo<T:Default>() {
    send(Invariant::<T>(PhantomData));
}
fn main() {
    foo::<i32>();
}

rustc correctly figures out that Invariant is Send.

[arielb1]

@eefriedman

That's contravariant. PhantomData<fn(*mut T)> would be invariant.

That's just a nit, anyway.

[arielb1]

You can in fact semantic-equivalently desugar closures using fn-items (with ABI differences, mostly involving fn-items not being zero-sized):

struct MutClosure<B, E> {
    body: B,
    environment: E
}
impl<B, E, P> FnOnce<P> for MutClosure<B,E> where B: Fn<(&mut E, P)> {
    type Output = B::Output;
    extern "rust-call" fn call_once(self, args: P) -> B::Output {
        self.body.call((&mut self.environment, args))
    }
}
impl<B, E, P> FnMut<P> for MutClosure<B,E> where B: Fn<(&mut E, P)> {
    extern "rust-call" fn call_mut(&mut self, args: P) -> B::Output {
        self.body.call((&mut self.environment, args))
    }
}
fn foo<T: Default>() {
    let foo = ...;
    let bar = ...;
    hof(|arg1, arg2| { /* closure body */ })
}
fn foo_desugared<T: Default>() {
    let foo = ...;
    let bar = ...;
    // it really doesn't matter if you pass extra early lifetimes in - they are substituted
    // early anyway.
    fn closure<'π, T: Default>((foo, bar): (&'π mut _, &'π mut _), (arg1, arg2): (_, _)) -> _ {
        // closure body
    }
    hof(MutClosure { body: closure::<T>, environment: (&mut foo, &mut bar) })
    //^ note we are creating a MutClosure<fn(...) {closure}, (_, _)>
}

[arielb1]

In other words, closures basically get their type-lifetime-erasing power from fn-items.

[steveklabnik]

Triage: looks like a discussion that didn't really go anywhere. Not sure what to do.

[arielb1]

Now fn items are always zero-sized, so closures can always be converted to the XxxClosure<["rust-call" fn item], env> form.

The fn item "truncation" issue is still present. It's more of a formalization issue than a compiler issue, through, so I'm closing this.