trait object: specific use case of variance and/or coercion

[plafer]

Filing this as a bug report because that's my best guess for what I believe is an inconsistency. Or perhaps simply the reference would need to be updated to explain the following.

Note that I will use <: to denote a "subtype" relationship.

I tried this code:

trait Module {}

struct Foo {
    module_box: Box<dyn Module + 'static>,
    module_rc: Rc<dyn Module + 'static>,
}

impl Foo {
    fn mut_box<'s>(&'s mut self) -> &'s mut (dyn Module + 's) {
        // this works
        self.module_box.as_mut() // : &'s mut (dyn Module + 'static)
    }

    fn mut_rc<'s>(&'s mut self) -> Option<&'s mut (dyn Module + 's)> {
        // this doesn't work
        Rc::get_mut(&mut self.module_rc) // : Option<&'s mut (dyn Module + 'static)>
        
        // but this does
//      match Rc::get_mut(&mut self.module_rc) {
//        Some(m) => Some(m),
//        None => None,
//      }
    }
}

I have 2 related questions:

1. why does the mut_box() implementation work?
2. why doesn't the mut_rc() implement work?

For (1), I first assumed that subtyping/variance was the reason, since (dyn Module + 'static) <: (dyn Module + 's) (although I wasn't able to 100% confirm). However, according to the reference, &'a mut T is invariant over T; so if I'm reading this correctly, &'s mut (dyn Module + 'static) is NOT a subtype of &'s mut (dyn Module + 's). I then thought maybe an implicit coercion could then explain why it compiled, but was not able to find one in the list of coercion types. Is this a bug?

Question (2) is very similar, except that the whole thing is wrapped in an Option, which seems to be the reason why it fails; the commented-out code unwraps the Option, which I believe sends us back to (1). Do we expect Rc::get_mut(&mut self.module_rc) to work, given that (1) works?

Meta

rustc --version --verbose:

rustc 1.68.0 (2c8cc3432 2023-03-06)
binary: rustc
commit-hash: 2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74
commit-date: 2023-03-06
host: aarch64-apple-darwin
release: 1.68.0
LLVM version: 15.0.6

Backtrace

error: lifetime may not live long enough
  --> fool/src/main.rs:21:9
   |
19 |     fn mut_rc<'s>(&'s mut self) -> Option<&'s mut (dyn Module + 's)> {
   |               -- lifetime `'s` defined here
20 |         // this doesn't work
21 |         Rc::get_mut(&mut self.module_rc) // : Option<&'s mut (dyn Module + 'static)>
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'s` must outlive `'static`
   |
   = note: requirement occurs because of a mutable reference to `dyn Module`
   = note: mutable references are invariant over their type parameter
   = help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance

error: could not compile `rust` due to previous error

[lcnr]

The region of trait objects can be changed using both subtyping and coercions. Note that the target of a mutable reference is always invariant. This is necessary for soundness as you can otherwise replace the target and then observe it after the mutable borrow ends:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=d669bdaf67af7053879378194c07c4d7

However, you cannot observe the underlying type of trait objects this way, so subtyping for trait objects is overly restrictive. We therefore allow coercions from dyn Trait + 'long to dyn Trait + 'short.

Coercions cannot automatically apply to fields as they could (in general) impact the layout of the given type, which is why mut_rc does not compile.

So to summarize: there are two different ways you can shorten the lifetime bound of trait objects, both of which only allow a subset of the theoretically sound behavior.

I am not sure where to best document this behavior 🤔 maybe in the reference section for trait objects?

[plafer]

We therefore allow coercions from dyn Trait + 'long to dyn Trait + 'short.

If I understand correctly, this explains why (1) compiles, right? AFAIK, this is not documented in the reference; I would expect to find it in the "type coercions" page.

Coercions cannot automatically apply to fields as they could (in general) impact the layout of the given type, which is why mut_rc does not compile.

But coercions from &T to &dyn Trait are allowed, even though the layout of &T is different from &dyn Trait (pointer vs fat pointer). Could you expand on this point? Is it that (except for unsized coercions), we don't want to coerce between types that will require runtime action?

Ultimately, I still find it surprising that Rc::get_mut(&mut self.module_rc) doesn't compile, but

match Rc::get_mut(&mut self.module_rc) {
  Some(m) => Some(m),
  None => None,
}

does. They are semantically equivalent; shouldn't either both compile, or both not compile?

[lcnr]

We therefore allow coercions from dyn Trait + 'long to dyn Trait + 'short.

If I understand correctly, this explains why (1) compiles, right? AFAIK, this is not documented in the reference; I would expect to find it in the "type coercions" page.

Yes

Coercions cannot automatically apply to fields as they could (in general) impact the layout of the given type, which is why mut_rc does not compile.

But coercions from &T to &dyn Trait are allowed, even though the layout of &T is different from &dyn Trait (pointer vs fat pointer). Could you expand on this point? Is it that (except for unsized coercions), we don't want to coerce between types that will require runtime action?

Yes, coercion from &T to &dyn Trait changes the layout and therefore require explicit support to actually transform the value of &T to a value of &dyn Trait.

Ultimately, I still find it surprising that Rc::get_mut(&mut self.module_rc) doesn't compile, but

match Rc::get_mut(&mut self.module_rc) {
  Some(m) => Some(m),
  None => None,
}

does. They are semantically equivalent; shouldn't either both compile, or both not compile?

The issue is that the coercion here happens inside of the Option. We would have to add explicit support to transform an Option<&mut T> to &Option<&mut dyn Trait>. This transformation has a runtime cost as the representation of &mut T and &mut dyn Trait differs.

While we could support it for Option, what about Result<Option<&mut T>, SomeError> or SomeOtherType<Option<&mut T>>. At some point the coercion gets too complex to support implicitly.

This is also not generally possible. You cannot transform &&mut T to &&mut dyn T because the coercion would have to change the size of the pointee (and mutate it).

[plafer]

We therefore allow coercions from dyn Trait + 'long to dyn Trait + 'short.

Could this rule be extended the allow coercions from Type<dyn Trait + 'long> to Type<dyn Trait + 'short>? IIUC, this should be doable since we know the layout won't change.

I considered whether this would make the language more complex; however I think it would make it less complex, as this rule extension would be in the theme of "coercions making things just work". Currently people have to learn about this edge case (as I did) whenever they run into a similar situation from mine, as opposed of it just working and them just moving on.

If not, then would it be possible to at least make the error message more intuitive? I didn't find the current one very helpful (even misleading):

error: lifetime may not live long enough
  --> src/main.rs:18:9
   |
16 |     fn mut_rc<'s>(&'s mut self) -> Option<&'s mut (dyn Module + 's)> {
   |               -- lifetime `'s` defined here
17 |         // this doesn't work
18 |         Rc::get_mut(&mut self.module_rc)
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ returning this value requires that `'s` must outlive `'static`
   |
   = note: requirement occurs because of a mutable reference to `dyn Module`
   = note: mutable references are invariant over their type parameter
   = help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance

In retrospect this is confusing because the error message points to the topic of "variance", which makes it hard to understand why mut_box compiled; I was asking myself "isn't there the same variance issue with mut_box?". Again, if possible, I believe extending the coercion rule (and documenting it) would remove any confusion.

[lcnr]

IIUC, this should be doable since we know the layout won't change.

expect that to make the impl far more complex 🤔 Agree that the error message could use a special case for trait objects or sth.

[aijkyo]

here is a similar case where the rustc seems to use coercion and subtyping in a confusing way(as return statement should be a coercion site according to https://doc.rust-lang.org/reference/type-coercions.html#coercion-sites)

trait Foo {}

struct Bar(Option<Box<dyn Foo>>);

impl Bar {
    // this code will not compile
    fn as_mut(&mut self) -> Option<&mut (dyn Foo)> {
        self.0.as_deref_mut().map(|x| {
            return x
        })
    }

    // this code will compile
    fn as_mut_coercion(&mut self) -> Option<&mut (dyn Foo)> {
        self.0.as_deref_mut().map(|x| {
           x
        })
    }

    // this code also compiles
    fn as_mut_explicit_coercion(&mut self) -> Option<&mut (dyn Foo)> {
        self.0.as_deref_mut().map(|x| {
           return x as &mut (dyn Foo)
        })
    }

    // this code also compiles
    fn as_mut_explicit_lifetime(&mut self) -> Option<&mut (dyn Foo + 'static)> {
        self.0.as_deref_mut().map(|x| {
           return x
        })
    }
}

Seems that as_mut will try to test the subtyping relationship(unexpected) but as_mut_coercion will try to do a coercion(expected).

And I think trait objects could be regarded as existential types, for example:

trait Foo {
  ...fn(&self, ...args: ...Args) -> OutputType;
  ...fn_mut(&mut self, ...args: ...ArgsMut) -> MutOutputType;
}

type &'a <mutability> (dyn Foo + 'b) = exist T: (Foo + 'b). struct {
    type = T: type,
    value: &'a mut <mutability> T,
    vptr: struct {
        ...fn(&T, ...args: ...Args) -> OutputType,
        ...fn_mut(&mut T, ...args: ...ArgsMut) -> MutOutputType;
    }
}

As we can see the struct contains value and vptr is immutable while the value is behind mutable reference inside the struct. There is no way for us to change the type inside the existential package(which is also the type of the value). We can only use vptr to change the status of the value.

If we have a package representing such a existential type. When we want to change it's type bound, the subtyping process is just to take everything out of the package and proving package.type meets the new type bound, then repackage as new existential type. It's basically the same process as coercion. So for a given type X, subtype relationship (&mut X as &mut (dyn Foo + 'long)) <: (&mut X as &mut (dyn Foo + 'short)) should hold. CMIIW.

Meta:

rustc 1.79.0-nightly (1388d7a06 2024-03-20)
binary: rustc
commit-hash: 1388d7a069d872bcfe5e5dd97ef61fa0a586fac0
commit-date: 2024-03-20
host: aarch64-unknown-linux-gnu
release: 1.79.0-nightly
LLVM version: 18.1.2