What about: using pointers to fields of an enum to change the active variant?

[RalfJung]

(Taken from #69 (comment).)

How do we feel about this code -- UB or not?

fn main() { unsafe {
    let mut x = Some(&0); 
    match x {
        Some(ref mut b) => {
            let u = b as *mut &i32 as *mut usize;
            // Just writing into a *mut u8
            *u = 0;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

This "magically" changes the discriminant. OTOH, it is very similar to

fn main() { unsafe {
    let mut x = Some(&0); 
    (&mut x as *mut _ as *mut usize) = 0;
    assert!(x.is_some());
} }

which is definitely allowed (it makes assumptions about layout, but we do guarantee layout of Option<&T>.

Other example:

fn main() { unsafe {
    let mut x = Some(true); 
    match x {
        Some(ref mut b) => {
            let u = b as *mut bool as *mut u8;
            // Just writing into a *mut u8
            *u = 2;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

#![feature(rustc_attrs)]

#[rustc_layout_scalar_valid_range_start(1)]
#[repr(transparent)]
pub(crate) struct NonZero(u32);

fn main() { unsafe {
    let mut x = Some(NonZero(1)); 
    match x {
        Some(NonZero(ref mut c)) => {
            // Just writing 0 into an &mut u32
            *c = 0;
        }
        None => panic!(),
    }
    assert!(x.is_some());
} }

Relevant questions:

• Is changing the discriminant through a pointer derived from an enum variant okay if there's never again an access with the outer pointer? In other words, does this act like a protector where "invalidating" the discriminant seen by parent pointers is immediate UB? If no, there are still two variants: one where the discriminant can be "repaired" again (i.e. we just check the discriminant again when the parent pointer is used), and one where it is permanently invalidated (when overwriting the discriminant the parent pointer becomes unusable but the child pointers remain usable). This was brought up here.

[hanna-kruppe]

To clarify, the last example doesn't need the unsafe {}, right? It's not "safe code" (you can modify it to be unquestionably UB by omitting the Option and reading x after the end) but that's due to the use of #[rustc_layout_scalar_valid_range_start], not due to any individual operation in main.

[RalfJung]

@rkruppe It needs the unsafe, we have fixed unsafety checks in rustc to require an unsafe block to create a reference into a rustc_layout_scalar_valid_range_start struct.

[RalfJung]

See #189 for me describing this problem again. ;)

An interesting observation: the following code is certainly fine; FFI code does something like this all the time when using Option<&T> on the interface:

fn main() {
    let mut x = Some(NonZeroU32::new(1).unwrap());
    let xref = &mut x as *mut _ as *mut u32;
    unsafe { *xref = 0 };
    println!("{:?}", x);
}

Operationally, this is basically the same as the problematic

use std::num::NonZeroU32;

fn main() {
    let mut x = Some(NonZeroU32::new(1).unwrap());
    let xref = match x {
        Some(ref mut c) => c as *mut NonZeroU32 as *mut u32,
        None => panic!(),
    };
    unsafe { *xref = 0 }; // writing 0 into a `*mut u32`
    println!("{:?}", x);
}

To distinguish the two, xref would need to have some kind of "provenance" remembering that it was created from a pointer to NonZeroU32, not a pointer to u32.

[CAD97]

Just some musings I came up with:

When moving a !Copy type, invalidating the old name is obviously correct.

Example:

let a: Thing1 = default();
let aref = &a as *const _;
let b: Thing2 = transmute(a);
// `aref` is invalid _even if_ we guaranteed `b` were in the same physical location
// as a logical move has occurred

So maybe the last copy of a Copy value could be a move as well?

Example:

let mut x = Some(&0);
match x {
    Some(ref mut b) => {
        let u = b as *mut bool as *mut u8;
        // `b` is no longer required to be valid, as it has been "moved"
        // Just writing into a *mut u8
        *u = 2;
    }
    None => panic!(),
}
dbg!(x);

On the other hand, the origin provenance makes sense as well. Using this example for clarity:

let mut x = Some(NonZeroU32::new(1).unwrap());
let xref = match x {
    Some(ref mut c) => c as *mut NonZeroU32 as *mut u32,
    None => panic!(),
};
unsafe { *xref = 0 }; // writing 0 into a `*mut u32`
dbg!(x);

It "makes sense" that the c binding only "owns" the right to write to the NonZeroU32.

Naively, it feels like this should behave the same as

let mut x = (1u32, 0u32);
let x1ref = &mut x.1 as *mut u32;
let xref = xref.sub(1) as *mut (u32, u32);
*xref = (0, 0);
dbg!(x);

Specifically, it's taking a reference into the "interior" of an allocated object and "upgrading" it to a pointer to the full object to write the full object. It just so happens that the physical pointers are the same.

My intuition here is suggesting something like guaranteed niche optimizations shouldn't matter for what operations you are allowed do with a pointer, just how you're able to accomplish said operations, as well as what memory reinterpretation is allowed. I don't know how practical that is, but it seems somewhat logical. (e.g. <*const Option<NonZeroU32>>::cast::<u32> is valid because of the guaranteed niche optimization, but <*const NonZeroU32>::cast::<Option<NonZeroU32>> is not valid, even if said pointer lives in that type (unless it was derived from a wider pointer).)

[RalfJung]

Specifically, it's taking a reference into the "interior" of an allocated object and "upgrading" it to a pointer to the full object to write the full object. It just so happens that the physical pointers are the same.

Yes, that is a way to look at it. Your "upgrading" example would be rejected by Stacked Borrows because that pointer is not allowed to write to the "outer" memory.
On the other hand, physical overlap is the thing that matters for aliasing, which is what Stacked Borrows is all about.
And note that there is nothing in the validity machinery that has any problem with deriving "outer" pointers from "inner" pointers". Only Stacked Borrows cares.

[RalfJung]

(Moved to new issue)