Layout optimization fails for enums with variants having values 0 and 255

[Kixunil]

When a C-like enum has two variants, one being set to 0 and the other being set to 255 the layout optimizations fail to apply even though they do apply if the values are different.

I tried this code:

struct Foo {
    _bar: Bar,
    _baz: [u8; 7],
}

enum Bar {
    _A = 0,
    _B = 255,
}

struct Error {
    _x: bool,
    _y: bool,
}

fn main() {
    println!("{}", core::mem::size_of::<Result<Foo, Error>>());
}

I expected to see this happen: 8 being printed because of layout optimization, since all values 1..=254 are invalid the optimizer can pick any of them to represent the Err variant and store the error data in the space that would be otherwise occupied by [u8; 7]. This already works if the values are different.

Instead, this happened: The output is 9, implying the optimization was not applied. If you change the value of _A or _B then the optimization is applied (prints 8).

Meta

rustc --version --verbose:

rustc 1.86.0-nightly (ae5de6c75 2025-01-29)
binary: rustc
commit-hash: ae5de6c759cd337ecdb2de4e94f47eaafb5d4606
commit-date: 2025-01-29
host: x86_64-unknown-linux-gnu
release: 1.86.0-nightly
LLVM version: 19.1.7

[primoly]

The compiler does not check all available discriminants. It only looks at the minimum and maximum values (in your case 0 and 255) and searches for one outside this range. Since this leaves no room for discriminants that fit in one byte, it will increase the tag’s size. That’s why increasing the lower or decreasing the higher discriminant fixes the issue.

[Kixunil]

Well, it'd be nice to fix is since this can happen in practice and it's actually useful.