Check for occupied niches #104862

saethlin · 2022-11-25T04:51:12Z

Implementation of rust-lang/compiler-team#624

Crater run has 62 crates that hit the check, 43 of those are published to crates.io. I see a lot of null function pointers and use of mem::uninitialized where the 0x1-filling collides with an enum niche.

But that is with full niche checks; checking transmute, plus any place where that we Copy, Move, or Inspect. Such checking is definitely too thorough to be on by default because it is 2x compile time overhead.

During implementation, this ran into llvm/llvm-project#68381

r? @ghost

rustbot · 2023-04-03T02:04:00Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

saethlin · 2023-04-03T02:50:17Z

This is not done yet. I wish generalized niche checks were as simple as what I whipped up here, and also CI doesn't pass for... reasons.

bors · 2023-05-02T17:45:33Z

☔ The latest upstream changes (presumably #111082) made this pull request unmergeable. Please resolve the merge conflicts.

Similar to the existing nullpointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@.

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR #104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

Similar to the existing nullpointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@.

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR #104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

Similar to the existing nullpointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang/rust#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@.

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang/rust#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

Similar to the existing nullpointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang/rust#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@.

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang/rust#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

Similar to the existing nullpointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@.

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

Insert checks for enum discriminants when debug assertions are enabled Similar to the existing null-pointer and alignment checks, this checks for valid enum discriminants on creation of enums through unsafe transmutes. Essentially this sanitizes patterns like the following: ```rust let val: MyEnum = unsafe { std::mem::transmute<u32, MyEnum>(42) }; ``` An extension of this check will be done in a follow-up that explicitly sanitizes for extern enum values that come into Rust from e.g. C/C++. This check is similar to Miri's capabilities of checking for valid construction of enum values. This PR is inspired by saethlin@'s PR rust-lang/rust#104862. Thank you so much for keeping this code up and the detailed comments! I also pair-programmed large parts of this together with vabr-g@. r? `@saethlin`

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 25, 2022

saethlin mentioned this pull request Nov 25, 2022

Insert alignment checks for pointer dereferences when debug assertions are enabled #98112

Merged

3 tasks