It seems to me that the only way for https://github.com/rust-lang/rust/pull/15833 to be an issue is if we never run `#[bench]`s anywhere during a PR merge (or on travis etc.). Maybe the test harness could run `#[bench]`s as normal tests during a normal run (i.e. execute the code exactly once, not bothering to time it).