Memory corruption (?) from tower-grpc-build on nightly

[vorner]

Hello

When playing with some development versions of tower-grpc-build (https://github.com/tower-rs/tower/grpc), I discovered a strange crash of the build script. I managed to shring it a bit, but I don't think this is minimal case (I just don't know how to continue pinning it down further).

Few observations:

• The string it talks about in the backtrace seems like random junk (which isn't always the same). I'm not sure how it got there, but in general, Rust probably should prevent creation of strings with uninitialized content.
• It doesn't happen on stable.
• It happens on today's and yesterday's nightly (exact version below).
• It doesn't happen on approx. a week old nightly, but I'm not at the computer that has the older version, I'll look exact version when I get to it in few hours.

As I needed to pin exact commits in Cargo.lock to make the branches and crates from git to compile together, I have the example as a whole repository: https://github.com/vorner/p-crash.

Steps to reproduce (I moved the failing code from build script to ordinary main.rs, therefore the OUT_DIR):

rustup default nightly
OUT_DIR="." cargo run

The backtrace:

thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside '\u{0}' (bytes 0..1) of ` âU�âUâU� PâU�Cbyte index 1 is not a char boundary; it is inside ams_p._ping_re`[...]', /checkout/src/libcore/str/mod.rs:2234:5
stack backtrace:
   0:     0x564e394ee1fb - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hb720ecb1cdd94d34
                               at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1:     0x564e394f1c4e - std::sys_common::backtrace::print::h3b9052ae1243ac3c
                               at /checkout/src/libstd/sys_common/backtrace.rs:68
                               at /checkout/src/libstd/sys_common/backtrace.rs:57
   2:     0x564e394e8930 - std::panicking::default_hook::{{closure}}::h8386650ae58050b2
                               at /checkout/src/libstd/panicking.rs:381
   3:     0x564e394e8465 - std::panicking::default_hook::h939eef3f926b9ec4
                               at /checkout/src/libstd/panicking.rs:397
   4:     0x564e394e8cbb - std::panicking::rust_panic_with_hook::hd65467d489bb99a6
                               at /checkout/src/libstd/panicking.rs:577
   5:     0x564e394e8b0e - std::panicking::begin_panic::h06b05d85af0a01cf
                               at /checkout/src/libstd/panicking.rs:538
   6:     0x564e394e8a7a - std::panicking::begin_panic_fmt::hdcdf37dcdaa48dbb
                               at /checkout/src/libstd/panicking.rs:522
   7:     0x564e394e8a12 - rust_begin_unwind
                               at /checkout/src/libstd/panicking.rs:498
   8:     0x564e39540e40 - core::panicking::panic_fmt::h1bceab3183b318fb
                               at /checkout/src/libcore/panicking.rs:71
   9:     0x564e3953f4ed - core::str::slice_error_fail::h92f09d0a955fe52f
                               at /checkout/src/libcore/str/mod.rs:0
  10:     0x564e394a04de - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}::h7353d5e4c348ff28
                               at /checkout/src/libcore/str/mod.rs:1987
  11:     0x564e3949e455 - <core::option::Option<T>>::unwrap_or_else::hde62f1acbfa5a0d6
                               at /checkout/src/libcore/option.rs:376
  12:     0x564e394a0ad9 - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::hea07b70cbc76dfe2
                               at /checkout/src/libcore/str/mod.rs:1987
  13:     0x564e394a0494 - core::str::traits::<impl core::ops::index::Index<core::ops::range::RangeFrom<usize>> for str>::index::he1272c2136040a65
                               at /checkout/src/libcore/str/mod.rs:1734
  14:     0x564e3949d564 - unicode_segmentation::word::UWordBounds::get_next_cat::hd9df4c8a50c69cf3
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:611
  15:     0x564e3949cc48 - <unicode_segmentation::word::UWordBounds<'a> as core::iter::iterator::Iterator>::next::hfaaee7c167be77af
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:227
  16:     0x564e3949d7ee - <&'a mut I as core::iter::iterator::Iterator>::next::h1d6ba2952b78d957
                               at /checkout/src/libcore/iter/iterator.rs:2380
  17:     0x564e394a36b8 - <core::iter::Filter<I, P> as core::iter::iterator::Iterator>::next::h735c3caf6113fcd5
                               at /checkout/src/libcore/iter/mod.rs:1362
  18:     0x564e3949d44b - <unicode_segmentation::word::UnicodeWords<'a> as core::iter::iterator::Iterator>::next::h05b4feaa8a7b6004
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:30
  19:     0x564e394a12ea - heck::transform::h2bd9c6ea08dc34e6
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/heck-0.3.0/src/lib.rs:81
  20:     0x564e3949c503 - <str as heck::snake::SnakeCase>::to_snake_case::h13cce73744a245f8
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/heck-0.3.0/src/snake.rs:37
  21:     0x564e3942fef1 - prost_build::ident::to_snake::h6cb82fcbe7a31a26
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/ident.rs:8
  22:     0x564e3944b177 - core::ops::function::FnMut::call_mut::he9df2a59f21ce2ef
                               at /checkout/src/libcore/ops/function.rs:146
  23:     0x564e3942dd36 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &'a mut F>::call_once::hc412d0b7423e7dd9
                               at /checkout/src/libcore/ops/function.rs:271
  24:     0x564e3940dadf - <core::option::Option<T>>::map::ha68eff2bf856d1c0
                               at /checkout/src/libcore/option.rs:404
  25:     0x564e3944789a - <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next::hf391941cd12c07f0
                               at /checkout/src/libcore/iter/mod.rs:1251
  26:     0x564e39447ce8 - <core::iter::Chain<A, B> as core::iter::iterator::Iterator>::next::hde73355ad686dd14
                               at /checkout/src/libcore/iter/mod.rs:758
  27:     0x564e39447aaa - <core::iter::Chain<A, B> as core::iter::iterator::Iterator>::next::h158ea3896fb6310f
                               at /checkout/src/libcore/iter/mod.rs:754
  28:     0x564e39449b76 - itertools::Itertools::join::h6e55fdae653e6cbd
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/itertools-0.6.5/src/lib.rs:1203
  29:     0x564e3943e209 - prost_build::code_generator::CodeGenerator::resolve_ident::hf5ad253193aac324
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:574
  30:     0x564e3943d192 - prost_build::code_generator::CodeGenerator::unpack_service::{{closure}}::h00de60f26a10433d
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:477
  31:     0x564e39433c77 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &'a mut F>::call_once::h4b79d43010f986cb
                               at /checkout/src/libcore/ops/function.rs:271
  32:     0x564e3940ce5e - <core::option::Option<T>>::map::h933162c45fb6401e
                               at /checkout/src/libcore/option.rs:404
  33:     0x564e394476c3 - <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next::h0913e1a7621c5c9e
                               at /checkout/src/libcore/iter/mod.rs:1251
  34:     0x564e3941dc9c - <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend::hb997b01e2b5e5058
                               at /checkout/src/liballoc/vec.rs:1844
  35:     0x564e3941ed6e - <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter::h777d37041a1c7591
                               at /checkout/src/liballoc/vec.rs:1827
  36:     0x564e3941f336 - <alloc::vec::Vec<T> as core::iter::traits::FromIterator<T>>::from_iter::h43714bb9d059e0ac
                               at /checkout/src/liballoc/vec.rs:1713
  37:     0x564e39444c87 - core::iter::iterator::Iterator::collect::h436cb81ed82afe86
                               at /checkout/src/libcore/iter/iterator.rs:1298
  38:     0x564e3943ca1b - prost_build::code_generator::CodeGenerator::unpack_service::hfea0f83504e71fec
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:465
  39:     0x564e39434d87 - prost_build::code_generator::CodeGenerator::generate::hab3bbb91478da3b9
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:111
  40:     0x564e3940a9c4 - prost_build::Config::generate::hd148016b47bb16b0
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/lib.rs:342
  41:     0x564e393d1aff - prost_build::Config::compile_protos::hc4c921ff79be61f5
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/lib.rs:321
  42:     0x564e393e02e6 - tower_grpc_build::Config::build::h85b9824f7574fc87
                               at /home/vorner/.cargo/git/checkouts/tower-grpc-02dc54165a968a91/a844add/tower-grpc-build/src/lib.rs:92
  43:     0x564e393dde07 - sscli::main::h0f8da34d077bac22
                               at src/main.rs:4
  44:     0x564e393e0171 - std::rt::lang_start::{{closure}}::h115d56dea3cc829d
                               at /checkout/src/libstd/rt.rs:74
  45:     0x564e394e89d7 - std::panicking::try::do_call::h9688ac8626758e25
                               at /checkout/src/libstd/rt.rs:59
                               at /checkout/src/libstd/panicking.rs:480
  46:     0x564e395069fe - __rust_maybe_catch_panic
                               at /checkout/src/libpanic_unwind/lib.rs:101
  47:     0x564e394edfd5 - std::rt::lang_start_internal::ha33cbd0fd3958dd3
                               at /checkout/src/libstd/panicking.rs:459
                               at /checkout/src/libstd/panic.rs:365
                               at /checkout/src/libstd/rt.rs:58
  48:     0x564e393e0151 - std::rt::lang_start::h40dceadf8934dbfd
                               at /checkout/src/libstd/rt.rs:74
  49:     0x564e393dde5d - main
  50:     0x7f0656239040 - __libc_start_main
  51:     0x564e393934d9 - _start
  52:                0x0 - <unknown>

It is likely something fishy is happening in some of the development branches, but I still suspect something strange happening in the compiler, both because of the uninitialized memory (or, memory looking uninitialized) and because it passes on stable, but not nightly.

[nikomatsakis]

cc @rust-lang/infra -- any chance somebody can bisect this a bit to see where the problem started?

[nikomatsakis]

triage: P-high

[vorner]

If it helps, this one goes through (rustc 1.24.0-nightly (9389e23a8 2017-12-31)):

OUT_DIR=. cargo +nightly-2018-01-01 run

And this one fails (rustc 1.24.0-nightly (b65f0bedd 2018-01-01)):

OUT_DIR=. cargo +nightly-2018-01-02 run

Anything more exact would need setting up of some non-trivial infrastructure on my own computer.

[kennytm]

Given the range 9389e23...b65f0be I'm pretty sure it is #46735, but let me bisect anyway...

---

Edit: bisect-rust failed due to can't find crate for `proc_macro` ಠ_ಠ needs to reduce it first.

[alexcrichton]

cc @Manishearth

[kennytm]

Reductions:

1. Reducing dependency to prost_build.

   extern crate prost_build;
   
   struct SG;
   impl prost_build::ServiceGenerator for SG {
       fn generate(&self, _: prost_build::Service, _: &mut String) {
       }
   }
   
   fn main() {
       let mut cfg = prost_build::Config::new();
       cfg.service_generator(Box::new(SG));
       cfg.compile_protos(&["p.proto"], &["."]).unwrap();
   }

2. Removed all dependencies. Repro on playground.

   fn main() {
       let pb_ident = "p.q";
       let mut ident_path = pb_ident.split('.');
       ident_path.next_back();
       let result = ident_path.collect::<Vec<_>>();
       assert_eq!(result, vec!["p"]);
   }

   Edit: Hold on, this program affects beta (1.24.0-beta.1) as well 😱

[kennytm]

Bisecting in the range 9389e23...8e7a609

Test script:

#!/bin/sh
set -eu
$RUSTC_RELATIVE 1.rs
! ./1

(1.rs has content shown in #47175 (comment))

Bisect result:

$ target/release/bisect --preserve --test test.sh --start 9389e23a8a754097e233c7bf3ea1bb404ccf1075 --end 8e7a609e635b728eba65d471c985ab462dc4cfc7
INFO:rust_sysroot: Getting commits from the git checkout in 9389e23a8a754097e233c7bf3ea1bb404ccf1075...8e7a609e635b728eba65d471c985ab462dc4cfc7
INFO:rust_sysroot: Received 23 commits
Searching in 23 commits; about 5 steps
thread 'main' panicked at 'byte index 1747 is not a char boundary; it is inside '\u{0}' (bytes 1746..1747) of `qpassertion failed: `(left == right)`
  left: ``,
 right: ``  1.rscapacity overflowsrc/l`[...]', src/libcore/str/mod.rs:2234:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'byte index 1747 is not a char boundary; it is inside '\u{0}' (bytes 1746..1747) of `qpassertion failed: `(left == right)`
  left: ``,
 right: ``  1.rscapacity overflowsrc/l`[...]', src/libcore/str/mod.rs:2234:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
searched commits 9389e23a8a754097e233c7bf3ea1bb404ccf1075 through 8e7a609e635b728eba65d471c985ab462dc4cfc7
regression in 8; Some(Commit { sha: "b65f0bedd2f22d9661ecb7092f07746dc2ccfb0d", date: 2018-01-01T19:04:33Z, summary: "Auto merge of #46735 - Manishearth:memchr-find, r=burntsushi" })

So yeah the cause is definitely #46735, and since it slips into beta it affects 1.24 beta as well.