compiler panic: "byte index 10 is not a char boundary"

[dwrensha]

$ rustc -Vv
rustc 1.36.0 (a53f9df32 2019-07-03)
binary: rustc
commit-hash: a53f9df32fbb0b5f4382caaad8f1a46f36ea887c
commit-date: 2019-07-03
host: x86_64-apple-darwin
release: 1.36.0
LLVM version: 8.0

$ echo Zm4gbWFpbigo2Lw= | base64 -D > main.rs

$ rustc main.rs
error: this file contains an un-closed delimiter
 --> main.rs:1:11
  |
1 | fn main((ؼ
  |        -- ^
  |        ||
  |        |un-closed delimiter
  |        un-closed delimiter

thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/libcore/str/mod.rs:2034:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: aborting due to previous error


error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.36.0 (a53f9df32 2019-07-03) running on x86_64-apple-darwin

Found with the help of libfuzzer-sys.

[ExpHP]

  12: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}
  13: syntax::source_map::SourceMap::find_width_of_character_at_span
  14: syntax::source_map::SourceMap::next_point
  15: syntax::parse::diagnostics::<impl syntax::parse::parser::Parser>::unexpected_try_recover
  16: syntax::parse::parser::Parser::parse_fn_args::{{closure}}
  17: syntax::parse::parser::Parser::parse_fn_args
  18: syntax::parse::parser::Parser::parse_fn_decl
  19: syntax::parse::parser::Parser::parse_item_fn
  20: syntax::parse::parser::Parser::parse_item_implementation
  21: syntax::parse::parser::Parser::parse_item_

There's two slicing operations in find_width_of_character_at_span. I'm trying to figure out how a non-boundary index could be appearing in there... maybe unexpected_try_recover is calling next_point with a span that begins at index 10? (where does this come from? Parser.prev_span?) Building a debug compiler to check those log statements...

Edit: Building the debug compiler was a bust. Embarassingly, things seem to have changed and I don't know how to get those debug macros to fire in the current compiler.

[pnkfelix]

@ExpHP do you mean you were using RUST_LOG and not seeing debug output?

If so, that is because the environment variable name under rustc was changed to RUSTC_LOG. (I make this mistake pretty much every day due to muscle memory.)

---

As an example, here is the tail of my log output:

% RUSTC_LOG=syntax::parse,syntax::source_map ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc  /tmp/off_index.rs
DEBUG 2019-07-11T12:58:22Z: syntax::parse::attr: parse_outer_attributes: self.token=Token { kind: OpenDelim(Paren), spa\
n: Span { lo: BytePos(8), hi: BytePos(9), ctxt: #0 } }
DEBUG 2019-07-11T12:58:22Z: syntax::parse::parser: parse_arg_general parse_pat (is_name_required:true)
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: local_begin=`SourceFileAndBytePos { sf\
: SourceFile(/tmp/off_index.rs), pos: BytePos(10) }`, local_end=`SourceFileAndBytePos { sf: SourceFile(/tmp/off_index.r\
s), pos: BytePos(11) }`
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: start_index=`10`, end_index=`11`
DEBUG 2019-07-11T12:58:22Z: syntax::source_map: find_width_of_character_at_span: source_len=`11`
thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/\
libcore/str/mod.rs:2039:5

[pnkfelix]

triage: P-medium. Removing nomination.

[pnkfelix]

This was broken between 1.15 and 1.16, but those are so old and the nature of the ICE itself has changed from 1.28 to 1.29, so I do not think bisection would be worthwhile.

[ExpHP]

Offtopic

@pnkfelix : Thanks. Unfortunately, I did guess the name of RUSTC_LOG after RUST_LOG didn't work, and all it showed me was this:

$ RUSTC_LOG=trace rustc +stage1 src/main.rs
 INFO 2019-07-11T13:33:39Z: jobserver::imp: created a jobserver: Client { read: File { fd: 3, path: "pipe:[2538076]", read: true, write: false }, write: File { fd: 4, path: "pipe:[2538076]", read: false, write: true } }
 INFO 2019-07-11T13:33:39Z: rustc_interface::util: codegen backend candidate: /home/lampam/asd/clone/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends
 INFO 2019-07-11T13:33:39Z: rustc_interface::util: probing /home/lampam/asd/clone/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends for a codegen backend
error: this file contains an un-closed delimiter
 --> src/main.rs:1:11
  |
1 | fn main((ؼ
  |        -- ^
  |        ||
  |        |un-closed delimiter
  |        un-closed delimiter

thread 'rustc' panicked at 'byte index 10 is not a char boundary; it is inside 'ؼ' (bytes 9..11) of `fn main((ؼ`', src/libcore/str/mod.rs:2039:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: aborting due to previous error

I set debug = true in my config.toml before building, and didn't see anything else in there. (I recall that debug! and trace! used to be explicitly mentioned in the comment for debug-assertions but it is no longer, so I did not enable it.)

[ExpHP]

Okay, I did the following evil, evil thing:

Pure evil, do not open

diff --git a/src/libsyntax_pos/span_encoding.rs b/src/libsyntax_pos/span_encoding.rs
index 525ec13623..d7da2206ab 100644
--- a/src/libsyntax_pos/span_encoding.rs
+++ b/src/libsyntax_pos/span_encoding.rs
@@ -74,6 +74,12 @@ pub const DUMMY_SP: Span = Span { base_or_index: 0, len_or_tag: 0, ctxt_or_zero:
 impl Span {
     #[inline]
     pub fn new(mut lo: BytePos, mut hi: BytePos, ctxt: SyntaxContext) -> Self {
+        if lo == BytePos(10) || hi == BytePos(10) {
+            eprintln!("=========================");
+            eprintln!("{:?}", (lo, hi));
+            eprintln!();
+            let _ = std::panic::catch_unwind(|| panic!()); // potentially print backtrace
+        }
         if lo > hi {
             std::mem::swap(&mut lo, &mut hi);
         }

And acquired the following backtrace of when the span was first constructed:

=========================
(BytePos(10), BytePos(11))

  15: syntax_pos::span_encoding::Span::new at /home/lampam/asd/clone/rust/src/libsyntax_pos/span_encoding.rs:81
  16: syntax_pos::SpanData::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:218
  17: syntax_pos::<impl syntax_pos::span_encoding::Span>::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:268
  18: syntax::tokenstream::TokenTree::close_tt at src/libsyntax/tokenstream.rs:141
  19: syntax::parse::parser::TokenCursor::next at src/libsyntax/parse/parser.rs:315
  20: syntax::parse::parser::Parser::next_tok at src/libsyntax/parse/parser.rs:528
  21: syntax::parse::parser::Parser::bump at src/libsyntax/parse/parser.rs:1031
  22: syntax::parse::parser::Parser::parse_ident_common at src/libsyntax/parse/parser.rs:632
  23: syntax::parse::parser::Parser::parse_ident at src/libsyntax/parse/parser.rs:617
  24: syntax::parse::parser::Parser::parse_pat_ident at src/libsyntax/parse/parser.rs:4132
  25: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3987
  26: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  27: syntax::parse::parser::Parser::parse_pat_list at src/libsyntax/parse/parser.rs:3582
  28: syntax::parse::parser::Parser::parse_parenthesized_pat_list at src/libsyntax/parse/parser.rs:3547
  29: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3938
  30: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  31: syntax::parse::parser::Parser::parse_arg_general at src/libsyntax/parse/parser.rs:1510
  32: syntax::parse::parser::Parser::parse_fn_args::{{closure}} at src/libsyntax/parse/parser.rs:5380
  33: syntax::parse::parser::Parser::parse_seq_to_before_tokens at src/libsyntax/parse/parser.rs:983
  34: syntax::parse::parser::Parser::parse_seq_to_before_end at src/libsyntax/parse/parser.rs:916
  35: syntax::parse::parser::Parser::parse_fn_args at src/libsyntax/parse/parser.rs:5368
  36: syntax::parse::parser::Parser::parse_fn_decl at src/libsyntax/parse/parser.rs:5429
  37: syntax::parse::parser::Parser::parse_item_fn at src/libsyntax/parse/parser.rs:5660

It also gets constructed a second time:

Second backtrace

=========================
(BytePos(10), BytePos(11))

  15: syntax_pos::span_encoding::Span::new at /home/lampam/asd/clone/rust/src/libsyntax_pos/span_encoding.rs:81
  16: syntax_pos::SpanData::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:218
  17: syntax_pos::<impl syntax_pos::span_encoding::Span>::with_lo at /home/lampam/asd/clone/rust/src/libsyntax_pos/lib.rs:268
  18: syntax::tokenstream::TokenTree::close_tt at src/libsyntax/tokenstream.rs:141
  19: syntax::parse::parser::TokenCursor::next at src/libsyntax/parse/parser.rs:315
  20: syntax::parse::parser::Parser::next_tok at src/libsyntax/parse/parser.rs:528
  21: syntax::parse::parser::Parser::bump at src/libsyntax/parse/parser.rs:1031
  22: syntax::parse::parser::Parser::expect_one_of at src/libsyntax/parse/parser.rs:590
  23: syntax::parse::parser::Parser::expect at src/libsyntax/parse/parser.rs:577
  24: syntax::parse::parser::Parser::parse_parenthesized_pat_list at src/libsyntax/parse/parser.rs:3555
  25: syntax::parse::parser::Parser::parse_pat_with_range_pat at src/libsyntax/parse/parser.rs:3938
  26: syntax::parse::parser::Parser::parse_pat at src/libsyntax/parse/parser.rs:3908
  27: syntax::parse::parser::Parser::parse_arg_general at src/libsyntax/parse/parser.rs:1510
  28: syntax::parse::parser::Parser::parse_fn_args::{{closure}} at src/libsyntax/parse/parser.rs:5380
  29: syntax::parse::parser::Parser::parse_seq_to_before_tokens at src/libsyntax/parse/parser.rs:983
  30: syntax::parse::parser::Parser::parse_seq_to_before_end at src/libsyntax/parse/parser.rs:916
  31: syntax::parse::parser::Parser::parse_fn_args at src/libsyntax/parse/parser.rs:5368
  32: syntax::parse::parser::Parser::parse_fn_decl at src/libsyntax/parse/parser.rs:5429
  33: syntax::parse::parser::Parser::parse_item_fn at src/libsyntax/parse/parser.rs:5660

(the region of the backtrace that differs between the two is items 22-29 in the first one, or 22-25 in the second)

---

Some interesting spots from here:

This is clearly where the value of 10 is created:

rust/src/libsyntax/tokenstream.rs

Lines 136 to 144 in 4bb6b4a

	/// Returns the closing delimiter as a token tree.
	pub fn close_tt(span: Span, delim: DelimToken) -> TokenTree {
	let close_span = if span.is_dummy() {
	span
	} else {
	span.with_lo(span.hi() - BytePos(delim.len() as u32))
	};
	TokenTree::token(token::CloseDelim(delim), close_span)
	}

TokenCursor::next calls this if there are no remaining tokens and it still hasn't seen the closing delimiter (it seems to have as an unstated precondition that the closing delimiter must exist).

Here's the innermost bit that's specific to ident parsing. Doesn't look that odd...

rust/src/libsyntax/parse/parser.rs

Lines 622 to 634 in 4bb6b4a

	token::Ident(name, _) => {
	if self.token.is_reserved_ident() {
	let mut err = self.expected_ident_found();
	if recover {
	err.emit();
	} else {
	return Err(err);
	}
	}
	let span = self.token.span;
	self.bump();
	Ok(Ident::new(name, span))
	}

[pnkfelix]

@ExpHP I’m not familiar with RUSTC_LOG=trace

I tend to list (comma separated) module paths in my own use of RUSTC_LOG; did you try that? You can see an example in the transcript I gave in my comment above