debuginfo-gdb tests fail: <error reading variable>

[canndrew]

This is really annoying because I haven't done a successful ./x.py test in months and it's causing me to submit shitty broken PRs.

I'm on NixOS with gdb-7.12.1. When running the test suite I get (with some cruft omitted):

---- [debuginfo-gdb] debuginfo-gdb/borrowed-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $3 = borrowed_enum::Univariant::TheOnlyCase(4820353753753434)
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/borrowed-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa1a: file /home/shum/src/rust/rust-a/src/test/debuginfo/borrowed-enum.rs, line 81.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, borrowed_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/borrowed-enum.rs:81
81	    zzz(); // #break
$1 = borrowed_enum::ABC::TheA{x: 0, y: 8970181431921507452}
$2 = borrowed_enum::ABC::TheB(0, 286331153, 286331153)
$3 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 30521] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/borrowed-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- [debuginfo-gdb] debuginfo-gdb/generic-struct-style-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $4 = generic_struct_style_enum::Univariant<i32>::TheOnlyCase{a: -1}
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/generic-struct-style-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa23: file /home/shum/src/rust/rust-a/src/test/debuginfo/generic-struct-style-enum.rs, line 86.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, generic_struct_style_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/generic-struct-style-enum.rs:86
86	    zzz(); // #break
$1 = generic_struct_style_enum::Regular::Case1{a: 0, b: 31868, c: 31868, d: 31868, e: 31868}
$2 = generic_struct_style_enum::Regular::Case2{a: 0, b: 286331153, c: 286331153}
$3 = generic_struct_style_enum::Regular::Case3{a: 0, b: 6438275382588823897}
$4 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 31117] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/generic-struct-style-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/generic-tuple-style-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $4 = generic_tuple_style_enum::Univariant<i64>::TheOnlyCase(-1)
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/generic-tuple-style-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa24: file /home/shum/src/rust/rust-a/src/test/debuginfo/generic-tuple-style-enum.rs, line 104.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, generic_tuple_style_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/generic-tuple-style-enum.rs:104
104	    zzz(); // #break
$1 = generic_tuple_style_enum::Regular::Case1(0, 31868, 31868, 31868, 31868)
$2 = generic_tuple_style_enum::Regular::Case2(0, 286331153, 286331153)
$3 = generic_tuple_style_enum::Regular::Case3(0, 6438275382588823897)
$4 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 31186] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/generic-tuple-style-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/packed-struct.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $5 = 14
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/packed-struct.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xb7d: file /home/shum/src/rust/rust-a/src/test/debuginfo/packed-struct.rs, line 134.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, packed_struct::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/packed-struct.rs:134
134	    zzz(); // #break
$1 = packed_struct::Packed {x: 123, y: 234, z: 345}
$2 = packed_struct::PackedInPacked {a: 1111, b: packed_struct::Packed {x: 2222, y: 3333, z: 4444}, c: 5555, d: packed_struct::Packed {x: 6666, y: 7777, z: 8888}}
$3 = packed_struct::PackedInUnpacked {a: -1111, b: packed_struct::Packed {x: -2222, y: -3333, z: -4444}, c: -5555, d: packed_struct::Packed {x: -6666, y: -7777, z: -8888}}
$4 = packed_struct::UnpackedInPacked {a: 987, b: packed_struct::Unpacked {x: 876, y: 765, z: 654, w: 543}, c: packed_struct::Unpacked {x: 432, y: 321, z: 210, w: 109}, d: -98}

------------------------------------------
stderr:
------------------------------------------
/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/packed-struct.debugger.script:13: Error in sourced command file:
No symbol 'sizeof' in current context

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/packed-struct.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/recursive-struct.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $2 = 1
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/recursive-struct.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0x225b: file /home/shum/src/rust/rust-a/src/test/debuginfo/recursive-struct.rs, line 235.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, recursive_struct::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/recursive-struct.rs:235
235	    zzz(); // #break
$1 = 0

------------------------------------------
stderr:
------------------------------------------
/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/recursive-struct.debugger.script:10: Error in sourced command file:
Attempting to access named field val of tuple variant recursive_struct::Opt<alloc::boxed::Box<recursive_struct::UniqueNode<u16>>>::Val, which has only anonymous fields

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/recursive-struct.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/struct-in-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $3 = struct_in_enum::Univariant::TheOnlyCase(struct_in_enum::Struct {x: 123, y: 456, z: 789})
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/struct-in-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa33: file /home/shum/src/rust/rust-a/src/test/debuginfo/struct-in-enum.rs, line 92.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, struct_in_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/struct-in-enum.rs:92
92	    zzz(); // #break
$1 = struct_in_enum::Regular::Case1(0, struct_in_enum::Struct {x: 2088533116, y: 2088533116, z: 31868})
$2 = struct_in_enum::Regular::Case2(0, 1229782938247303441, 4369)
$3 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 32359] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/struct-in-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/struct-style-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $4 = struct_style_enum::Univariant::TheOnlyCase{a: -1}
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/struct-style-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa24: file /home/shum/src/rust/rust-a/src/test/debuginfo/struct-style-enum.rs, line 101.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, struct_style_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/struct-style-enum.rs:101
101	    zzz(); // #break
$1 = struct_style_enum::Regular::Case1{a: 0, b: 31868, c: 31868, d: 31868, e: 31868}
$2 = struct_style_enum::Regular::Case2{a: 0, b: 286331153, c: 286331153}
$3 = struct_style_enum::Regular::Case3{a: 0, b: 6438275382588823897}
$4 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 32437] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/struct-style-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/tuple-style-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $4 = tuple_style_enum::Univariant::TheOnlyCase(-1)
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/tuple-style-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xa24: file /home/shum/src/rust/rust-a/src/test/debuginfo/tuple-style-enum.rs, line 101.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, tuple_style_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/tuple-style-enum.rs:101
101	    zzz(); // #break
$1 = tuple_style_enum::Regular::Case1(0, 31868, 31868, 31868, 31868)
$2 = tuple_style_enum::Regular::Case2(0, 286331153, 286331153)
$3 = tuple_style_enum::Regular::Case3(0, 6438275382588823897)
$4 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 32555] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/tuple-style-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/unique-enum.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $3 = unique_enum::Univariant::TheOnlyCase(123234)
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/unique-enum.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0xe90: file /home/shum/src/rust/rust-a/src/test/debuginfo/unique-enum.rs, line 85.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, unique_enum::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/unique-enum.rs:85
85	    zzz(); // #break
$1 = unique_enum::ABC::TheA{x: 0, y: 8970181431921507452}
$2 = unique_enum::ABC::TheB(0, 286331153, 286331153)
$3 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 32629] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/unique-enum.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474

---- [debuginfo-gdb] debuginfo-gdb/union-smoke.rs stdout ----
	NOTE: compiletest thinks it is using GDB with native rust support
NOTE: compiletest thinks it is using GDB version 7012001

error: line not found in debugger output: $1 = union_smoke::U {a: (2, 2), b: 514}
status: exit code: 0
command: /run/current-system/sw/bin/gdb -quiet -batch -nx -command=/home/shum/src/rust/rust-a/build/x86_64-unknown-linux-gnu/test/debuginfo/union-smoke.debugger.script
stdout:
------------------------------------------
...
Breakpoint 1 at 0x9dc: file /home/shum/src/rust/rust-a/src/test/debuginfo/union-smoke.rs, line 50.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/kk71vkqipf30qc165718jmp0s8cggn2y-glibc-2.24/lib/libthread_db.so.1".

Breakpoint 1, union_smoke::main () at /home/shum/src/rust/rust-a/src/test/debuginfo/union-smoke.rs:50
50	    zzz(); // #break
$1 = <error reading variable>
$2 = <error reading variable>
A debugging session is active.

	Inferior 1 [process 307] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

------------------------------------------
stderr:
------------------------------------------

------------------------------------------

thread '[debuginfo-gdb] debuginfo-gdb/union-smoke.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2474


failures:
    [debuginfo-gdb] debuginfo-gdb/borrowed-enum.rs
    [debuginfo-gdb] debuginfo-gdb/generic-struct-style-enum.rs
    [debuginfo-gdb] debuginfo-gdb/generic-tuple-style-enum.rs
    [debuginfo-gdb] debuginfo-gdb/packed-struct.rs
    [debuginfo-gdb] debuginfo-gdb/recursive-struct.rs
    [debuginfo-gdb] debuginfo-gdb/struct-in-enum.rs
    [debuginfo-gdb] debuginfo-gdb/struct-style-enum.rs
    [debuginfo-gdb] debuginfo-gdb/tuple-style-enum.rs
    [debuginfo-gdb] debuginfo-gdb/union-smoke.rs
    [debuginfo-gdb] debuginfo-gdb/unique-enum.rs

[petrochenkov]

In case this is not going to be fixed - after test caching lands, these tests will be skippable by touching their timestamp files.

[TimNN]

The problem here is that when gdb gained native rust support, there were still some bugs in that support. When the debug info tests got updated for gdb with native rust support, those bugs were fixed, however this means that there are some versions of gdb which still exhibit those bugs, meaning some tests will fail.

These failing tests were meant to be ignored for buggy gdb versions in #39039, that only gdb up to 7.12 was ignored. The best fix would probably be to update the list of ignored versions to include 7.12.1 (I'll send a PR for that).

[amboar]

On this, I ran into the issue while doing testing on powerpc64le and debugged it in parallel to this issue getting resolved. I've installed gdb from the tip of binutils-gdb master, which is reporting its version as 7.12.50.DATE, and the tests are still being ignored.

$ /usr/local/bin/gdb --version
GNU gdb (GDB) 7.12.50.20170213-git

Invoking ./x.py test src/test/debuginfo still uses /usr/bin/gdb despite /usr/local/bin/gdb being first in PATH, but invoking the test suite directly to specify /usr/local/bin/gdb still gives 'ignored' results:

"/home/arj/rust/build/powerpc64le-unknown-linux-gnu/stage0-tools/powerpc64le-unknown-linux-gnu/debug/compiletest" "--compile-lib-path" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/stage2/lib/rustlib/powerpc64le-unknown-linux-gnu/lib" "--rustc-path" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/home/arj/rust/src/test/debuginfo" "--build-base" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/test/debuginfo" "--stage-id" "stage2-powerpc64le-unknown-linux-gnu" "--mode" "debuginfo-gdb" "--target" "powerpc64le-unknown-linux-gnu" "--host" "powerpc64le-unknown-linux-gnu" "--llvm-filecheck" "/home/arj/rust/build/powerpc64le-unknown-linux-gnu/llvm/build/bin/FileCheck" "--host-rustcflags" "-Crpath -O" "--target-rustcflags" "-Crpath -O -Lnative=/home/arj/rust/build/powerpc64le-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/local/bin/gdb" "--llvm-version" "3.9.1\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp" "--android-cross-path" ""
...
test result: FAILED. 0 passed; 1 failed; 106 ignored; 0 measured

From the commit we should be only ignoring versions 7.11.90-7.12.9, so it seems something is still wrong.

Further, the test suite makes gdb source the in-tree rust pretty printers. It seems that with a version of Rust prior to this fix to ignore "broken" gdb versions we can make the tests pass by breaking the import of the in-tree pretty printer scripts:

mv src/etc/debugger_pretty_printers_common.py{,.backup} &&
    touch src/etc/debugger_pretty_printers_common.py &&
    ./x.py test src/test/debuginfo

Given the rust support in gdb is tested based on gdb's version and not by gdb's output, this says to me that the gdbr-check lines are expecting the output that the "broken" gdb Rust support is giving us. That is, it appears that the test suite annotations and the in-tree pretty-printer scripts are out of sync.

I didn't dive too deeply into the pretty-printer scripts so maybe there's more to the story.

[amboar]

I've pushed up a commit at amboar/rust@67bffe3 that "fixes" the expectations. The patch was generated by a script, and the script is included in the commit message so people can roughly reproduce. I had to switch between enabling and disabling the --ignored flag on compiletest to generate the full patch. A brief inspection of the diff showed a handful of changes that will need review, but for the most part the values themselves haven't been touched, just the formatting of the output.

[TimNN]

I'm no expert when it comes to the debuginfo tests, however:

• The ignore version stuff should at most ignore ~16 Test or so, most definitely not all of them.
• As far as I know, the (gdb) debuginfo tests are run without the python pretty printer.

Can you post a more complete log? I'm especially interested in the output of the failed test.

[amboar]

@TimNN yeah something's pretty fishy here, I think I'll start a new issue rather than hijack this one any further.