runtime: TestGdbPython flaky on linux

[bradfitz]

Just saw this TestGdbPython flake on linux-amd64:

https://storage.googleapis.com/go-build-log/149e81d1/linux-amd64_fd28318f.log

--- FAIL: TestGdbPython (3.49s)
	runtime-gdb_test.go:59: gdb version 7.7
	runtime-gdb_test.go:193: gdb output: Loading Go Runtime support.
		Loaded  Script                                                                 
		Yes     /tmp/workdir/go/src/runtime/runtime-gdb.py                             
		Breakpoint 1 at 0x47c190: file /tmp/workdir/go/src/fmt/print.go, line 263.
		
		Breakpoint 1, fmt.Println (a=..., err=..., n=<optimized out>) at /tmp/workdir/go/src/fmt/print.go:263
		263	func Println(a ...interface{}) (n int, err error) {
		BEGIN info goroutines
		* 1 running  runtime.systemstack_switch
		* 2 running  runtime.forcegchelper
		  3 waiting  runtime.gopark
		  4 runnable runtime.runfinq
		END
		#1  0x00000000004828a0 in main.main () at /tmp/go-build461513624/main.go:14
		14		fmt.Println("hi")
		BEGIN print mapvar
		$1 = map[string]string = {["ghi"] = "jkl", ["abc"] = "def"}
		END
		BEGIN print strvar
		$2 = "abc"
		END
		BEGIN info locals
		mapvar = map[string]string = {["ghi"] = "jkl", ["abc"] = "def"}
		slicevar =  []string = {"def"}
		strvar = "abc"
		END
		#0  fmt.Println (a=..., err=..., n=<optimized out>) at /tmp/workdir/go/src/fmt/print.go:263
		263	func Println(a ...interface{}) (n int, err error) {
		BEGIN goroutine 1 bt
		#0  fmt.Println (a=..., err=..., n=<optimized out>) at /tmp/workdir/go/src/fmt/print.go:263
		#1  0x00000000004828a0 in main.main () at /tmp/go-build461513624/main.go:14
		END
		BEGIN goroutine 2 bt
		No such goroutine:  2
		END
		Breakpoint 2 at 0x4828cd: file /tmp/go-build461513624/main.go, line 18.
		hi
		
		Breakpoint 2, main.main () at /tmp/go-build461513624/main.go:19
		19	}  // END_OF_PROGRAM
		BEGIN goroutine 1 bt at the end
		#0  main.main () at /tmp/go-build461513624/main.go:19
		END
		
	runtime-gdb_test.go:258: goroutine 2 bt failed: No such goroutine:  2
FAIL
FAIL	runtime	28.524s

/cc @aclements

[aclements]

@hyangah, is this similar to the bug you fixed recently about getting the state of goroutines?

[hyangah]

@aclements do you mean https://go-review.googlesource.com/c/go/+/49691?
I don't know. Maybe.

If when the test passes the gdb output should look like the following

--- PASS: TestGdbPython (0.44s)
	runtime-gdb_test.go:59: gdb version 7.7
	runtime-gdb_test.go:193: gdb output: Loading Go Runtime support.
		Loaded  Script                                                                 
		Yes     /tmp/workdir/go/src/runtime/runtime-gdb.py                             
		Breakpoint 1 at 0x47c1c0: file /tmp/workdir/go/src/fmt/print.go, line 263.
		
		Breakpoint 1, fmt.Println (a=..., err=..., n=) at /tmp/workdir/go/src/fmt/print.go:263
		263	func Println(a ...interface{}) (n int, err error) {
		BEGIN info goroutines
		* 1 running  runtime.systemstack_switch
		  2 waiting  runtime.gopark
		  17 waiting  runtime.gopark
		  33 runnable runtime.runfinq
		END
		#1  0x00000000004828d0 in main.main () at /tmp/go-build668958384/main.go:14
		14		fmt.Println("hi")
		BEGIN print mapvar
		$1 = map[string]string = {["abc"] = "def", ["ghi"] = "jkl"}
		END
		BEGIN print strvar
		$2 = "abc"
		END
		BEGIN info locals
		mapvar = map[string]string = {["abc"] = "def", ["ghi"] = "jkl"}
		slicevar =  []string = {"def"}
		strvar = "abc"
		END
		#0  fmt.Println (a=..., err=..., n=) at /tmp/workdir/go/src/fmt/print.go:263
		263	func Println(a ...interface{}) (n int, err error) {
		BEGIN goroutine 1 bt
		#0  fmt.Println (a=..., err=..., n=) at /tmp/workdir/go/src/fmt/print.go:263
		#1  0x00000000004828d0 in main.main () at /tmp/go-build668958384/main.go:14
		END
		BEGIN goroutine 2 bt
		#0  runtime.gopark (lock=0x528af0 , reason="force gc (idle)", traceEv=20 '\024', traceskip=1, unlockf=) at /tmp/workdir/go/src/runtime/proc.go:292
		#1  0x0000000000428c5e in runtime.goparkunlock (lock=, reason=..., traceEv=, traceskip=) at /tmp/workdir/go/src/runtime/proc.go:297
		#2  0x00000000004289ea in runtime.forcegchelper () at /tmp/workdir/go/src/runtime/proc.go:248
		#3  0x000000000044ee01 in runtime.goexit () at /tmp/workdir/go/src/runtime/asm_amd64.s:1385
		#4  0x0000000000000000 in ?? ()
		END
		Breakpoint 2 at 0x4828fd: file /tmp/go-build668958384/main.go, line 18.
		hi
		
		Breakpoint 2, main.main () at /tmp/go-build668958384/main.go:19
		19	}  // END_OF_PROGRAM
		BEGIN goroutine 1 bt at the end
		#0  main.main () at /tmp/go-build668958384/main.go:19
		END

Note the difference in the output of 'info goroutine' (2 running goroutines vs 1 running goroutine).

Is there any way to reliably reproduce the failure case?
I tried to gomote and test in the linux-amd64 buildlet, but failed to reproduce the failure with -count=1000 (too ~450s). Not from my linux either. I tried larger than 1000 in gomote and the run was SIGQUIT.

[bcmills]

Here's a flake with very similar output on linux-ppc64le-buildlet:
https://build.golang.org/log/f5c124d74c9e6a71da5614b8b13db9328ec08910