runtime: TestGdbPythonCgo is flaky on linux-mips64le-rtrk builder

[bcmills]

2020-03-10T16:25:09-9f74f0a/linux-mips64le-rtrk
2020-02-01T06:01:05-866920a/linux-mips64le-rtrk
2019-11-27T16:41:28-b11d02e/linux-mips64le-rtrk
2019-11-19T02:41:53-2d8c199/linux-mips64le-rtrk
2019-11-15T18:31:19-398f9e1/linux-mips64le-rtrk
2019-11-13T15:52:21-bf49905/linux-mips64le-rtrk
2019-11-12T22:30:48-995ade8/linux-mips64le-rtrk

CC @ianlancetaylor @bogojevic @milanknezevic

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
--- FAIL: TestGdbPythonCgo (6.02s)
    runtime-gdb_test.go:69: gdb version 8.2
    runtime-gdb_test.go:236: gdb output: Loading Go Runtime support.
        Loading Go Runtime support.
        Breakpoint 1 at 0x1200ad4b4: file /tmp/gobuilder-mips64le/tmp/go-build883242937/main.go, line 15.
        Couldn't get registers: No such process.
        BEGIN info goroutines
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 442, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN print mapvar
        No symbol "mapvar" in current context.
        END
        BEGIN print strvar
        No symbol "strvar" in current context.
        END
        BEGIN info locals
        No frame selected.
        END
        BEGIN goroutine 1 bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN goroutine 2 bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        BEGIN goroutine all bt
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 530, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        Breakpoint 2 at 0x1200ad55c: file /tmp/gobuilder-mips64le/tmp/go-build883242937/main.go, line 20.
        Cannot execute this command while the selected thread is running.
        BEGIN goroutine 1 bt at the end
        Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 536, in invoke
            self.invoke_per_goid(goid, cmd)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 539, in invoke_per_goid
            pc, sp = find_goroutine(goid)
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 467, in find_goroutine
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
        END
        
    runtime-gdb_test.go:265: info goroutines failed: Traceback (most recent call last):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 442, in invoke
            for ptr in SliceValue(gdb.parse_and_eval("'runtime.allgs'")):
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 91, in __getitem__
            if i < 0 or i >= self.len:
          File "/tmp/gobuilder-mips64le/go/src/runtime/runtime-gdb.py", line 84, in len
            return int(self.val['len'])
        gdb.MemoryError: Cannot access memory at address 0x120193958
        Error occurred in Python command: Cannot access memory at address 0x120193958
FAIL
FAIL	runtime	177.962s

[bcmills]

See also #35743, #20821.

[bcmills]

2020-09-15T14:51:32-f1c400a/linux-mips64le-rtrk
2020-08-28T20:01:41-54e18f1/linux-mips64le-rtrk
2020-03-25T20:14:03-e8be350/linux-mips64le-rtrk

[bcmills]

greplogs --dashboard -md -l -e '(?ms)FAIL: TestGdbPythonCgo .*Cannot access memory at address' --since=2020-09-17

2022-01-07T18:20:24-ade5488/linux-mips64le-rtrk
2021-12-20T15:13:56-7c94355/linux-mips64le-rtrk
2021-12-07T21:54:30-cf1ec17/linux-mips64le-rtrk
2021-11-13T00:23:16-39bc666/linux-mips64le-rtrk
2021-09-17T21:29:32-3fa7dbe/linux-mips64le-rtrk
2021-05-26T22:43:54-1d5298d/linux-mips64le-rtrk
2021-04-02T01:13:58-41e8a9f/linux-mips64le-rtrk
2021-03-23T05:11:39-e7aa0f9/linux-mips64le-rtrk

[bcmills]

This test is still flaking regularly, creating noise in the dashboard and potentially masking other issues.

@bogojevic, @milanknezevic: is this a known bug for the gdb version installed on this builder? Is the gdb binary current, and if not can it be upgraded to see whether the underlying problem has been fixed?

[bcmills]

There is already a skip on this builder for mipsle and mips64 due to #18784. 😠

[gopherbot]

Change https://golang.org/cl/376654 mentions this issue: runtime: expand TestGdbPythonCgo skip to include mips64le