testing: complain loudly during concurrent use of T.FatalX and T.SkipX

[dsnet]

The testing package is explicit about the following:

A test ends when its Test function returns or calls any of the methods FailNow, Fatal, Fatalf, SkipNow, Skip, or Skipf. Those methods, as well as the Parallel method, must be called only from the goroutine running the Test function.

However, this is easy to overlook and it is not immediately obvious that the code below is a bad test:

func TestFoo(t *testing.T) {
    go func() {
        t.Fatal("fatal") // Called from outside the Test function
    }()
    time.Sleep(1 * time.Second)
}

This currently outputs (what a user expects):

--- FAIL: TestFoo (1.00s)
    foo_test.go:10: fatal

However, since t.Fatal is not safe to call outside of the Test function, this is poor feedback since it makes the user think that the test is correctly written when it is actually racy. Instead it should complain loudly that this is wrong.

As I'm debugging poor tests, I've noticed that the calling of t.Fatal in a goroutine is actually a fairly common phenomenon.

Related: #15674

/cc @mpvl @bradfitz

[rsc]

Seems infeasible. FailNow/SkipNow are fundamentally built around runtime.Goexit, and there is no way to tell whether they are being called from the "right" goroutine. Bad API design maybe, but it's the way it is.

We can't change from runtime.Goexit to a panic in general; the whole point of using Goexit was that it does not look like a panic and cannot be stopped.

I don't see any way to change the current behavior nor do I see any way to make the current behavior louder.

[bradfitz]

@rsc, the http2 package has a testing-time-only mechanism to track that certain methods only run on the expected goroutines.

See https://github.com/golang/net/blob/master/http2/gotrack.go#L22

We could do something like that.

[rsc]

I've been wondering if we should take the goroutine ID out of the runtime.Stack output, precisely for this reason. :-(

[bradfitz]

@rsc, that'd be fine, if you give me an as-useful debugging primitive elsewhere.

[ianlancetaylor]

Reopening as @balshetzer has an approach that may work, based on looking for certain functions in the stack trace. I guess we can give it a try and see if it seems reasonable.