Description
In #41824 (comment), @rsc wrote:
The fact that CI systems can today opt in to running 1 iteration of benchmarks (using -benchtime=1x) but do not seems to me strong evidence that the Go command should not start doing it either.
…
As has been pointed out, CI systems and users have the ability to opt in even today, using -benchtime=1x. You could even set GOFLAGS=-benchtime=1x to force it into all your go test commands.
As an experiment I tried doing this for my own day-to-day work, and it was awful. It turns out that the -benchtime=1x causes test results not to be cached, which means that GOFLAGS=-benchtime=1x causes every test to be re-run whenever go test is invoked, even if the source files haven't changed and even if the tests contain no actual benchmarks.
Invalidating the test cache is probably the right behavior when -benchtime is actually being used for benchmarking but is entirely inappropriate when the reason for setting the -benchtime flag is to test that the benchmarks continue to run successfully (which was the point of #41824 to begin with).
This issue tracks the need for “testing that benchmarks still work” without also invalidating cache results.