Test iterations vary based on the -short flag

[armfazh]

Not an issue that needs to be addressed right now, but it can be quite nice for things like this is to allow the iterations to vary based on the -short flag. For example:

func NumTrials() {
    if testing.Short() {
        return 16
    }
    return 1 << 8 // or more
}

If you want to get even more fancy, something I've done before is to also define custom -long and -stress flags. So you can do:

// Custom test command line flags.
var (
	long   = flag.Bool("long", false, "enable long running tests")
	stress = flag.Bool("stress", false, "enable stress tests (implies -long)")
)

// timeallowed returns how long a single test is allowed to take.
func timeallowed() time.Duration {
	switch {
	case testing.Short():
		return time.Second / 10
	case *long:
		return 30 * time.Second
	case *stress:
		return 2 * time.Minute
	default:
		return time.Second
	}
}

// Repeat the given trial function. The duration is controlled by custom
// command-line flags. The trial function returns whether it wants to continue
// testing.
//
//	-short		run for less time than usual
//	-long		allow more time
//	-stress		run for an extremely long time
func Repeat(t *testing.T, trial func(t *testing.T) bool) {
	start := time.Now()
	d := timeallowed()
	n := 1
	for time.Since(start) < d && trial(t) {
		n++
	}
	t.Logf("%d trials in %s", n, time.Since(start))
}

I usually put something like this in an internal/test package. Then you wrap your test in a test.Repeat(...) call and it will run until the time limit.

Originally posted by @mmcloughlin in #102