Skip to content

--rerun-fails reruns all tests in packages if packages are also passed to go test args #417

Open
@matthewhughes-uw

Description

That is, if you run gotestsum --rerun-fails --packages ./... -- ../.. then on failure all packages matching ./... will be re-tested. I don't think this is necessarily a bug, but might warrant some more explicit documentation

Reproduction:

$ go mod init module
$ mkdir --parents pkg/foo pkg/bar
// pkg/foo/foo.go
package foo

func BoringFunc() int {
	return 3
}
// pkg/foo/foo_test.go
package foo

import (
	"testing"
)

func TestBoringFunc(t *testing.T) {
	if BoringFunc() != 3 {
		t.Fatal("wrong")
	}
}
// pkg/bar/bar.go
package bar

func BoringFunc() int {
	return 4
}
// pkg/bar/bar_test.go
package bar

import (
	"testing"
)

func TestBoringFunc(t *testing.T) {
	if BoringFunc() != 3 {
		t.Fatal("wrong")
	}
}

running gotestsum --rerun-fails --packages ./... re-runs only the failing test, as expected:

$ gotestsum --rerun-fails --packages ./... -- -count=1
✓  pkg/foo (2ms)
✖  pkg/bar (2ms)

DONE 2 tests, 1 failure in 0.188s

✖  pkg/bar (2ms)

DONE 2 runs, 3 tests, 2 failures in 0.371s

✖  pkg/bar (2ms)

=== Failed
=== FAIL: pkg/bar TestBoringFunc (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 1) (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 2) (0.00s)
    race_test.go:9: wrong

DONE 3 runs, 4 tests, 3 failures in 0.539s

However, when run with a list of packages passed as args, all tests are re-run:

$ gotestsum --rerun-fails --packages ./... -- -count=1 ./...
✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

DONE 2 tests, 1 failure in 0.209s

✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

DONE 2 runs, 4 tests, 2 failures in 0.395s

✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

=== Failed
=== FAIL: pkg/bar TestBoringFunc (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 1) (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 2) (0.00s)
    race_test.go:9: wrong

DONE 3 runs, 6 tests, 3 failures in 0.556s

Maybe it's worth updating the docs to mention how packages should (or shouldn't) be passed when using --rerun-fails?

Further, I guess gotestsum could try and determine if the user passed both --packages and a list of packages as args to go test and do something then? But I think an update to the docs would be enough.

I ran into this behaviour in a private repo at my work, but here's another example I found in the wild: https://github.com/hashicorp/nomad/blob/54aafa574d2dc4bd281b28f45dc00593c1e337da/GNUmakefile#L287

Activity

added
bugSomething isn't working
docsUpdate or add documentation
on Aug 8, 2024
dnephin

dnephin commented on Aug 8, 2024

@dnephin
Member

Thank you for the bug report! I wonder if some are using this as a way to re-run everything on any failure. I do something similar occasionally by specifying extra packages to gotestsum --watch.

I think a warning log message and a docs update would be great!

matthewhughes934

matthewhughes934 commented on Oct 4, 2024

@matthewhughes934

I think a warning log message and a docs update would be great!

Looking into this, is there an existing solution to seeing if a list of packages was passed to go test? For example:

  • gotestsum --rerun-fails --packages ./... -- ./... should emit a warning
  • gotestsum --rerun-fails --packages ./... -- -coverprofile 'some.file' shouldn't emit a warning
added a commit that references this issue on Nov 7, 2024
added a commit that references this issue on Dec 4, 2024
added a commit that references this issue on Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocsUpdate or add documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      `--rerun-fails` reruns all tests in packages if packages are also passed to `go test` args · Issue #417 · gotestyourself/gotestsum