Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--rerun-fails always makes goleak and datarace errors pass after re-running #442

Open
odubajDT opened this issue Sep 23, 2024 · 6 comments

Comments

@odubajDT
Copy link

Description

When using --rerun-fails=1 option with gotestsum, all of the tests pass. The behavior is very deterministic. Instead when removing the option, I am getting a lot of goleak and data-race errors. It's quite interesting that the behavior with re-running the tests is very deterministic, as I would expect from a big number of tests and multiple runs, it should fail at least once also in the re-run. It seems to me that in the 2. run, the gotestsum just ignores the errors which popped up during the first run

@dnephin
Copy link
Member

dnephin commented Sep 25, 2024

Hello, thank you for the bug report! Can you try with --debug? That will print the exact go test command that is run.

The re-runs are done one test at a time because of some limitations with the go test -run flag. Those limitations have since been fixed I believe, so it's possible we can switch to running one package at a time instead.

The tests may be passing because the data-race is due to a goroutine that is leaked from a previous test. When only a single test is run the data-race is gone. I'm less sure about the goleak passing. It's also possible the rerun is not setting the correct flags somehow, but we should see that with --debug.

@odubajDT
Copy link
Author

odubajDT commented Sep 25, 2024

Thanks for the response!

first variant with --rerun-fails=1 --debug options:

gotestsum --rerun-fails=1 --debug --packages="./..." -- -race -timeout 600s -parallel 4 --tags=""
exec: [go test -json -race -timeout 600s -parallel 4 --tags= ./...]

rerun the same test:

exec: [go test -json -test.run=^$ -race -timeout 600s -parallel 4 --tags= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/cadvisor]

Full log:

/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --rerun-fails=1 --debug --packages="./..." -- -race -timeout 600s -parallel 4 --tags=""
exec: [go test -json -race -timeout 600s -parallel 4 --tags= ./...]
go test pid: 8235
go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0
✓  . (1.071s)
∅  internal/cadvisor/testutils
✖  internal/cadvisor (494ms)
✓  internal/cadvisor/extractors (1.108s)
✓  internal/ecsInfo (1.033s)
✓  internal/host (1.039s)
∅  internal/metadata
∅  internal/stores/kubeletutil
✓  internal/k8sapiserver (1.06s)
✓  internal/stores (1.072s)

DONE 98 tests, 1 failure in 74.185s

exec: [go test -json -test.run=^$ -race -timeout 600s -parallel 4 --tags= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/cadvisor]
go test pid: 15675
✓  internal/cadvisor (1.018s)

@odubajDT
Copy link
Author

On the other side, the same test without the --rerun-fails option

/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --debug --packages="./..." -- -race -timeout 600s -parallel 4 --tags=""
exec: [go test -json -race -timeout 600s -parallel 4 --tags= ./...]
go test pid: 8394
go: downloading gopkg.in/evanphx/json-patch.v4 v4.12.0
∅  internal/cadvisor/testutils
✖  internal/cadvisor (491ms)
✓  . (1.091s)
✓  internal/cadvisor/extractors (1.084s)
✓  internal/ecsInfo (1.039s)
✓  internal/host (1.043s)
∅  internal/metadata
∅  internal/stores/kubeletutil
✓  internal/k8sapiserver (1.058s)
✓  internal/stores (1.066s)

=== Failed
=== FAIL: internal/cadvisor  (0.00s)
PASS
goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 6 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc000059b20, 0xc00033ac20)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 5
make[2]: *** [../../Makefile.Common:130: test] Error 1
make[1]: *** [Makefile:187: receiver/awscontainerinsightreceiver] Error 2
make: *** [Makefile:127: gotest] Error 2
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 7 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc000059b60, 0xc00033ac40)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 5
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 8 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc000059bc0, 0xc00033ac50)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 5
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 9 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc000059c00, 0xc00033ac60)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 5
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 36 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a4240, 0xc00018d700)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 35
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 37 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a4280, 0xc00018d720)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 35
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 38 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a42e0, 0xc00018d730)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 35
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 39 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a4320, 0xc00018d740)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 35
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 44 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a43a0, 0xc00018daf0)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 43
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 45 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a43e0, 0xc00018db20)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 43
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 46 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a4440, 0xc00018db40)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 43
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
 Goroutine 47 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep on top of the stack:
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.(*MapWithExpiry).sweep(0xc0001a4480, 0xc00018db60)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:142 +0x1b0
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/metrics.NewMapWithExpiry in goroutine 43
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/aws/metrics/metric_calculator.go:126 +0x265
]
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awscontainerinsightreceiver/internal/cadvisor	0.491s

DONE 98 tests, 1 failure in 72.943s

@odubajDT
Copy link
Author

Full logs can be found as part of this PR open-telemetry/opentelemetry-collector-contrib#35413

I created it just for showcasing the problem

@odubajDT
Copy link
Author

Hello, any news about this?

1 similar comment
@odubajDT
Copy link
Author

odubajDT commented Nov 4, 2024

Hello, any news about this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants