Junit reporter reports failures that are not reported by default reporter

[alvaroaleman]

Hey I have this PR: kubernetes-sigs/controller-runtime#1053 where for pull-controller-runtime-test-master the junit reporter reports a failure:

/home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/internal/testing/integration/internal/process_test.go:203
Expected process to exit.  It did not.
/home/prow/go/src/sigs.k8s.io/controller-runtime/pkg/internal/testing/integration/internal/process_test.go:219

however, this failure is only visible in the junit reporter, the default reporter for that package reports successes only: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_controller-runtime/1053/pull-controller-runtime-test-master/1285363491917533186#1:build-log.txt%3A993

This is where the junit is written: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_controller-runtime/1053/pull-controller-runtime-test-master/1285363491917533186#1:build-log.txt%3A987

And this is the raw junit file: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_controller-runtime/1053/pull-controller-runtime-test-master/1285363491917533186/artifacts/junit_Internal%20Suite_1.xml

Any idea what the culprit could be? I went a bit through ginkgos code but couldn't find anything that would explain this.

[onsi]

Sorry I missed this months ago - is this still an issue you're grappling with @alvaroaleman ?

[alvaroaleman]

I ended up working around this by grepping for failures in the junit after the test run and doing an exit 1 if that yields anything. IMHO it is pretty unexpected to have tests exit with rc 0 after failures, regardless of how they are reported.

[onsi]

Um. Yeah. If there's a failure and Ginkgo is exiting zero that's a serious bug in either Ginkgo or go/testing - and one I haven't seen before. I'll take a closer look at output links you shared once I'm at a computer (they're not loading on my phone).

[onsi]

hey @alvaroaleman - sadly it looks like the various artifacts associated with that particular test run are no longer available. I'll keep an eye out for this. Based on your description it isn't just that the ginkgo process exited with 0 but that the default reporter didn't even emit the failure? That's certainly very very strange. I looked at the test in question and it looks fine to me.

I'm going to close this now but if you see this again please open this issue back up and I'll jump on it. Apologies, again, for being so unresponsive in the past.