Tearing down test suite for "controllers" package is flaky

[invidian]

What steps did you take and what happened:
Running while make test TEST_ARGS='-count 1' ; do sleep 1; done or with modified Makefile to run tests for ./controllers/ instead of ./... sometimes (for me it failed after 2 tries) fails with the following error message:

Ran 16 of 16 Specs in 5.151 seconds
SUCCESS! -- 16 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
Tearing down test suite
I0211 19:18:12.347693 2695933 controller.go:228] controller-runtime/manager/controller/cluster "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
I0211 19:18:12.347727 2695933 controller.go:228] controller-runtime/manager/controller/machine "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Machine"
I0211 19:18:12.347734 2695933 controller.go:228] controller-runtime/manager/controller/machinedeployment "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineDeployment"
I0211 19:18:12.347735 2695933 deleg.go:130] controller-runtime/webhook "msg"="shutting down webhook server"
I0211 19:18:12.347749 2695933 controller.go:228] controller-runtime/manager/controller/cluster "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
I0211 19:18:12.347754 2695933 controller.go:228] controller-runtime/manager/controller/machineset "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet"
I0211 19:18:12.347693 2695933 controller.go:228] controller-runtime/manager/controller/machinehealthcheck "msg"="Stopping workers" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineHealthCheck"
E0211 19:18:12.347928 2695933 controller.go:301] controller-runtime/manager/controller/cluster "msg"="Reconciler error" "error"="Patch \"http://127.0.0.1:45095/apis/cluster.x-k8s.io/v1alpha4/namespaces/md-test/clusters/test-cluster/status\": context canceled" "name"="test-cluster" "namespace"="md-test" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="Cluster"
E0211 19:18:12.348235 2695933 machineset_controller.go:159] controller-runtime/manager/controller/machineset "msg"="Failed to reconcile MachineSet" "error"="failed to retrieve InfrastructureMachineTemplate external object \"md-test\"/\"md-template\": Get \"http://127.0.0.1:45095/apis/infrastructure.cluster.x-k8s.io/v1alpha4/namespaces/md-test/infrastructuremachinetemplates/md-template\": context canceled" "name"="md-d8t9l-6dc69c8f65" "namespace"="md-test" "reconciler group"="cluster.x-k8s.io" "reconciler kind"="MachineSet"
E0211 19:18:12.348424 2695933 runtime.go:78] Observed a panic: "send on closed channel" (send on closed channel)
goroutine 8499 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1914ca0, 0x1cfd350)
        /home/invidian/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /home/invidian/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1914ca0, 0x1cfd350)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apimachinery/pkg/watch.(*Broadcaster).Action(...)
        /home/invidian/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/watch/mux.go:198
k8s.io/client-go/tools/record.(*recorderImpl).generateEvent.func1(0xc000c9be00, 0xc000770000)
        /home/invidian/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/record/event.go:344 +0xb7
created by k8s.io/client-go/tools/record.(*recorderImpl).generateEvent
        /home/invidian/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/record/event.go:341 +0x3bd
panic: send on closed channel [recovered]
        panic: send on closed channel

goroutine 8499 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /home/invidian/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1914ca0, 0x1cfd350)
        /usr/lib/go/src/runtime/panic.go:969 +0x1b9
k8s.io/apimachinery/pkg/watch.(*Broadcaster).Action(...)
        /home/invidian/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/watch/mux.go:198
k8s.io/client-go/tools/record.(*recorderImpl).generateEvent.func1(0xc000c9be00, 0xc000770000)
        /home/invidian/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/record/event.go:344 +0xb7
created by k8s.io/client-go/tools/record.(*recorderImpl).generateEvent
        /home/invidian/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/record/event.go:341 +0x3bd
FAIL    sigs.k8s.io/cluster-api/controllers     18.363s
FAIL
make: *** [Makefile:121: test] Error 1

What did you expect to happen:
Tests should be reliable and pass every time OR tearing down test suite should not make test fail.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Run:
I=0; while make test TEST_ARGS='-count 1' ; do I=$((I+1)); sleep 1; done; echo "Failed after $I tries"
if you want to see how often it fails.

Environment:

• Cluster-api version: bfc1498
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[vincepri]

Hm, this should have been fixed with the update on CR v0.8.2

[vincepri]

Actually, v0.8.2 fixed part of the issue. We'll need kubernetes/kubernetes#98922 to get merged and a new release of controller runtime.

/milestone v0.4.0
/assign

[fabriziopandini]

/priority important-soon

[vincepri]

This should be fixed now on main, can you all run some tests and report if you have any issues?

[invidian]

I've run unit tests several times now. I found some other flakes, but this one seems to be fine now. Cool!

[fabriziopandini]

/close
as per comment above

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close
as per comment above

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.