Verify that cleanup is complete

[mlavacca]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

In KGO the Dataplane has a couple of finalizers that are needed to perform the deletion of cluster-wide resources (ClusterRoles and ClusterRoleBindings). When the Dataplane is deleted, the controller is in charge of collecting the cluster-wide resources and removing the finalizer from the Dataplane, so that it can be collected as well.
All the integration tests use the cleaner utility to perform the cleanup. In that function, all the resources marked to be collected are deleted, but there is no check that those resources have been really deleted.
When having resources with finalizers, it is not enough, because it happens that those resources are marked for deletion, but there is no controller in place to remove the finalizers, hence they hang forever marked for deletion.

Proposed Solution

cleaner.Cleanup should be improved so that after deleting each resource, it should periodically check that all the resources don't actually exist anymore. Only when all the resources are really deleted, the cleanup can return and the test can be concluded.

Additional information

No response

Acceptance Criteria

• In the KGO integration tests all the resources are correctly collected.