Integration tests not idempotent because of faulty cleanup

[mlavacca]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When integration tests are executed, a lot of cleaning functions are deferred to the end of the test. The problem is that we use the exitOnErr function to assert that no error must occur. Whenever such a function fails, the program exits, and all the deferred functions are not executed anymore. This leads to having orphaned resources.

Expected Behavior

We should use the new mechanism introduced in KTF to clean resources, instead of the deferred functions. See #2618 as a similar issue in KTF.

[mflendrich]

Currently we don't see strong drivers behind this issue. Does not conern CI runs; @jrsmroz worked around this by purging the kind cluster between re-runs. Labeling as nice-to-have therefore.