Recreate managed resource secrets that have deletion timestamp

[dimityrmirchev]

How to categorize this PR?

/area quality
/kind cleanup

What this PR does / why we need it:
In #8398 a fix for unwanted garbage collection of MR secrets was provided, but in some systems there are still such secrets that have deletion timestamp. After #8745 the finalizer that protects them will be gone. This PR introduces a recreation step so gardenlet can recreate these secrets on startup.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
/invite @rfranzke

Thanks for reporting!

Release note:

NONE

[rfranzke]

Why is this needed? You could just continue using secret in the WaitUntilResourceDeleted call?

[dimityrmirchev]

WaitUntilResourceDeleted uses the passed object and continuously modifies it via Get calls. The initial thought was to not modify the passed secret to this function. Either way it is called with argument original.DeepCopy so I guess we can apply your suggestion.

[dimityrmirchev]

@rfranzke thanks for the review. I implemented most of the suggestions. My only concern with the flow.Parallel() is that these can be many (possibly in the hundreds?) secrets that need to be recreated. I see that the current implementation does not provide a setting to set the degree of parallelism. Wouldn't hundreds of goroutines making calls to the API server cause client or server side throttling in which case we might observe failures?

[rfranzke]

@rfranzke thanks for the review. I implemented most of the suggestions. My only concern with the flow.Parallel() is that these can be many (possibly in the hundreds?) secrets that need to be recreated. I see that the current implementation does not provide a setting to set the degree of parallelism. Wouldn't hundreds of goroutines making calls to the API server cause client or server side throttling in which case we might observe failures?

Well, I assume that the API server can properly handle a few hundred create/update requests to secrets. This is a one time action anyways and only affects the seed API server and nothing else. I wouldn't be so afraid. These operations are fast and simple (no continuous listing required, etc.). If you still want to limit it, you can follow an easy approach like in https://github.com/gardener/gardener/blob/master/pkg/utils/gardener/secretsrotation/etcd.go#L105-L136 with the rate.NewLimiter and then limiter.Wait in the flow task function, but I think it's overkill. For sure, you should introduce the parallelism, please.

[dimityrmirchev]

@rfranzke Thanks for the rate.Limiter hint. I think that the concerns should be addressed now. PTAL

[dimityrmirchev]

/retest-required

[rfranzke]

/lgtm
/approve

[gardener-prow]

LGTM label has been added.

Git tree hash: 777fe881dd8ffec7c6af4967bc9f9306acbe6733

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment