[14 Jan 2025] - [sidecar] logs are getting flooded as sidecar code is have endless Retry mechanism

[asundriya]

What happened:
logs are getting flooded as Drive is have endless Retry mechanism. WE can say spamming is happening on the failure command.

What you expected to happen:
logs should have a counter for retry when we see errors while bucket creation/access/grant.
As same error message is repeated day/night if failure is not fixed. User should have a control, how many time system should retry

How to reproduce this bug (as minimally and precisely as possible):

1. Induce an error while bucket any of the workflow creation/access/grant.
2. Look into the provisioner log, you will see same error message is repeated day/night if failure is not fixed.
   If the issue remains for couple of days, then it will eat all memory and space of the system

Issue is
The log handling for the COSI APIs is handled by the sidecar (https://github.com/kubernetes-sigs/container-object-storage-interface-provisioner-sidecar) where if an error is occured it endless goes on a retry mechanism. However, the sidecar does not currently stop retrying after some time. We should have a tweaking counter for the same

[BlaineEXE]

The sidecar is expected to be an operator of sorts, so I do expect it to retry until it's successful. That's part of the control theory that keeps Kuberentes and its systems stable.

That said, normally I also expect some sort of backoff mechanism. I would propose that the fix for this issue report focus on ensuring a reaosonable backoff. We can probably start with the backoff strategy and timing recommended by controller-runtime (reference needed) and readjust as needed in the future if this continues to come up.

[BlaineEXE]

One of my first thoughts when triaging/planning bugs is what sort of testing is needed. Regression tests are very important.

For bugs related to reconcile retry timing and logging, I have found that timing-related log-output expectations are hard to codify. In Rook, we tend to not create regression tests for these cases and instead just try to do our best to make sure system internals are logging helpful info without frequent spam.

@shanduur , my inclination here is to not require deeply involved unit/e2e tests, but I'm curious to your input here as well.

[BlaineEXE]

FYI, we have discussed this also in relation to other v1alpha2 things, and we plan to focus on this for the implementation portion of v1alpha2 work. There are other places where COSI needs to wait for something to happen and does so with a wait loop rather than watching dependent resources.

Switching to the controller-runtime framework will give us better tools for waiting on dependent resources without active waits like this one. This is planned work, but it will likely happen under the umbrella of switching to controller-runtime and subsequent follow-up work for v1alhpa2 implementation.

I'm not sure this is necessarily a bug -- more of an optimization IMO-- but it won't hurt to classify it as one for those who view it through the bug lens.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[BlaineEXE]

Since we are working toward v1alpha2 KEP version, which includes changes around what the sidecar watches and how/when it is expected to retry, I am expecting this to be resolved during the v1alpha2 code changes. I will leave this issue open as a reminder for us to prioritize making sure we aren't flooding logs during reconciliation.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[shanduur]

/remove-lifecycle rotten