Remove Requeue Feature

[buehler]

Regarding several issues with the requeue logic (#421, #365, #280)
it could be an idea to remove the requeue "feature" altogether. Since the watcher timeout already triggers a requeue of all those entities, this could be used as "requeue" logic and problems with duplicates would be gone.

wdyt?

/cc @erin-allison

[erin-allison]

Honestly, I'm inclined to go pretty much the exact opposite direction.

I'm not a big fan of the default being to reconcile all entities every minute, as that can become an issue as number of entities grows (or if you're querying external systems too frequently). Kubernetes has functions where we can (and imo, should) be instead using a longer polling interval with watch bookmarks to do thing less often, and then have more control over if we want to dump the cache + reconcile everything on a certain timer.

This will allow a higher degree of flexibility and less load on the K8S API server during execution.

[erin-allison]

Also, having a "requeue" is a pretty standard feature such that I could predict it being an issue if we were to remove that function.

I think if we can test/merge #394, that is a step in the right direction that would help resolve some issues, simply needing to then implement watch bookmarks later on (also, kubebuilder's tutorial even follows the same logic as I did: requeue is taken as you want to reconcile no later than X, though other events could occur it again sooner).

[erin-allison]

And on that note, I should have some more time again to work on this in a couple weeks once I'm done moving.

[buehler]

Alright :-) you convinced me! Thanks for the clarification