Requeue Overhaul Tracking

[DirectXMan12]

We need to rework our requeue handling. Namely:

• RequeueAfter should never consider backoff (🐛 Reset Failure count for reconcile loops using RequestAfter #396), unless combined with an error
• Requeue should maybe have an escape hatch to ignore backoff, but you probably want backoff
• Requeue-on-error should probably ignore "non-transient" errors (e.g. notfound from cache), but determining non-transience is tricky (e.g. notfound is always non-transient from cached data, but may be transient from live data, so this might require x/errors)

Ref #377

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[DirectXMan12]

/remove-lifecycle stale
/lifecycle frozen

[vincepri]

Closing in favor of #617

/close

[k8s-ci-robot]

@vincepri: Closing this issue.

In response to this:

Closing in favor of #617

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.