Document ways to trigger/not trigger exponential backoff

[djzager]

Related to #617

As a user of controller-runtime, it is not immediately clear what returns from a Reconcile() will force a requeue with (or without) an exponential backoff. Some quick examples:

return reconcile.Result{requeue: true}, nil

return reconcile.Result{requeueAfter: time.Duration(60) * time.Second}, nil

// err != nil
return reconcile.Result{}, fmt.Errorf("My custom error string")

[djzager]

/kind documentation

[djzager]

I see mentions of backoff in

controller-runtime/pkg/internal/controller/controller_test.go

Lines 352 to 354 in 40070e2

	// TODO(directxman12): we should ensure that backoff occurrs with error requeue
	It("should not reset backoff until there's a non-error result", func() {

I think the two potential landing spots for documenting the requeue/backoff behavior would be:

1. in the godoc for controller-runtime (either in the controller directly or in workqueue?)
2. in the kubebuilder book (maybe here?). This would require a separate issue against github.com/kubernetes-sigs/kubebuilder

[vincepri]

/priority important-longterm
/help

[k8s-ci-robot]

@vincepri:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/priority important-longterm
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[cflewis]

I care about this. I spent a couple days writing code to do this and then someone on my team said "maybe this is a Kubebuilder feature" which got me to here.

What do I need to do to make an exponential backoff during a reconcile happen?

[vincepri]

@cflewis Today, you only need to return an error to make it go in exponential backoff loop.

[cflewis]

Can you configure how high the back off will go? And where is the exponential backoff in the code base? I'm trying to follow the threads (eventually ending up at `client-go/workqueue`) but I don't see evidence of any exponential backoff 🤔

…

On Fri, Jun 5, 2020 at 2:57 PM Vince Prignano ***@***.***> wrote: @cflewis <https://github.com/cflewis> Today, you only need to return an error to make it go in exponential backoff loop. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#808 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAKVPFYW3ADGRJ2TL56SF3RVFS2ZANCNFSM4KYU2KXA> .

[djzager]

You can configure the Rate Limiter on the Options to the Controller you create: https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/controller/controller.go#L40-L43

I noticed that the ratelimiter mentioned is internal to controller runtime but I believe it is intentionally identical to the one provided in client-go https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/ratelimiter/ratelimiter.go#L21-L30

https://godoc.org/k8s.io/client-go/util/workqueue#RateLimiter for more information about rate limiters. You may be specifically interested in https://github.com/kubernetes/client-go/blob/master/util/workqueue/default_rate_limiters.go#L77-L83

I haven't tested this, but taking from the example provided in the godoc:

// Create a new Controller that will call the provided Reconciler function in response
// to events.
c, err := controller.New("pod-controller", mgr, controller.Options{
    Reconciler: reconcile.Func(func(o reconcile.Request) (reconcile.Result, error) {
        // Your business logic to implement the API by creating, updating, deleting objects goes here.
        return reconcile.Result{}, nil
    }),
   RateLimiter: workqueue.NewItemExponentialFailureRateLimiter(baseDelay, maxDelay),
})
if err != nil {
    log.Error(err, "unable to create pod-controller")
    os.Exit(1)
}

Hope this helps.

[cflewis]

This is exactly what I need, thank you. Now to just upgrade all my k8s deps

…

< On Sun, Jun 7, 2020 at 7:13 PM David Zager ***@***.***> wrote: You can configure the Rate Limiter on the Options to the Controller you create: https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/controller/controller.go#L40-L43 I noticed that the ratelimiter mentioned is internal to controller runtime but I believe it is intentionally identical to the one provided in client-go https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/ratelimiter/ratelimiter.go#L21-L30 https://godoc.org/k8s.io/client-go/util/workqueue#RateLimiter for more information about rate limiters. You may be specifically interested in https://github.com/kubernetes/client-go/blob/master/util/workqueue/default_rate_limiters.go#L77-L83 I haven't tested this, but taking from the example provided in the godoc <https://godoc.org/sigs.k8s.io/controller-runtime/pkg/controller#ex-Controller> : // Create a new Controller that will call the provided Reconciler function in response // to events. c, err := controller.New("pod-controller", mgr, controller.Options{ Reconciler: reconcile.Func(func(o reconcile.Request) (reconcile.Result, error) { // Your business logic to implement the API by creating, updating, deleting objects goes here. return reconcile.Result{}, nil }), RateLimiter: workqueue.NewItemExponentialFailureRateLimiter(baseDelay, maxDelay), }) if err != nil { log.Error(err, "unable to create pod-controller") os.Exit(1) } Hope this helps. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#808 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAKVPBN6NGA7BILTXL72NTRVRCNXANCNFSM4KYU2KXA> .