Set a controller debounce time

[clux]

Would you like to work on this feature?

no

What problem are you trying to solve?

If we are setting a Controller with N associated streams (via watches/owns), then without a debounce time we usually get between 2 and N reconciliations triggered per resource due to there not being enough time to deduplicate schedulerequests.

(how many reconciliations we see depends basically on how quickly the reconciler finishes because we do still do deduplication in the shceduler while the object is reconciling).

Idempotency is of course encouraged as a partial solution, but users still have to verify that everything is there, and it can cause quite a bit of startup load (as reported in #318). A general fix here is to allow an N seconds delay for events to deduplicate properly in the scheduler. An optimal user number will likely depend on the normal boot / list time, so not sure what a good default is. Maybe we leave it as zero/one sec, but document that you can set it to something like 2-5s?

NB: This issue is basically a rewrite of the last remaining part of #318 which i'll close, as this deserves a clearer issue that someone can pick up. I don't think it's that hard to solve.

Describe the solution you'd like

Define a configurable constant in controller/mod that we can push into the Scheduler for when an update happens, and for us to use the intial scheduling delay inside the applier:

• kube/kube-runtime/src/scheduler.rs

  Lines 67 to 68 in db585dd

  	// TODO: this should add a little delay here to actually debounce
  	self.queue.reset_at(&entry.queue_key, request.run_at);

• Initial schedule:

  kube/kube-runtime/src/controller/mod.rs

  Line 279 in db585dd

  run_at: Instant::now() + Duration::from_millis(1),

This should be propagated through Controller into applier as some kind of controller::Config struct (probably also used by #1248).

Describe alternatives you've considered

idempotency or predicates, but that can never really help avoid this fully.

Documentation, Adoption, Migration Strategy

extra parameter for applier, additive change for Controller users

Target crate for feature

kube-runtime

[aryan9600]

atm, the controller inserts a 1ms debounce time as you have pointed out. if users don't wish to provide or don't care about a debounce time, should we continue using 1ms as the default debounce time or set debounce to 0?

[clux]

1ms vs 0 is pretty inconsequential. think we do 1ms just to ensure it's actually scheduled in the future and give everything room to breathe. both should be fine and compatible with DelayQueue.

personally, i'm more curious about whether the default debounce should be longer (like 1s) as a default.

[aryan9600]

it really depends on how long the reconciliation would take, if you're writing a controller that reaches out to external services like a Git repo, then its almost impossible to predict how long will a full reconcile take, but i feel like 1s is a good default that will mostly work for all controllers that do everything in cluster.