You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Consumers of reflector caches might occasionally notice that a Store is not guaranteed to be fully initialized before a webhook server or controller starts, leading to startup errors and readiness checks being passed too early. This is because the stream has to be awaited "a little bit" before this happens, to populate the Store fully. Unfortunately, there is no easy mechanism for users to wait for this.
Describe the solution you'd like
Something that can be awaited by something wrapping reflector, Store or WatchStreamExt. This is purposefully vague because I am looking for good suggestions.
The big problem with actually implementing this is that the watcher stream (which populates the Store) and the Store are separate objects (connected by this line), so we cannot just await store initialization without actually polling the stream (to start the watcher machinery). However, polling the stream is usually done by the controller/app itself.
IF we had something simple to .await, then fixing webhook servers would be much more ergonomic to setup, and it could also be used to prevent early Controller starts in both the native setup (easy Controller mode; it owns the stream/store, it can await) and the custom setup (hard unstable-controller for #1080; user owns the store/stream, users can await).
(Personally, i am a little curious if this full machinery is necessary; stores are designed in such a way that they keep its state until re-inits are coming in so they should always have state. On the other hand, this type of setup would allow for a store that could potentially invalidate its own cache down the line if it does get stuck after one init.)
Webhook servers can currently work around the problem by looping through raw watcher events before chaining through reflectors and setting some global state for a /health check, but this is obviously ugly.
Documentation, Adoption, Migration Strategy
easy if it can be done additively, but it might not.
Target crate for feature
kube-runtime
The text was updated successfully, but these errors were encountered:
clux
added
question
Direction unclear; possibly a bug, possibly could be improved.
runtime
controller runtime related
labels
Jun 7, 2023
What problem are you trying to solve?
Consumers of
reflector
caches might occasionally notice that aStore
is not guaranteed to be fully initialized before a webhook server or controller starts, leading to startup errors and readiness checks being passed too early. This is because the stream has to be awaited "a little bit" before this happens, to populate theStore
fully. Unfortunately, there is no easy mechanism for users to wait for this.Describe the solution you'd like
Something that can be awaited by something wrapping
reflector
,Store
orWatchStreamExt
. This is purposefully vague because I am looking for good suggestions.The big problem with actually implementing this is that the watcher stream (which populates the
Store
) and the Store are separate objects (connected by this line), so we cannot just await store initialization without actually polling the stream (to start the watcher machinery). However, polling the stream is usually done by the controller/app itself.IF we had something simple to
.await
, then fixing webhook servers would be much more ergonomic to setup, and it could also be used to prevent earlyController
starts in both the native setup (easy Controller mode; it owns the stream/store, it can await) and the custom setup (hard unstable-controller for #1080; user owns the store/stream, users can await).Describe alternatives you've considered
There is a larger proposal for a
SafeStore
usingCancellationToken
in https://github.com/kube-rs/kube/pull/1128/files#diff-5bf1be34972f407369ee8a0252b3abf07720efb03bca6b67043b0721a83240a4 . This might actually be the way to go, but opening this up more generally in the hopes to get more movement here.(Personally, i am a little curious if this full machinery is necessary; stores are designed in such a way that they keep its state until re-inits are coming in so they should always have state. On the other hand, this type of setup would allow for a store that could potentially invalidate its own cache down the line if it does get stuck after one init.)
Webhook servers can currently work around the problem by looping through raw watcher events before chaining through reflectors and setting some global state for a
/health
check, but this is obviously ugly.Documentation, Adoption, Migration Strategy
easy if it can be done additively, but it might not.
Target crate for feature
kube-runtime
The text was updated successfully, but these errors were encountered: