Ensure APIStreaming is used if available

[alvaroaleman]

We should be using https://kubernetes.io/blog/2024/12/17/kube-apiserver-api-streaming/ if the server supports it. It could be that there are no changes required for that, but we should validate that assumption before the next release.

[nathanperkins]

Thanks! Looking forward to this feature and was wondering how we would enable it in controller-runtime. Automatic enablement would be really cool :)

[sbueringer]

I think CR should have the same default behavior as the underlying client-go version (it's not enabled per default in client-go v0.32)

[sttts]

xref @p0lyn0mial's https://github.com/kubernetes-sigs/controller-runtime/pull/2934/files where he proved that controller-runtime can adopt watchlist.

[p0lyn0mial]

xref @p0lyn0mial's https://github.com/kubernetes-sigs/controller-runtime/pull/2934/files where he proved that controller-runtime can adopt watchlist.

the test results from the pr can be found in this comment.

[alvaroaleman]

One thing @sbueringer noticed is that it doesn't seem the live clients LIST will use watchlist under the hood by default

And generally speaking, this doesn't necessarily mean that watchlist was actually used, just that nothing broke or am I missing something?

All tests from controller-runtime passed, except for ClientWithCache Get when getting unstructured objects should call client reader when not cached which doesn't use the watchlist feature.

[alvaroaleman]

For the cache, it seems our usage of cache.NewSharedIndexInformer will give us WatchList support if available without anything further needed (It gets set from the client-go featuregate in NewReflectorWithOptions which in turn is called from *controller.Run which in turn is called from *sharedIndexInformer.Run), is that correct @p0lyn0mial ?

I think to use WatchList in the live client, a change like this is required for typed/unstructured/metadata, similiar to what the generated clients do:

$ g diff
diff --git a/pkg/client/typed_client.go b/pkg/client/typed_client.go
index 92afd9a9..63014d9b 100644
--- a/pkg/client/typed_client.go
+++ b/pkg/client/typed_client.go
@@ -20,6 +20,7 @@ import (
        "context"

        "k8s.io/apimachinery/pkg/runtime"
+       "k8s.io/client-go/util/watchlist"
 )

 var _ Reader = &typedClient{}
@@ -157,6 +158,17 @@ func (c *typedClient) List(ctx context.Context, obj ObjectList, opts ...ListOpti
        listOpts := ListOptions{}
        listOpts.ApplyOptions(opts)

+       watchListOptions, hasWatchListOptionsPrepared, err := watchlist.PrepareWatchListOptionsFromListOptions(*listOpts.AsListOptions())
+       if err == nil && hasWatchListOptionsPrepared {
+               if err := r.Get().
+                       NamespaceIfScoped(listOpts.Namespace, r.isNamespaced()).
+                       Resource(r.resource()).
+                       VersionedParams(&watchListOptions, c.paramCodec).
+                       WatchList(ctx).
+                       Into(obj); err == nil {
+                       return nil
+               }
+       }
        return r.Get().
                NamespaceIfScoped(listOpts.Namespace, r.isNamespaced()).
                Resource(r.resource()).

I can not off-hand think of a good way to write tests for this though

[p0lyn0mial]

There are two FGs. The WatchList used by the kube-apiserver and enabled by default as of 1.32 and the WatchListClient used by the client (client-go/informers) disabled by default.

In order to use the new feature on a 1.32 cluster you need to explicitly enable WatchListClient for client-go library. I had to force it in my PR to test the new feature.
After enabling the FG it will work for both the informers and the "live" client/calls.

[sbueringer]

The live client Alvaro is referring to is implemented here:

controller-runtime/pkg/client/typed_client.go

Line 151 in 10a0272

func (c *typedClient) List(ctx context.Context, obj ObjectList, opts ...ListOption) error {

(or at least this is the part for typed objects). We are not using a List method from client-go.

	r, err := c.resources.getResource(obj)
	if err != nil {
		return err
	}

	listOpts := ListOptions{}
	listOpts.ApplyOptions(opts)

	return r.Get().
		NamespaceIfScoped(listOpts.Namespace, r.isNamespaced()).
		Resource(r.resource()).
		VersionedParams(listOpts.AsListOptions(), c.paramCodec).
		Do(ctx).
		Into(obj)

Our assumption at the moment is that this doesn't automatically use Watchlist when the WatchListClient feature flag is enabled.

[p0lyn0mial]

Our assumption at the moment is that this doesn't automatically use Watchlist when the WatchListClient feature flag is enabled.

ah, yes, this is correct, the feature was added to the generated client not to the rest client int the client-go lib.

[alvaroaleman]

Note: Due to lack of time from the maintainers, we will not update the live client to use apistreaming for the next c-r release. It already works in the cache which is the majority use-case and is default disabled in client-go anyways, so we do not want to block the release on this.

[nathanperkins]

Just to confirm my understanding:

• Controller Reconcile() uses the cache and will have APIStreaming enabled by default in controller-runtime v0.20.
• Calling client.List() on a live client (without caches) won't use APIStreaming in v0.20, and needs more work to support.
• Not sure what happens if you have a cached client and call List().

[sbueringer]

I think the following is the current state:

• server-side WatchList feature gate is enabled per default for Kubernetes >= 1.32
• client-side WatchListClient feature gate is disabled per default

So per default watchList is not used at all.

The client-side feature gate can be enabled by setting the "KUBE_FEATURE_WatchListClient" env variable to "true".

Then if the feature gate is enabled:

• informers (or more specifically the underlying reflector) will use WatchList for their initial "List" call
• cached clients are not using WatchList because they are just reading from the local informer store
• live clients:
  • metadata client uses WatchList if the feature gate is enabled
  • typed client & unstructured client do not use WatchList (as of now) if the feature gate is enabled

I'll open a PR to cover the last point (#3136)

After my PR:

server-side WatchList	client-side WatchListClient	behavior
true	true	watchList
false	true	watchList fails, fallback to list
true	false	list
false	false	list

Additional notes:

• Even if both feature gates are enabled, there are cases where list will be used, e.g. https://github.com/kubernetes/kubernetes/blob/v1.32.0/staging/src/k8s.io/client-go/util/watchlist/watch_list.go#L57