Controller-runtime and custom cache

[fgiloux]

Trying to setup a custom cache to filter APIBindings it seems that the resulting URL is not properly processed.

options.NewCache = cache.BuilderWithOptions(cache.Options{
		SelectorsByObject: cache.SelectorsByObject{
			&apisv1alpha1.APIBinding{}: {
				Field: fields.SelectorFromSet(fields.Set{
					"spec.reference.workspace.exportName": "settings-configuration.pipeline-service.io",
					"spec.reference.workspace.path":       "root:pipeline-service:management",
				}),
			},
		},
	})

gives

W0909 17:14:34.487460 1472564 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: failed to list *v1alpha1.APIBinding: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/apis.kcp.dev/v1alpha1/apibindings": Path not resolved to a valid virtual workspace
E0909 17:14:34.487484 1472564 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: Failed to watch *v1alpha1.APIBinding: failed to list *v1alpha1.APIBinding: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/apis.kcp.dev/v1alpha1/apibindings": Path not resolved to a valid virtual workspace

and also for other resources that my controller owns

W0909 17:14:34.050696 1472564 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: failed to list *v1.NetworkPolicy: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/networking.k8s.io/v1/networkpolicies": Path not resolved to a valid virtual workspace
E0909 17:14:34.050739 1472564 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: forbidden: User "kcp-admin" cannot get path "/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io/apis/networking.k8s.io/v1/networkpolicies": Path not resolved to a valid virtual workspace

Without custom cache it works and the url is

https://xxx.xxx.xxx.xxx:6443/services/apiexport/root:pipeline-service:management/settings-configuration.pipeline-service.io"

I haven't had time to investigate and I am not excluding that I did something wrong but this is the only modification.

[ncdc]

If you're managing the NewCache function, you need to make sure it does this:

controller-runtime/pkg/kcp/wrappers.go

Lines 62 to 72 in 824b15a

	func NewClusterAwareCache(config *rest.Config, opts cache.Options) (cache.Cache, error) {
	c := rest.CopyConfig(config)
	c.Host += "/clusters/*"
	opts.NewInformerFunc = informers.NewSharedIndexInformer
	opts.Indexers = k8scache.Indexers{
	kcpcache.ClusterIndexName: kcpcache.ClusterIndexFunc,
	kcpcache.ClusterAndNamespaceIndexName: kcpcache.ClusterAndNamespaceIndexFunc,
	}
	return cache.New(c, opts)
	}

[fgiloux]

@ncdc thank you for the pointer and sorry for the delay.
Adding something like this seems to work:

func ClusterAwareBuilderWithOptions(options cache.Options) cache.NewCacheFunc {
	return func(config *rest.Config, opts cache.Options) (cache.Cache, error) {
		if options.Scheme == nil {
			options.Scheme = opts.Scheme
		}
		if options.Mapper == nil {
			options.Mapper = opts.Mapper
		}
		if options.Resync == nil {
			options.Resync = opts.Resync
		}
		if options.Namespace == "" {
			options.Namespace = opts.Namespace
		}
		if opts.Resync == nil {
			opts.Resync = options.Resync
		}

		return kcp.NewClusterAwareCache(config, options)
	}
}

I would think it belongs to the wrapper. What do you think?
cc @varshaprasad96

[ncdc]

@stevekuznetsov

[fgiloux]

Note: "seems to work" except that there is no label conversion for APIBinding so that I cannot filter through spec.reference.workspace.exportName and spec.reference.workspace.path but it would work for Pods for instance. Obviously the fact that not many resources support the mechanism significantly diminishes its value

[stevekuznetsov]

Yeah, field selectors are very hard-coded and limited :|

[stevekuznetsov]

Could you use an index here?

[fgiloux]

Could you use an index here?

I am not sure to follow. My understanding is that indexes help with retrieving resources from cache not avoiding them to get cached. Please let me know what I am missing. My original intent was not to cache the resources that I don't need to save some memory. As a workaround I am filtering the event and add unnecessary apibindings into the cache. As I am not expecting a huge amount of apibindings it is not a major concern for me.

An open question is: do we see value in making a cluster aware version of controller-runtime BuilderWithOptions?
As you stated for field selectors it is not granted but possibly for the other configurations? I am happy to create a PR if the answer is yes.

[stevekuznetsov]

Please let me know what I am missing.

Sorry, I didn't realize your intent was to make your cache size smaller in memory. The index won't help there.

do we see value in making a cluster aware version of controller-runtime BuilderWithOptions?

I swear we had a PR open to do this .. yes, I think it's a good idea ish.

[fgiloux]

ack. I will look at that tomorrow.