keda-metrics-adapter calling apiserver qps are quite high

[bamboo12366]

Report

we create arounds 200 scaleObjects in the cluster, the qps from keda-metrics-adapter to apiserver is quite high, which create burden to apiserver

the code for listing here: https://github.com/kedacore/keda/blob/main/pkg/provider/provider.go#L90

Expected Behavior

Not calling apiserver so frequently, should be done by reconcile by watching constantly

Actual Behavior

Calling apiserver every time the metrics request to get the so resource

Steps to Reproduce the Problem

large amounts of scaleObjectx

Logs from KEDA operator

No response

KEDA Version

2.6.1

Kubernetes Version

1.20

Platform

Other

Scaler Details

No response

Anything else?

1. maybe we can using the shared informer with scaledObject Controller, when get new so can get directly from informer cache
2. or create a shared cache between controller and provider

if you guys feel ok either way, I can give a try

[zroubalik]

Yeah, this is something that needs improvement, @bamboo12366 would be nice if you can try to tackle this.

FYI there's already controller present in the MetricsAdapter (currently used for metricNames), so you might want to go this direction: https://github.com/kedacore/keda/blob/main/controllers/keda/metrics_adapter_controller.go

[zroubalik]

There might be even an easier solution, replace following section:

keda/pkg/provider/provider.go

Lines 84 to 90 in 99619e3

	// get the scaled objects matching namespace and labels
	scaledObjects := &kedav1alpha1.ScaledObjectList{}
	opts := []client.ListOption{
	client.InNamespace(namespace),
	client.MatchingLabels(selector),
	}
	err = p.client.List(ctx, scaledObjects, opts...)

with

...
	scaledObject := &kedav1alpha1.ScaledObject{}
	p.client.Get(ctx, namespaceNameParsedFromMetricSelector, scaledObject)
...

we should be able to parse ScaledObject.Name from a label scaledobject.keda.sh/name that is set on ScaledObject and most likely is part of the metricSelector.

It's definitely worth exploring.

[bamboo12366]

hey @zroubalik

can I just using the manager.GetClient() and pass the client to the provider, thus the provider and scaledObject Controller are sharing the same informer cache together

the code will be like
https://github.com/kedacore/keda/blob/main/adapter/main.go#L119-L123

with

        mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:    scheme,
		Namespace: namespace,
	})
	if err != nil {
		return nil, nil, err
	}
	if err := runScaledObjectController(ctx, mgr, handler, logger, externalMetricsInfo, externalMetricsInfoLock, maxConcurrentReconciles, stopCh); err != nil {
		return nil, nil, err
	}
	
	return kedaprovider.NewProvider(ctx, logger, handler, mgr.GetClient(), namespace, externalMetricsInfo, externalMetricsInfoLock), stopCh, nil

[zroubalik]

ScaledObject controller is running in KEDA Operator Pod, Provider in Metrics Server Pod.

[zroubalik]

The client for Metrics Server (thus provider) is setup here: https://github.com/kedacore/keda/blob/main/adapter/main.go

I happy to see any improvements on this side.

[bamboo12366]

yea, what trouble me is the Metrics Server pare.
I will create a PR work on this

[bamboo12366]

@zroubalik master, could you help me review on this #2922?