Make Knative eventing sources more serverless and scalable

[aslom]

Problem
A short explanation of the problem, including relevant restrictions.

Sources are running and consuming resources even when there is no events. And when large number of events is received source do not automatically scale up

Persona:
Which persona is this feature for?

System Integrator, Event consumer (developer)

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

When event sources are not receiving events they scale down to zero (or almost zero) and scale up when umber of events to send increases

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

weeks

Additional context (optional)
Add any other context about the feature request here.

Ref #2152

[slinkydeveloper]

Same as #2152 (comment) + this requires non-HTTP autoscaling (eg autoscaling based on kafka messages like https://github.com/kedacore/keda)

[aslom]

@slinkydeveloper Yes. Something like "activator" component that waits for activity related to registered sources and starts source? It could also work as auto-scaling to provide metrics to scale up source service/deployment (and down when no activity)?

[zroubalik]

Keda (is pull based autoscaler) so it supports mulitple event sources scaling (0-1) OOTB today, just as @slinkydeveloper mentioned. In case you are interested I could provide help with Keda on this.

[aslom]

@zroubalik @slinkydeveloper that would add KEDA as dependency for those knative sources that need pull based scaling?

If I understand correctly KEDA design we could use ScaledObject (example) to scale knative source deployment?

[zroubalik]

@aslom Exactly, in ScaledObject you can specify the specific settings for each Knative Source (ie. how do you want to scale it).

I can imagine a very simple controller that creates ScaledObject with preferred settings when Event Source is created.

[aslom]

I can see that two ways to implement

• modify event source controller to add pulling event statistic logic and scaling (no dependency on KEDA)
• modify event source controller to create ScaledObject that is then used by KEDA to pull event statics and do scaling (dependency on KEDA )

Or do both which I think may the best - if users have KEDA they can configure Knative to use KEDA. Otherwise can use built-in pulling and scaling in controller to scale sources

@zroubalik if the picture below is correct for KEDA and what do you think about the options?

Feel free to edit slides: https://docs.google.com/presentation/d/1JNV25z0ZkjA_vhZy0y3dag_xoku3dtunMYtoggyFu6Y/edit?usp=sharing

[zroubalik]

@aslom Hi, I think the picture is correct and described correctly both options.

As for the options, Keda provides all necessary scaling capabilities for multiple sources OOTB today. So it is up for a decision, where do we want to put the effort. Whether we want to reimplement the same functionality inside Knative, reuse Keda or implement both options.

I am willing to help with the Keda and controller part.

I'd like to note that Keda is going to release v1.0 soon and is planned to move Keda under CNCF. So there shouldn't any concerns on this, wrt adding it as an dependency for Knative Eventing.

[aslom]

@zroubalik that sounds good. As the very first step we should be able to use Keda by creating Scaler yaml?

That means that controller when it sees KEDA in source configuration then it creates Scaler yaml and assumes KEDA will do scaling.

That should allow us to do quick testing - I will start with Kafka source that is already supported in KEDA and see how scaling works. (I already have experience with Kafka source)

Does that sound reasonable?

[zroubalik]

Yeah, that scenario sounds good. I have an experience with Keda, so I'll be happy to write the controller for managing it. @aslom or do you plan to write that?

[aslom]

@zroubalik yes going ot write first version for quick review

[aslom]

@zroubalik do you know what is current state of scaling to zero in KEDA and HPA? It seems that is not possible as defaultHPAMinReplicas is used if MinReplicaCount is set to zero?
https://github.com/kedacore/keda/search?q=defaultHPAMinReplicas&unscoped_q=defaultHPAMinReplicas

It seems to be related to HPA not supporting it in some versions of K8S? minReplicas is set in HPA created by KEDA to 1 (as expected on code above) and if I manually edit it to

  minReplicas: 0

then I am getting:

$ kubectl get hpa keda-hpa-kafkasource-kafka-source-2a42349b-0fdd-11ea-9cac-264e53a4499c -o yaml > tmp-hpa.yaml
# edited
$ kubectl apply -f tmp-hpa.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
The HorizontalPodAutoscaler "keda-hpa-kafkasource-kafka-source-2a42349b-0fdd-11ea-9cac-264e53a4499c" is invalid: spec.minReplicas: Invalid value: 0: must be greater than 0

Seems to be fixed in k8s 1.16+
kubernetes/kubernetes#69687 (comment)
kubernetes/kubernetes#74526
https://github.com/kubernetes/kubernetes/pull/74526/files#diff-4364e3f754b12e39f60eea33105f6c7eL48

Going to test it with 1.16 but I wondered if it is documented somewhere in KEDA? I could not find an issue about it.

BTW: the confusing part was that even I manually scaled:

$ kubectl scale deployment kafkasource-kafka-source-2a42349b-0fdd-11ea-9cac-264e53a4499c --replicas=0

kafkasource deployment would not disappear - took me some time to realize that HPA was scaling it back to 1 ...

[zroubalik]

@aslom scaling to 0, is done by KEDA controller (it sets deployment to 0 replicas). It is not managed by HPA at the moment, as you correctly mentioned that this feature is fixed in k8s 1.16+, so it might get into the KEDA later, as we are supporting older k8s versions as well.

Scaling to 0 should work correctly, I wonder what caused the problem you are facing. You might hit the cooldown period?

[aslom]

@zroubalik are you sure that HAP is not used by KEDA? I am using the lates version I deployed from github 2 days ago. Where I can find about how ScaledObject and HPA works in KEDA? When I create ScaledObject I can see corresponding HPA is created. When I delete ScaledObject then HPA is deleted. The created HPA has minScale 1 and that prevents kafka source from going wiht replicas to 0 when I am running kubectl scale deployment