Make Knative eventing more serverless and scalable

[aslom]

Make Knative eventing more serverless and scalable

Problem
A short explanation of the problem, including relevant restrictions.

Knative eventing (sources, channels, brokers, ...) should scale to zero when not used and scale up in proportion to number of events

Persona:
Which persona is this feature for?

System Integrator, Event consumer (developer)

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

When no events coming into Knative eventing then it should scale down to zero (knative eventing services should take minimal or no memory or cpu, channels should scale down as well). When events are received over HTTP it should scale up.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

weeks

Additional context (optional)
Add any other context about the feature request here.

Let start discussion about what would be the lowest hanging fruits?

[aslom]

Improvements should help with related issue about Broker internal design improvements #1555

[aslom]

There may be other related issue such as about making control-plane scalable as well?

[aslom]

I would like to find who is interested in this topic: @nachocano @slinkydeveloper @Harwayne @matzew @lionelvillard ?

[aslom]

What would be the best way to capture possible options short term (MVP, before 1.0?) and long term ideal solution?

I am thinking to start with design docs for each sub-topic such as sources and channels?
Using design doc using template?
https://github.com/knative/community/blob/master/CONTRIBUTING.md#design-documents
https://docs.google.com/document/d/1QRREfL8gSVSURHMkDLjDtMh1BBsNiwfmaBiGnJUDQ2g/edit

[mikehelmick]

/cc @grantr

[nachocano]

Related to this: #2060 #2061 #2083

…

On Wed, Nov 6, 2019 at 8:36 AM Mike Helmick ***@***.***> wrote: /cc @grantr <https://github.com/grantr> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2152?email_source=notifications&email_token=ABD65DF33MYNWNTLZJEUXBLQSLW7HA5CNFSM4JJWYSR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDHFGAY#issuecomment-550392579>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABD65DGKUC33DZ4BOIMAXRTQSLW7HANCNFSM4JJWYSRQ> .

[grantr]

/area performance

[slinkydeveloper]

I think it's quite interesting the idea to scale up, but I'm not sure it would be simple to go to 0.
Are we sure that we can consider every knative component stateless, so it can be scaled down to 0 without data loss?
+1 for the document

[aslom]

@slinkydeveloper hopefully we can scale control-plane to zero - only spin it up when CR is created/modified/deleted? For data plan may be more complex as some sources requires pull or persistent polling connection so something needs ot keep running even if there is no events recevied?

[slinkydeveloper]

There is any way atm to let k8s spin up control-plane only when CR changes happens?

[antoineco]

@slinkydeveloper that would require one component to always be up and running to watch for changes to relevant resources and then start the corresponding controller. Kubernetes does not actively notify controllers when changes occur (push), the controllers maintain a long connection to receive events about resources they are interested in (pull).