Perf Enhancement

[rohitagg2020]

We need to look into what are the possible ways to improve the kapp performance.

To Do Task:

1. Figure out all the places we are making calls to k8s server and if we can reduce them.
2. Another possible place:
   When kapp deploy is using configmaps (vs. kapp deploy -a label:my-app-name=xxx), is the usedGVs property in the main configmap an optimisation so that when querying based on labelKey/labelValue from the main configmap to identfiy app resources, it doesn't have to query every single resource type in the whole cluster, and can at least restrict it to just resources within those api groups?
   usedGVs is currently used to decide which apis we can ignore if they are not available (sometimes some api servers die, or something else happens making it not possible to list resources). we still optimistically query all types, but, we may evolve that behaviour to improve perf over time.

• Use memoization to reduce redundant api calls to get the Resources supported by the K8s cluster.
• In case of kapp controller, we should make the kapp inspect disable by default. Individual APP CR's can opt-in for the kapp inspect if they want. carvel-kapp-controller#467
• Service Account Scoping - to see what permissions does SA have and how APP CR's are consuming them in kapp controller. - [ry]not doing, see comment by @rohitagg2020 below
• Go through all the API calls made by kapp deploy/delete/inspect and see if we can reduce them.
• While querying the resources by label selector, we should query only the GV's which have been used earlier to deploy rather than querying all the GV's.
• Match resource type associated resources with API group and resource, thus reducing API calls in waiting stage

[GrahamDumpleton]

The ConfigMap in this case should not use some special format. It would be better to record the list of resources in a format which is also compatible with what could be supplied in a kapp Config resource object passed to kapp. This means that even when using kapp deploy -a label:my-app-name=xxx a manually constructed kapp Config with the list of resources could be manually passed to kapp when run.

In the case where ConfigMap is used the sequence of events would be:

• Check to see if a ConfigMap already exists for the application (so not a new application) and extract the kapp Config snippet from the ConfigMap, incorporating this into the resources processed by kapp.
• When kapp generates resources, it generates an updated list of resources (since set of resources may change), and store the updated list back into the ConfigMap as kapp Config snippet, if changed, so it exists for the next time kapp reconciliation needs to run.
• In case where is new application, then it doesn't need to do full reconcilation (ie. query all resources in cluster by label), and so just needs to apply the resources and add the list of resources as kapp Config snippet to the initially created ConfigMap.

If this approach is taken kapp-controller would benefit without needing any changes to it.

As to use of -a label:my-app-name=xxx it would be up to the user to always provide a valid kapp Config snippet with list of resources if they want the optimisation, which they would be able to do since they have knowledge of what is being created if they are the author of the resources. The kapp Config with list of resources would be just another resource file in their set of resource files passed to kapp.

cc @jorgemoralespou

[GrahamDumpleton]

Worth noting when I first raised this issue on Slack, but which isn't noted in the issue, is how this whole problem could be getting exacerbated by the fact that kapp by defaults keeps a deployment history of size 200 (--app-changes-max-to-keep). As the number of applications deployed using kapp-controller grows then eventually for each of those you get 200 config maps stored. So if you have 100 applications, that is 20000 config maps alone which have to be processed, all directly because of the kapp defaults. Realistically I can't see any reason for the history to be more than 5-10 at most as a default.

[rohitagg2020]

Service Account Scoping - to see what permissions does SA have and how APP CR's are consuming them.
Details: Looked into APP CR. Each APP CR's has one service account associated with it. If service account has cluster wide full permissions, then we wont see any 403's. But for the SA with limited set of perms, we can see the 403's when kapp make HTTP calls. Next question is how can we limit the service account to look into the Resources where it has permissions. For this, looked into can-i. But this is again expensive as we have to specify VERB and RESOURCE.

[100mik]

Just noted that we Get() and Update() the app config map each time we update the app meta. Previously, we were making these calls while updating the app meta with UsedGVs and with LastChange details.

The GK scoping will introduce 4 more calls per run.

(The pros of GK scoping outweighs this by a landslide. Just taking note of it.)