SDD 0031: Central Component Version tracking (#177)

Co-authored-by: Simon Gerber <simon.gerber@vshn.ch>

docs/modules/SDDs/pages/0031-component-version-tracking.adoc

@@ -0,0 +1,172 @@
+= SDD 0031 - Central Component Version tracking
+
+:sdd_author:    Sebastian Widmer
+:sdd_owner:     Project Syn IG
+:sdd_reviewers: Simon Gerber
+:sdd_date:      2024-05-13
+:sdd_status:    pending
+
+include::partial$meta-info-table.adoc[]
+
+[NOTE]
+.Summary
+====
+This describes how we want to extend the Lieutenant API, CRD, and operator to allow for central component version tracking.
+====
+
+== Motivation
+
+Currently, component versions used in a Commodore compile are tracked in the cluster catalog repository.
+They are stored in the commit message.
+
+This approach has several drawbacks:
+* The version information is not easily accessible programmatically.
+* To get an overview one needs to find every repository and check the commit history.
+
+To improve this situation, we want to introduce a central component version tracking system.
+The system should be centralized and accessible programmatically.
+
+Lieutenant is the central component for managing the cluster catalogs and already has a REST API.
+It is backed by a CRD which already stores some state information and cluster metadata.
+
+We want to extend the Lieutenant API, CRD, and operator to store version information for each Commodore compile.
+
+=== Goals
+
+* Enable a central, programmatically accessible component and configuration repos (global, tenant) version tracking system
+* Allow Grafana or other monitoring tools to display component version information
+
+=== Non-Goals
+
+* Show class from which the version is derived
+* Show the version of the images, charts, or libraries used in the component
+
+== Design Proposal
+
+=== CRD
+
+We add a new field to the `Cluster` CRD, under the `.status` key, called `compileMeta`.
+
+The `compileMeta` field lists all component versions under the `instances` key.
+The key is the name of the component instance, the value contains the name of the component.
+With aliased instances the component name can be different from the instance name.
+Aliased instances are defined using `alias as component` in the Commodore configuration.
+See xref:reference/architecture.html#_component_instantiation[Component instantiation and aliasing] for more information.
+Currently Commodore only supports one version per component, but this structure allows for reporting multiple versions per component in the future.
+
+Configuration packages are tracked under the `packages` key.
+
+The `compileMeta` field also contains a timestamp of the last Commodore compile under the `lastCompile` key.
+Additionally it tracks the configuration repository versions under the `global` and `tenant` key.
+
+Each version entry contains the following fields:
+
+* `url`: The URL of the component repository
+* `version`: The configured version of the component
+Can be a branch name, tag, or commit hash.
+The `version` can point to different commits depending on when the compile was done.
+* `gitSha`: The commit hash of the commit referenced by the `version` field
+With the `gitSha` and the `url` it is possible to uniquely identify the commit.
+
+The `packages` and `instances` fields additionally contain:
+
+* `path`: The path in the repository where the package is located.
+This field is optional.
+
+The `instances` field additionally contains:
+
+* `component`: The name of the component, might be different from the instance name if the instance is aliased.
+
+[source,yaml]
+----
+apiVersion: syn.tools/v1alpha1
+kind: Cluster
+metadata:
+  name: my-cluster
+status:
+  compileMeta:
+    lastCompile: "2024-05-13T12:00:00Z"
+    packages:
+      appcat:
+        url: https://github.com/vshn/component-appcat
+        path: package
+        version: master
+        gitSha: "125a78afd733f47d50d3eee9cd7c5bcaca2282b2"
+    instances:
+      alerts-exporter:
+        component: alerts-exporter
+        url: https://github.com/appuio/component-alerts-exporter.git
+        version: "master"
+        gitSha: "3244ddb83a91279fb378f011358ac118987747a2"
+      keycloak-prod:
+        component: keycloak
+        url: https://github.com/projectsyn/component-keycloak.git
+        version: v15.0.0
+        gitSha: "008b8d1ad07001eec17c197270d5667b7ae66d98"
+    global:
+      url: git.syn.tools/my-org/global-config.git
+      version: "master"
+      gitSha: "a64d207a11bb595702470917beffed9df0227a36"
+    tenant:
+      url: git.syn.tools/chewserver/syn-tenant-repo.git
+      version: "master"
+      gitSha: "4e6699968153c608f048b140f6af351816b14303"
+----
+
+=== API
+
+The Lieutenant API will be extended to have a push endpoint `POST /clusters/{clusterId}/compileMeta` with the same token authentication as the current API.
+The API accepts a JSON payload with the contents of the `compileMeta` field and handles the update of the `Cluster` CRD.
+
+The CRD update should be done with server side merge to avoid frequent API errors due to concurrent, unrelated updates with the operator or other clients.
+
+=== Operator
+
+The Lieutenant operator will be extended to create Prometheus / OpenMetrics metrics for the component versions.
+The metrics should contain all the information from the `compileMeta` field.
+
+[source]
+----
+syn_lieutenant_cluster_compile_meta_last_compile{cluster="my-cluster"} 1.624e+12
+syn_lieutenant_cluster_compile_meta_package{cluster="my-cluster", name="appcat", url="https://...", version="master", path="package", gitSha="..."} 1
+syn_lieutenant_cluster_compile_meta_instance{cluster="my-cluster", name="alerts-exporter", component="alerts-exporter", url="https://...", version="master", gitSha="..."} 1
+syn_lieutenant_cluster_compile_meta_instance{cluster="my-cluster", name="keycloak-prod", component="keycloak", url="https://...", version="v15.0.0", gitSha="..."} 1
+syn_lieutenant_cluster_compile_meta_global{cluster="my-cluster", url="https://...", version="master", gitSha="..."} 1
+syn_lieutenant_cluster_compile_meta_tenant{cluster="my-cluster", url="https://...", version="master", gitSha="..."} 1
+----
+=== Commodore
+
+Commodore will be extended to send updated compilation information for catalog compilations which actually push a new catalog commit.
+Any other catalog compilations will not update the compile metadata on the Lieutenant API.
+
+Commodore will use its existing API token to push the compilation metadata to Lieutenant.
+
+=== Implementation Details/Notes/Constraints
+
+Different versions for different component instances::
+https://github.com/projectsyn/commodore/issues/563[projectsyn/commodore#563] wants to allow different versions for different component instances.
+The current design with instance name as the top key allows for this.
+
+Configuration packages::
+Configuration package versions are not yet added to the commit message.
+We need to add them to the commit message and the Lieutenant API.
+
+== Alternatives
+
+=== Store version information on the cluster and use metric forwards
+
+VSHN has a centralized metrics system and all VSHN managed clusters send metrics to it.
+
+Instead of storing the version information in the Lieutenant CRD, we could store it in a config map in the cluster.
+Steward or kube-state-metrics could then make the information available as metrics.
+
+This approach requires a lot of additional setup for Commodore users without a centralized metrics system.
+It also requires the cluster to be able to reach the metrics system.
+
+We want to avoid this approach since it removes some of the batteries included philosophy of Commodore.
+
+== References
+
+* https://github.com/projectsyn/commodore/issues/389[projectsyn/commodore#389 Component version reporting]
+* https://github.com/projectsyn/commodore/pull/388[projectsyn/commodore#388 Draft PR]
+* xref:reference/architecture.html#_component_instantiation[Component instantiation and aliasing]