Add observability agent application for clusters

[stevencrake-nscale]

This PR introduces observability agent support for Kubernetes clusters, enabling telemetry collection and forwarding to the centralised observability platform.

The approach taken is a temporary™️ one to getting the agent on customer clusters. An ADR was written detailing this decision, here. We will review the approach in future.

Caveat for testing: right now the chart requires a manual application of the aggregator basic auth secret until mTLS is added in the coming week.

Summary

• Adds new observabilityAgent feature flag to KubernetesClusterSpec
• Creates observability-agent HelmApplication definition for k8s-deploy-observability-agent chart
• Adds new ApplicationBundle version 1.4.0 with observability agent included
• Integrates observability agent provisioning into cluster add-ons workflow
• Provides conditional deployment (opt-in) based on cluster spec features provided in the request

[stevencrake-nscale]

will be tagged when an RC is ready

[stevencrake-nscale]

we don't have a clear way to obtain the currently expected values for this required value.

the value expects e.g.

• glo1
• sta1

the argument for setting it to nks is that it clearly identifies telemetry coming from an NKS cluster via the observability agent

[stevencrake-nscale]

if we standardise all of our clusters to use the canonical region name such as no-glo1-dev, no-glo1, no-sta1 etc. then we could remove the environment key altogether.

[squaremo]

This PR introduces observability agent support for Kubernetes clusters, enabling telemetry collection and forwarding to the centralised observability platform.

I am on board with the idea of optionally setting up telemetry for my clusters. Say I have a Prometheus running centrally, how do I tell this agent to send metrics to it?

[stevencrake-nscale]

This PR introduces observability agent support for Kubernetes clusters, enabling telemetry collection and forwarding to the centralised observability platform.

I am on board with the idea of optionally setting up telemetry for my clusters. Say I have a Prometheus running centrally, how do I tell this agent to send metrics to it?

@squaremo
Right now the chart takes a single otel endpoint, we've discussed receiving an array, with our one provided by default and allowing users to provide additional

https://github.com/nscaledev/k8s-deploy-observability-agent/blob/e8617da5033768d04dbd6ffac64edf1375744e5d/charts/observability-agent/templates/collector.yaml#L115-L119

[squaremo]

Right now the chart takes a single otel endpoint, we've discussed receiving an array, with our one provided by default and allowing users to provide additional

I don't think this is a good idea. That would mean the chart, and this component, are specialised to our deployment by default -- which is wrong in principle (to wit: "don't give open things proprietary dependencies"); but also in practice an avoidable pain, for e.g., local deployments. Better to default it to empty and provide the deployment particulars in our own manifests.