feat: Automatic CHI reconciliation on operator secret changes

[kamushadenes]

This PR introduces a mechanism to automatically reconcile all ClickHouseInstallations (CHIs) when the operator's configured Kubernetes Secret (specified in config.yaml under clickhouse.access.secret) is updated.

Key Changes:

• Secret Watcher & Configuration Update:

  • The ConfigManager now includes a watcher for the configured ClickHouse access secret (pkg/chop/config_manager.go).
  • When the secret is added, updated, or deleted, the ConfigManager updates its internal configuration (username, password) and calculates a SHA256 hash of the secret's data.
  • This hash (CHOPSecretHash) is stored in the OperatorConfig (pkg/apis/clickhouse.altinity.com/v1/type_configuration_chop.go).

• CHI Reconciliation Trigger:

  • A callback mechanism is introduced in ConfigManager that gets invoked when the CHOPSecretHash changes (pkg/chop/config_manager.go).
  • This callback triggers the reconcileAllCHIsOnConfigChange function (cmd/operator/app/thread_chi.go).
  • This function iterates through all CHIs in the watched namespaces and performs the following to trigger a reconciliation:
    • Adds/Updates an annotation internal.altinity.com/chop-secret-hash on the CHI's metadata and its pod templates with the new chopSecretHash.
    • Increments the CHI's .metadata.generation.
  • These changes ensure that the CHI controller detects a change and re-processes the CHI, applying the updated secret credentials.

• Action Plan Update:

  • The ActionPlan (pkg/model/common/action_plan/action_plan.go) now also considers changes in annotations when determining if there are actions to be performed. This is necessary for the annotation-based reconciliation trigger to work correctly.

• Controller Enhancement:

  • The CHI controller (pkg/controller/chi/controller.go) is now equipped with a chiLister for efficient access to CHI resources when reconciling all CHIs.

Benefits:

• Ensures that ClickHouse clusters managed by the operator promptly pick up changes to their access credentials or other configurations stored in the designated secret, without requiring manual intervention or operator restarts.
• Improves the security and manageability of the operator by allowing dynamic updates to sensitive information. That allows, for example, the use of external-secrets.

How to Test:

1. Configure the operator to use a Kubernetes secret for ClickHouse access credentials in its config.yaml.
2. Deploy the operator and a CHI.
3. Modify the data (e.g., password) in the configured Kubernetes secret.
4. Observe that the operator logs indicate a configuration change and trigger reconciliation for the CHI.
5. Verify that the CHI's pods are eventually updated/restarted to use the new credentials (depending on how the CHI is configured to use the secret).

Caveats

For this reconciliation to happen, the ClickHouse pods needs to be restarted. A future update might change that.

[alex-zaitsev]

This is a major behaviour change, so we will move it to 0.26

[kamushadenes]

Let's close that for now as we went another route.