Add TimeZone support in flux controllers

[mfilotto]

Describe the bug

I tried to set TZ var env with my timezone (Pacific/Noumea) in all flux controllers with no luck

Steps to reproduce

1. Modify flux kustomization file to inject TZ var env

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
  # target all controllers to add TZ var env
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/env/-
        value:
          name: TZ
          value: "Pacific/Noumea"
    target:
      kind: Deployment
      labelSelector: "app.kubernetes.io/part-of=flux"

2. Then read the logs of the new pods to see if timezone has changed

Expected behavior

Flux controllers should log with the expected timezone

Screenshots and recordings

No response

OS / Distro

CentOS 7.9

Flux version

v0.36.0

Flux check

flux check
► checking prerequisites
✔ Kubernetes 1.24.7 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.26.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.30.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.28.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.31.0
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[pjbgf]

I see three ways around this:

1. Support this at container image level, by installing the tzdata package as part of the official Flux container images.
2. Support this at application level (via time.LoadLocation() in combination with a new flag --timezone), so if we ever start shipping controllers using distroless/scratch base images, we would continue to support this feature.
3. Do not support this, and ask users to rely on their log ingestion tools to account for timezones at processing/ingestion time.

I would support the first two approaches. But it would be good to get some more @fluxcd/core-maintainers views on it.

[makkes]

I'd prefer option 1 so we don't have to change any code to enable logging timestamps in another timezone. If we ever decided to use distroless (which I'm a huge supporter of), we could just ship the tz data files ourselves.

[mfilotto]

There is an old issue on this topic
#460 Support setting timezone in controller images

[mfilotto]

Any update on this one please ?

[pjbgf]

An additional way to achieve this is to build the controllers with -tags timetzdata or adding an import to time/tzdata. As per
https://pkg.go.dev/time/tzdata.