[v2-11] Backport: Add environment variable controlling the log grooming frequency (#46237)

[0BVer]

Description

Backport of #46237 to v2-11.

This PR adds an environment variable AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES and a Helm chart configuration frequencyMinutes to control how often the log groomer sidecar runs.

Conflicts

Resolved conflicts in:

• chart/templates/dag-processor/dag-processor-deployment.yaml
• chart/templates/scheduler/scheduler-deployment.yaml
• chart/templates/triggerer/triggerer-deployment.yaml
• chart/templates/workers/worker-deployment.yaml
• chart/values.schema.json
• tests/charts/log_groomer.py

[jscheffl]

Note: If a patch of Airflow 2.11 is made it will only receive very critical security patches. No new features are accepted.

If you want to have that feature in, please fork the repo private or patch the generated Docker for your purpose.

The Helm-Chart is maintained on main and will be released from there, there is no 2.11 specific helm chart release. As it was merged in Chart version 1.16 you can consume the parameter already today.

Therefore closing the PR.

[0BVer]

Hi @jscheffl.

Since this change is essentially restoring a previously removed environment variable, I was under the impression that it might be acceptable for v2-11 subject to approval.

This PR re-introduces the AIRFLOW__LOG_CLEANUP_FREQUENCY_MINUTES environment variable in clean-logs.sh, which was previously missing in v2.

Currently, the Helm chart (v1.16 using Airflow v2 default) already includes this parameter, but it remains non-functional because the underlying Docker image lacks the necessary environment variable support.

While Airflow 3 is exciting, many production environments still rely on v2. This backport is essential for users who need this control but prefer to stay on the official upstream release without managing custom patches.

Also, seeing @potiuk's approval on the original PR gave me confidence that this change might be considered for v2-11 as well (#46237 (comment)).

Of course, I am well aware that adding features to older branches can complicate maintenance and security. Please know that I fully respect the project's policies, and I will gladly comply with whatever decision you make.

[potiuk]

Also, seeing @potiuk's approval on the original PR gave me confidence that this change might be considered for v2-11 as well (#46237 (comment)).

The current plan I have is that when preparing 2.11 release I will most likely use the v3-1 Dockerfile. This means that scripts and env variables will be present there. But I think it would be great if you extract just env variable and script changes that are relevant to the Dockerfile and make a PR ti 2-1-test - in case for whatever reason current 3-1 Dockerfile will not be easy to use for 2.11.