Status of testing of Apache Airflow Helm Chart 1.19.0rc1

[jedcunningham]

We are kindly requesting that contributors to Apache Airflow Helm Chart 1.19.0rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• add readiness and liveliness support for git sync relay sidecars (#50218): @pgvishnuram
• Added support for job launcher (#52024): @avikbesu @rbm897
  Linked issues:
  • KubernetesJobOperator is not creating jobs using official chart (#50595)
• enable optional subPath for logs volume mount (#52350): @gbloisi-openaire
• Separate workers service accounts (#52357): @Miretpl
  Linked issues:
  • Introduce workers.celery and workers.kubernetes config sections (#51460)
• Add Horizontal Pod Autoscaler (HPA) Support for Airflow API Server (#52392): @carlos54 @jscheffl @HsiuChuanHsu
  Linked issues:
  • Support HPA and topologySpreadConstraints for apiServer (#51935)
• Fix KEDA Query to Use executor Field Instead of queue for Multiple Executors (#52840): @noamst-monday @HsiuChuanHsu
  Linked issues:
  • KEDA Query for multiple executors should use "executor" field instead of "queue" (#49001)
• [helm] Add missing airflow.fullname on kubernetes objects (#52953): @zyst1988 @albundy83
  Linked issues:
  • Missing webserver-secret-key-secret (#53375)
• fix: Add api-server support for k8s pod template (#53533): @LipuFei
• fix: Trim non-alphanumeric characters from the executor label (#53534): @LipuFei
• Fix Syntax Error in Airflow Job Launcher Role (#53822): @rbm897 @HsiuChuanHsu
  Linked issues:
  • Added support for job launcher (#52024)
• Allow overriding schedulerName on worker/tasks pods (#53983): @SamWheating
• update git-sync to 4.4.2 (#54085): @eladkal
• Allow setting restartPolicy for batch jobs in chart (#54354): @ihorlukianov
• fix helm schema validation for executor value (#54682): @chrisfregly-angi @chrisfregly
  Linked issues:
  • Helm Schema Validation Prevents Specifying Custom Executors (#54673)
• Add apiServer to helm chart container resources docs (#54698): @Aakcht
• helm: statsd deployment volume mount without subpath for live reloading (#54986): @MaicoTimmerman
• Make term Dag consistent in docs v2 (#55099): @jscheffl
  Linked issues:
  • Make term Dag consistent in docs (#55097)
• Update webserver secret note in Helm chart NOTES.txt and prod guide (#55106): @amordoch
• Fix KEDA query for Kubernetes Executor (#55559): @wolfdn
• Add EdgeExecutor to KEDA query (#55560): @wolfdn
• Temporary fix to Bitnami psql chart licensing issues (#55820): @amoghrajesh
• Customize statsd dagprocessor labels (#55832): @goncalo-m-c
• Create an explicit control for createUserJob (#56057): @prakharcode @markhc
  Linked issues:
  • Airflow v3 user creation command fails (#51304)
• fix(docs): handle array parameter flattening in _get_params (#56389): @prdai @jscheffl
• fix: scheduler, triggerer, worker airflow components have inconsistent log volume (#56418): @danielhoherd @rich7420
  Linked issues:
  • scheduler, triggerer, worker airflow components have inconsistent log volume (#56015)
• Add ability to disable apiserver from the helm chart using apiServer enabled (#56493): @bluek1te @jscheffl
• Fixing YAML block scalar when providing sshKey for git-sync (#56716): @RobinM-code
• Fix formatting in production-guide.rst (#56754): @Spasnof
• Move triggerer from pod-log-reader-role to pod-launcher-role (#56872): @AutomationDev85
• Extract prek hooks for Helm (#57263): @jscheffl
• Fix wrong path pattern after moving prek check to chart folder (#57311): @jscheffl
• Enable ruff PLW2101,PLW2901,PLW3301 rule (#57700): @jscheffl
• Synchronize default versions in all split .pre-commit-config.yaml (#57851): @potiuk
• Missing SCC Role bindings for redis and api-server (#57985): @ronaldorcampos
• Add registry.secretNames and registry.connections options to Helm chart (#58094): @invalid-email-address @aeroyorch
  Linked issues:
  • Support multiple image pull secrets in Helm values (#57615)
• Convert all airflow distributions to be compliant with ASF requirements (#58138): @CalvinKirs @potiuk
  Linked issues:
  • Add NOTICE File (#58131)
• Add database cleanup cronjob to clean old records (#58155): @ronaldorcampos
• Keda best practices + better documentation (#10358) (#58246): @FoxHelms
• Doc: update chart info about built-in secrets and environment variables (#58317): @Aakcht
• helm: ensure graceful redis shutdown (#58432): @arkadiuszbach
• Remove FastAPI upper constraint (#58524): @pierrejeambrun
• Chart: Enhance Celery Worker Sets support for multi-queue configurations (#58547): @glennhsh @RonaldGalea @HsiuChuanHsu
  Linked issues:
  • Deploy multiple (celery) workers with the Helm Chart (#34219)
  • Add support for multiple Celery worker groups with queue-specific configurations in Helm chart (#56591)
• Extra tests for db cleanup and fix cronjob image pull secrets (#58626): @ronaldorcampos
• Fix flower network policy condition when multiple executors (#58635): @ronaldorcampos
• Add database cleanup options and remove deprecated securityContext field (#58663): @invalid-email-address @aeroyorch
• Make cleanup-pods cronjob conditional on kubernetes executor (#58695): @ronaldorcampos
• Add database cleanup docs to Helm productions docs (#58707): @invalid-email-address @aeroyorch
• helm: correctly derive celery sync_parallelism from scheduler CPU limits (#58733): @arkadiuszbach
• Fix Liveness / Readiness / Startup probe path for Airflow 3.x #58129 (#58734): @jvoravong @jason810496 @gichul-hong
  Linked issues:
  • Liveness / Readiness / Startup probe path for Airflow 3.x (#58129)
• Helm chart: start Redis directly, not via shell (#58790): @arkadiuszbach @amordoch
  Linked issues:
  • helm: ensure graceful redis shutdown (#58432)
• Extend airflow_ti_running metrics by scheduled, queued and deferred (#58819): @AutomationDev85
• fix(helm): allow additional PodDisruptionBudget config properties (#58864): @Vishalup29
• Add pdbs for triggerer and workers (#59068): @mrrsm
• Optimize K8s API usage for watching events (#59080): @wolfdn
• Fix ingress Helm Chart notes generation (#59122): @Miretpl
• Ensure that git-sync actually runs when dags.gitSync.enabled=true and dags.persistence.enabled=true (#59123): @jstaf
• Add support for disabling SA token automount for Scheduler (#59156): @dan-osterrath @dano-nx @fsellecchia
  Linked issues:
  • Ability to set automountServiceAccountToken in values.yaml (#30722)
  • Disable automountServiceAccountToken for Kubernetes executor (#59099)
• Remove getting labels of unexisted sections (#59213): @Miretpl
• Add newsfragment for 52357 PR (#59214): @Miretpl
• [Chart] Fix rendering condition of "git_sync_ssh_key_volume" (#59418): @zezaeoh
• Add workers.celery.replicas field (#59730): @Miretpl
• [helm] Remove defaultUser of apiServer in values.schema.json (#59762): @nhuantho
• [helm] Isolate defaultUser handling in createUserJob (#59767): @nhuantho
• Chart: add custom envs to database cleanup (#59804): @Aakcht
• add redis statefulset persistentVolumeClaimRetentionPolicy support (#59955): @pgvishnuram
• Add workers.celery.revisionHistoryLimit field (#60056): @Miretpl
• Add workers.celery.command & workers.kubernetes.command (#60067): @Miretpl
• Add checksum for JWT secret in API server and scheduler deployments i… (#60111): @arjav1528 @connorwang
  Linked issues:
  • Redeployment of API server without the scheduler causes dag failure (#60040)
• Allow custom volumeClaimTemplates when logs.persistence.enabled is true (#60118): @arjav1528 @m4dm4rtig4n
  Linked issues:
  • [helm-chart] 1.16.0: Cannot add custom volumeClaimTemplates to workers (#59571)
• Remove unneeded logic in api-server-deployment.yaml (#60147): @Miretpl
• Add workers.celery.args field (#60163): @Miretpl
• Add workers.celery.livenessProbe section (#60186): @Miretpl
• Add workers.celery.persistence section (#60238): @Miretpl
• Remove unused workers.persistence.containerLifecycleHooks field (#60239): @Miretpl @rzjfr
  Linked issues:
  • Chart: add support for container lifecycle hooks (#32349)
  • Add workers.celery.persistence section (#60238)
• Add PodDisruptionBudget support for DAG Processor (#60294): @Arunodoy18
• fix(chart): allow revisionHistoryLimit to be set to 0 (#60340): @cuerty
  Linked issues:
  • revisionHistoryLimit: 0 is silently ignored due to truthy check in templates (#60339)
• New year means updated Copyright notices (#60344): @potiuk
• Add workers.celery.updateStrategy field (#60351): @Miretpl
• Add workers.celery.strategy field (#60354): @Miretpl
• Add workers.celery.podManagementPolicy field (#60359): @Miretpl
• Add workers.celery.securityContexts & workers.kubernetes.securityContexts (#60396): @Miretpl
• Fix Compatibility of Celery Worker Sets with Workers Separation (#60420): @jedcunningham @glennhsh @csp33 @Miretpl
  Linked issues:
  • Can't configure Kubernetes and Celery workers in Helm Chart (#28880)
  • Chart: Enhance Celery Worker Sets support for multi-queue configurations (#58547)
• Add workers.celery.kerberosInitContainer field (#60427): @henry3260
• Docs: Clarify ingress settings for Airflow 2 vs 3 in values.yaml (#60434): @seanfariaparetointel @vighneshtule
  Linked issues:
  • Update Helm values.yaml (#53611)
• Chart: Default airflow version to 3.1.6 (#60453): @ephraimbuddy
• Chart: fix securityContext.containers/ingress.apiServer in values.schema.json (#60575): @Aakcht
• Docs: Improve wording in Helm chart introduction (#60595): @dilshadshaik3031
• Improve dag_bundle_config_list Configuration in Airflow Helm Chart (#60645): @uplsh580
• Finish workers.celery.kerberosInitContainer & add workers.kubernetes.kerberosInitContainer (#60751): @Miretpl
• Fix missing templating in API server extraInitContainers (#60812): @subhash-0000 @thejoeejoee
  Linked issues:
  • Missing templating in api server extra init containers (#60780)
• Support PR: Add TTL-enabled LRU cache for StatsD metrics aggregation (#60933): @AutomationDev85 @shubham36deshpande
• Move core newsfragment out of chart directory (#60994): @jedcunningham
• Remove workers.celery breaking change (#61049): @Miretpl
• Fix Keycloak provider redirect_uri to use HTTPS behind reverse proxy (#61095): @arjav1528 @atalakey4work
  Linked issues:
  • Airflow Keycloak provider redirect to HTTP instead of HTTPS (#60922)
• Fix formatting & make newsfragment #60933 match changes (#61102): @Miretpl @AutomationDev85
  Linked issues:
  • Support PR: Add TTL-enabled LRU cache for StatsD metrics aggregation (#60933)
• Chart: use same bitnami postgres image as 1.18.0 (#61156): @jedcunningham @amoghrajesh
  Linked issues:
  • Temporary fix to Bitnami psql chart licensing issues (#55820)

Thanks to all who contributed to the release (probably not a complete list!):
@bugraoz93 @rich7420 @ido177 @bluek1te @jbampton @pierrejeambrun @kaxil @noamst-monday @ihorlukianov @aeroyorch @nhuantho @henry3260 @FoxHelms @RonaldGalea @LipuFei @arjav1528 @dan-osterrath @arkadiuszbach @gbloisi-openaire @ephraimbuddy @prakharcode @connorwang @Arunodoy18 @wolfdn @gichul-hong @invalid-email-address @goncalo-m-c @carlos54 @jason810496 @m4dm4rtig4n @seanfariaparetointel @amordoch @jscheffl @Vishalup29 @MaicoTimmerman @SamWheating @HsiuChuanHsu @csp33 @Miretpl @dano-nx @zezaeoh @rzjfr @jvoravong @chrisfregly @glennhsh @pgvishnuram @vighneshtule @uplsh580 @mrrsm @AutomationDev85 @subhash-0000 @dilshadshaik3031 @gopidesupavan @jstaf @fsellecchia @danielhoherd @CalvinKirs @Aakcht @prdai @cuerty @albundy83 @vatsrahul1001 @Spasnof @amoghrajesh @jedcunningham @potiuk @atalakey4work @shubham36deshpande @avikbesu @RobinM-code @ronaldorcampos @zyst1988 @rbm897 @eladkal @chrisfregly-angi @thejoeejoee @markhc @johnslavik

[chrisfregly]

Yes, issue #54673 is addressed in the latest Helm Chart RC 1.19.0rc1.

[Miretpl]

All mine looks good:

1. Separate workers service accounts #52357 / Add newsfragment for 52357 PR #59214 - reverted by Revert "Separate workers service accounts (#52357)" #60721
2. Fix ingress Helm Chart notes generation #59122
3. Remove getting labels of unexisted sections #59213
4. Add workers.celery.replicas field #59730
5. Add workers.celery.revisionHistoryLimit field #60056
6. Add workers.celery.command & workers.kubernetes.command #60067
7. Add workers.celery.args field #60163
8. Add workers.celery.livenessProbe section #60186
9. Add workers.celery.persistence section #60238
10. Add workers.celery.updateStrategy field #60351
11. Add workers.celery.strategy field #60354
12. Add workers.celery.podManagementPolicy field #60359
13. Add workers.celery.securityContexts & workers.kubernetes.securityContexts #60396
14. Finish workers.celery.kerberosInitContainer & add workers.kubernetes.kerberosInitContainer #60751
15. Remove workers.celery breaking change #61049
16. Remove unneeded logic in api-server-deployment.yaml #60147
17. Remove unused workers.persistence.containerLifecycleHooks field #60239
18. Fix Compatibility of Celery Worker Sets with Workers Separation #60420
19. Fix formatting & make newsfragment #60933 match changes #61102