Status of testing of Apache Airflow Helm Chart 1.16.0rc1

[jedcunningham]

We are kindly requesting that contributors to Apache Airflow Helm Chart 1.16.0rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• enable AIRFLOW__CELERY__BROKER_URL_CMD when passwordSecretName is true (#40270): @jx2lee @chrisluedtke
  Linked issues:
  • Can't use AIRFLOW__CELERY__BROKER_URL_CMD in the Helm Chart (#39985)
• Update metrics names to allow multiple executors to report metrics (#40778): @syedahsn
• Chart: add a specific internal IP address for the ClusterIP service (#40912): @alizmhdi @fbertos
  Linked issues:
  • Allow config a static ip for the pgbouncer clusterip service (#40676)
• fix: Fixing Helm chart flower ingress service reference (#41179): @andormarkus
  Linked issues:
  • [helm] - Flower ingress incorrect when fullnameOverride applied (#41175)
• Properly implement termination grace period seconds (#41374): @eakmanrq @jedcunningham
• update git-sync to 4.3.0 (#41411): @JKrehling
• Support template hostname in NOTES.txt (#41423): @romsharon98
• Upgrade build and chart dependencies (#41570): @potiuk
• [redis]adding additional values for existing persistence claim (#41619): @kongdewen
• Fix pod-template-file when kerberos enabled (#41645): @romsharon98
• Add hostAliases support for Triggerer in helm chart (#41725): @HassanAlahmed
• Feature/add pgbouncer and statsd ingress (#41759): @romsharon98
• Consolidated fix for volumeClaimTemplates (apiVersion and PVC) (#41771): @darth-drew
• add flexibility for redis service (#41811): @kongdewen
• Enable HPA for Airflow Webserver (#41955): @JoaVirtudes19
• Update StatsD Image Tag from failed dependencies check (#42264): @jscheffl
• fix(helm): render global volumes and volume mounts into cleanup job (#40191) (#42268): @iiskandarov-clgx @dada-engineer
  Linked issues:
  • Include .Values.volumeMounts into chart/templates/cleanup/cleanup-cronjob.yaml (#40191)
• add env support for migratedatabase job (#42345): @pgvishnuram
• Rebuild pgbouncer image with newer version (#42349): @robertvolkmann
• Add heartbeat metric for DAG processor (#42398): @jedcunningham @rawwar
  Linked issues:
  • Add heartbeat metric for DAG processor (#42380)
• feat: allow customizing podManagementPolicy (#42673): @EvertonSA
• Updating pre-commit hook and prometheus statsd exporter versions from failed dependency check (#42809): @gopidesupavan
• Chart: Fix VCT for scheduler in local and persistent mode (#42946): @darth-drew @Aakcht
  Linked issues:
  • Consolidated fix for volumeClaimTemplates (apiVersion and PVC) (#41771)
• Upgrade statsd image in k8s to latest version (#43393): @potiuk
• Support multiple executors in chart (#43606): @LipuFei @jx2lee @HadarSha @rcheatham-q
  Linked issues:
  • Helm chart does not support multiple executors (#43224)
  • Chart: Allow hybrid executors (#41524)
  • Hybrid Executor Aliasing raises exception (#41797)
• Update helm chart docs for private registry params (#43721): @omkar-foss
• Add newsfragment PR 43393 (#44091): @Lee-W
• Remove scheduler automate serviceaccount token (#44173): @romsharon98
• Add tests cases for multiple executors in chart (#44424): @romsharon98
• Swap internal RPC server for API server in the helm chart (#44463): @ashb
  Linked issues:
  • Convert the LocalExecutor to run tasks using new Task SDK supervisor code (#44427)
• Set container name to envSourceContainerName in KEDA ScaledObject (#44963): @jx2lee @yovio-rca
  Linked issues:
  • Keda does not work properly when using gitSync (#44798)
• Helm Chart: Add startupProbe to flower deployment (#45012): @topherinternational @MWatter @jedcunningham
  Linked issues:
  • Helm Chart: Add startupProbe to flower deployment (#44846)
• Chart: add OpenSearch remote logging options (#45082): @topherinternational @Owen-CH-Leung
  Linked issues:
  • Add feature to read log from opensearch (#41799)
• Chart: add info that storageClassName can be templated (#45176): @romsharon98 @Aakcht
• More controls for pgbouncer secrets configuration (#45248): @andrii-korotkov-verkada
  Linked issues:
  • Support not generating secrets at all, giving an option to rely on other means to get secrets (#45171)
• change doc for reinitFrequency in chart (#45343): @romsharon98
• Upgrade pgBouncer version to 1.24.0 (#45542): @goktugkose
• Use standalone dag processor for AF3 in chart (#45659): @jedcunningham
• allow passing custom env to log groomer sidecar containers (#46003): @pgvishnuram
• Update scheduler-deployment to cope with multiple executors (#46039): @krisgeus @romsharon98
• feature: Add environment variable controlling the log grooming frequency (#46237): @stefankeidel
• Rename apiServer to _apiServer to preserve it in beta until v3 release (#46477) (#46480): @andrii-korotkov-verkada @jedcunningham
  Linked issues:
  • Move api server in helm chart back to beta with _ before it (#46477)
• Chart: be consisent with denoting templated params (#46481): @jedcunningham
• Fix scheduler ServiceAccount automount for multi-executor (#46486): @jedcunningham
• Chart: Default airflow version to 2.10.5 (#46624): @utkarsharma2
• Chart - Introduce worker.extraPorts to expose additional ports to worker container (#46679): @Jorricks
• Replace disallowed characters in metadata label (#46811): @krisgeus @alecsg77
  Linked issues:
  • Helm chart fails with multiple executors (#46733)
• Remove old UI and webserver (#46942): @jscheffl @vincbeck @pierrejeambrun @vatsrahul1001 @bbovenzi @ashb @gopidesupavan @jedcunningham @amoghrajesh @uranusjr @potiuk
• Move execution_api_server_url config to the core section (#46969): @jedcunningham
• Move fastapi-api command to api-server (#47076): @jedcunningham
• Start the api-server in AF3, webserver in AF2 (#47085): @jedcunningham
• Add optional configuration for startupProbe initialDelaySeconds (#47094): @oryanperl
• [Helm Chart] Grant Airflow API Server Permission to Read Pod Logs (#47212): @vatsrahul1001
• Fix typo in helm-chart values schema (#47269): @stephsi
• Move api-server to port 8080 (#47310): @jedcunningham
• Update pgbouncer to 1.23.1 and pgbouncer_exporter to 0.18.0 in helm chart (#47416): @mjmammoth @potiuk @Jarvl
  Linked issues:
  • Pgbouncer exporter doesn't support metrics exposed by updated pgbouncer. (#47295)
• AIP-84 Move public api under /api/v2 (#47760): @prabhusneha @jedcunningham
• [Metric] New ti.running metric with queue field label (#47773): @AutomationDev85
• docs: enhance airflowLocalSettings value description (#47855): @timonviola
  Linked issues:
  • airflow_local_settings.py overriden as an empty string (#47789)
• Default to the FabAuthManager in the chart (#47976): @jedcunningham
• Changing default_capacity to capacity for issue 47800 (#48032): @RNHTTR @jroachgolf84
  Linked issues:
  • Rename the triggerer's default_capacity config to just capacity (#47800)
• Release notes for helm chart 1.16.0 (#48525): @jedcunningham

Thanks to all who contributed to the release (probably not a complete list!):
@HadarSha @alizmhdi @MWatter @Jarvl @omkar-foss @pierrejeambrun @romsharon98 @vatsrahul1001 @jx2lee @LipuFei @stephsi @dada-engineer @robertvolkmann @RNHTTR @amoghrajesh @kongdewen @Owen-CH-Leung @andrii-korotkov-verkada @andormarkus @alecsg77 @goktugkose @JKrehling @iiskandarov-clgx @rawwar @vincbeck @utkarsharma2 @AutomationDev85 @prabhusneha @bbovenzi @JoaVirtudes19 @jroachgolf84 @yovio-rca @eakmanrq @Aakcht @uranusjr @topherinternational @potiuk @Lee-W @jscheffl @fbertos @HassanAlahmed @jedcunningham @darth-drew @timonviola @krisgeus @pgvishnuram @Jorricks @oryanperl @syedahsn @gopidesupavan @ashb @mjmammoth @ephraimbuddy @chrisluedtke @rcheatham-q @EvertonSA @bugraoz93 @stefankeidel

[jscheffl]

I checked for #42264 which was super-seeded by #43393 and this is in.

Also the change in #46942 is correctly included.

[alecsg77]

I checked #46811 linked to #46733 issue, and it is solved.
I first deployed a 1.15.0 chart with CeleryExecutor, then upgraded to 1.16.0rc1 with CeleryExecutor and KubernetesExecutor.
All good.

[stephsi]

I checked if #47269 is addressed in the latest RC and it is.

[amoghrajesh]

I didnt have any particular change but checked that stuff works as expected generally by installing the helm chart with Kind. Looks good

[EvertonSA]

Hi I also confirms the change I have applied on #42673 works when installing helm from scratch (not on existing airflow installation). but i haven't done unit testing. Can you let me know if unit test needed for my change?

[jx2lee]

I checked below PRs, looks good
#40270
#43606
#44963
#43606
#44963

[Owen-CH-Leung]

I checked #41799 and it looks good to me

[jedcunningham]

Hi I also confirms the change I have applied on #42673 works when installing helm from scratch (not on existing airflow installation). but i haven't done unit testing. Can you let me know if unit test needed for my change?

Adding a unit test would be great! But, it won't block this release. You can read more about them in the helm chart testing docs.

[potiuk]

All my changes are in. Run the chart and it works nicely and installs airflow 2.10.5 and other latest images.

[jedcunningham]

Thanks for testing the RC @jscheffl, @alecsg77, @stephsi, @amoghrajesh, @EvertonSA, @jx2lee, @Owen-CH-Leung, and @potiuk. The release is underway!