Status of testing of Apache Airflow 3.0.5rc1 & Task SDK 1.0.5rc1

[kaxil]

We are kindly requesting that contributors to Apache Airflow RC 3.0.5rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Add ordering to AssetEvent query in SchedulerJobRunner (#52231): @dingo4dev
  Linked issues:
  • Incorrect ordering of events in triggering_asset_events (#52230)
• Allow DEFAULT_QUEUE to be configurable from airflow settings in Task SDK (#52786): @dabla
• Fix task configuration defaults for AbstractOperator (#52871): @kaxil
• Allow Variable and XCom set and delete from trigger code (#53514): @karenbraganz
• Fixed Task group names duplication in Task's task_id for MappedOperator (#53532): @evgenii-prusov @kacpermuda @Lee-W
  Linked issues:
  • Task group names are duplicated in Task's task_id for MappedOperator (#52334)
• Restore get_previous_dagrun functionality for task context (#53655): @kaxil @cristian-fatu
  Linked issues:
  • DagRun.get_previous_dagrun() was removed (#53628)
• Restore proper DAG callback execution context (#53684): @kaxil @kiran2706 @cardilloscreations @jnstockley @benitezfede
  Linked issues:
  • Callback context is missing the most useful attributes for notifications (#52824)
  • context doesn't contain TaskIntance when used in on_*_callback at the DAG level (#51402)
  • Feature(call-back): Enhance DAG-level callback context with enriched metadata and test coverage (#51949)
  • Fix DAG callbacks missing dag_run in context (#53654)
• Do not ignore include_prior_dates in xcom_pull when map_indexes is not specified (#53809): @amoghrajesh
  Linked issues:
  • Improve xcom_pull to cover different scenarios for mapped tasks (#51568)
• Fix custom xcom backend deserialize when BaseXCom.get_all is used (#53814): @amoghrajesh
• [v3-0-test] Fix dag_versions property when created_dag_version is None with bundle_version set (#54010): @vatsrahul1001
• Fix type error with TIH when reading served log (#54114): @yaming-github
• [v3-0-test] Do not include GitHub bots in release test requests (#54137) (#54150): @amoghrajesh
• [v3-0-test] Add FAQ entry about testing connections and "Canary" Dag (#54110) (#54151): @fritz-astronomer
• [v3-0-test] Set minimum version for common.messaging to 1.0.3 (#54160) (#54176): @potiuk
  Linked issues:
  • Move MESSAGE_QUEUE_PROVIDERS array to where it belongs - to msq_queue (#51774)
• [v3-0-test] Add IP validation to example_dag_decorator DAG (#54136) (#54208): @Nacl122
• [v3-0-test] Fix: Apply DAG permission filter to dashboard (closes #53938) (#54126) (#54215): @suhail-zemoso
• [v3-0-test] Fix: Validate and handle invalid extra field in connections UI and … (#54235): @pierrejeambrun @Prasanth345
• [v3-0-test] Connection Extra additional test case for validation (#54239) (#54244): @pierrejeambrun
• fix map_index is null dynamic task mapping bug (#54249): @MarcusCramer91 @isaiahiruoha
  Linked issues:
  • map_index = null for passed parameters in tasks in child task group when parent task group is expanded (#52881)
• [v3-0-test] Fix redirects (#54238) (#54251): @gopidesupavan
• [v3-0-test] fix(UI): Correct illegible DAG params text in dark mode (#54245) (#54260): @RoyLee1224
• [v3-0-test] Fix reproducibility issue for tarballs with symlink (#54264) (#54267): @potiuk
• [v3-0-test] Fix for Edit and Delete request issue 53681 (#53815) (#54268): @mandeepzemo
• [v3-0-test] Fetch from remote before performing reproducible package checks for PMCs (#54206) (#54275): @amoghrajesh
• [v3-0-test] Validate publish documentation versions (#54272) (#54282): @gopidesupavan
• [v3-0-test] Fix cherry-pick detection in airflow-github script (#54287) (#54289): @kaxil
• [v3-0-test] Massively optimize airflow-github script performance (96% faster) (#54292) (#54293): @kaxil
• [v3-0-test] Fix redirects repository urls, use head repo when running PR workflows (#54301) (#54303): @gopidesupavan
• CI: Bump uv, pre-commit and golang versions (#54304): @gopidesupavan
• [v3-0-test] Use env var connections instead of mocks in kube system tests (#54261) (#54320): @amoghrajesh
• [v3-0-test] Initialize custom variables before running db-reset (#54324) (#54327): @potiuk
• [v3-0-test] Bump uv and pre-commit versions (#54304) (#54328): @gopidesupavan
• [v3-0-test] Convert auth manager into cachable flag and configure it for more com… (#54330): @potiuk
• Prevent DetachedInstanceError when processing executor event (#54334): @kaxil
• Fix DetachedInstanceError when accessing DagRun.created_dag_version (#54362): @kaxil
• [v3-0-test] Fail upgrade pre-commit when pre-commits need to be updat… (#54373): @potiuk
• [v3-0-test] Upgrade to latest Lucas Pre-commit (#54374) (#54375): @potiuk
• [v3-0-test] Use env var connections instead of actual db connections in kube systests (#54385) (#54391): @amoghrajesh
• [v3-0-test] Allow downgrading to 2.11 from 3.x (#54371) (#54399): @ashb
  Linked issues:
  • Create FAB's user/role tables on migration, not only on initdb (#54227)
• [v3-0-test] Allow failure callbacks for stuck in queued TIs that fail (#53435) (#54401): @karenbraganz @kaxil @dimberman @ashb @KarthikeyanDevendhiran
  Linked issues:
  • Allow for retry when tasks are stuck in queued (#43520)
  • Allow failure callbacks for stuck in queued TIs that fail (#53435)
  • Issue with task_queued_timeout Causing Silent Task Failures in Airflow 2.10.5 (#51301)
• [v3-0-test] Skip running db queries for Spans when they are disabled (#54075) (#54404): @Shlomit-B
• [v3-0-test] Migrate HiveToDynamoDBOperator and SqlToS3Operator to use get_df (#50126) (#54409): @guan404ming @ashb
• [v3-0-test] fix: corrects issues with edge rendering on the graph view (#53998) (#54412): @codecae
• [v3-0-test] Improve condition for pushing the cache for images (#54415) (#54418): @potiuk
• TaskName indentation limited by available --chakra-spacing vars (#54419): @codecae
• [v3-0-test] Update selective checks to run core tests when serialization files changes (#54423) (#54426): @gopidesupavan
• [v3-0-test] Fix Certain DAG import errors ("role does not exist") don… (#54432): @sjyangkevin
• [v3-0-test] Prevent repeated warning of triggers being added twice in triggerer (#54438) (#54441): @ephraimbuddy
• [v3-0-test] Explicitly set the JWTGenerator private_key field to None for service logs retrieval (#54442) (#54444): @ianbuss
• [v3-0-test] Ensure TaskInstance.end_date and duration are populated before invoking failure callbacks (#52729) (#54458): @Programmer-RD-AI
• [v3-0-test] Handle overflow in TaskInstance next_retry_datetime (#48557) (#54460): @perry2of5
• [v3-0-test] Refactor example task to use context and logging for parameter access (#54450) (#54463): @Programmer-RD-AI
• [v3-0-test] Hide individual Public Interface subsections from sidebar navigation (#54457) (#54465): @sunank200
• Bump zizmor pre-commit to fix static checks (#54468): @kaxil

Thanks to all who contributed to the release (probably not a complete list!):
@karenbraganz @kiran2706 @Shlomit-B @Lee-W @cardilloscreations @MarcusCramer91 @isaiahiruoha @evgenii-prusov @sjyangkevin @cristian-fatu @gopidesupavan @dabla @codecae @ianbuss @mandeepzemo @guan404ming @ashb @kacpermuda @kaxil @potiuk @dimberman @benitezfede @vatsrahul1001 @dingo4dev @perry2of5 @suhail-zemoso @fritz-astronomer @Nacl122 @KarthikeyanDevendhiran @Programmer-RD-AI @sunank200 @pierrejeambrun @ephraimbuddy @Prasanth345 @amoghrajesh @jnstockley @RoyLee1224 @yaming-github

[kacpermuda]

#53532 is ok.

[kaxil]

📣 Apache Airflow 3.0.5rc2 & Task SDK 1.0.5rc2 are now available for testing.

Install using the following command:

uv venv
uv pip install -U apache-airflow==3.0.5rc2 --pre

Docs | Release notes | More details (Mailing list thread)

[RoyLee1224]

#54260 is fine.

[codecae]

#54412 and #54419 lgtm!

[sjyangkevin]

#54432 is good.

Test Process

breeze start-airflow --use-airflow-version 3.0.5rc1 --python 3.10 --backend postgres --db-reset --auth-manager FabAuthManager

Setting access control for DAG with a non-existing role will raise an import error. This import error will not be removed while DAG parsing. DAG processor log will also inform this error.

[dingo4dev]

#52231 lgtm

[Shlomit-B]

In #54404, the main functions are skipped as expected.
_recreate_unhealthy_scheduler_spans_if_needed still runs and performs DB queries, though span creation is a no-op. I’ll double-check whether these queries are necessary and open a small follow-up PR if needed.

[prdai]

#54463 and #54458 is fine

[yaming-github]

#54114 is good

[Lee-W]

Verified #53532 and a few dags. work fines

[jnstockley]

#51402 looks good to me

[potiuk]

Checked all my changes -> most CI, some dependency changes are OK. Reproducibility works now.

[gopidesupavan]

My changes related to CI, executed a few example dags LGTM.