Status of testing of Apache Airflow 2.8.1rc1

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 2.8.1rc1 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Add log lookup exception for empty op subtypes (#35536): @vchiapaikeo
• Create latest log dir symlink as relative link (#36019): @Kache
• Fix Python-based decorators templating (#36103): @potiuk
  Linked issues:
  • Using requirements file in VirtualEnvPythonOperation appears to be broken (#36102)
• Do not let EventsTimetable schedule past events if catchup=False (#36134): @avkirilishin
• Add test to evaluation of tasks in mapped task group (#36149): @avkirilishin
• Bugfix/logging for pausing (#36182): @alephmelo @romsharon98
  Linked issues:
  • Fix action logging decorator for un/pausing events (#33713)
• Kubernetes executor running slots leak fix (#36240): @llamageddon83 @dirrao @crabio
  Linked issues:
  • Airflow progressive slowness (#32928)
  • Leak in Kubernetes Executor Running Tasks Slot Count (#35675)
  • Airflow Scheduler with Kubernetes Executor has errors in logs and stuck slots with no running tasks (#36478)
• enable mark task as failed/success always (#36254): @romsharon98 @internetcoffeephone
  Linked issues:
  • "Mark state as..." button options grayed out (#36219)
• Stop deserializing pickle when enable_xcom_pickling is False (#36255): @hussein-awala
• Replace deprecated get_accessible_dag_ids and use get_readable_dags in get_dag_warnings (#36256): @hussein-awala
• Check DAG read permission before accessing DAG code (#36257): @hussein-awala
• Upgrade to just released pip 23.3.2 (#36271): @potiuk
• Bugfix: Webserver returns 500 for POST requests to api/dag/*/dagrun from anonymous user (#36275): @tullis @eliskovets @pateash
  Linked issues:
  • Anonymous users are no longer able to add DAG run notes, even with the ADMIN role (#36206)
  • Webserver returns 500 for POST requests to api/dag/*/dagrun from anonymous user (#36110)
• Increase width of execution_date input in trigger.html (#36278): @HurSungYun
• Use batch_is_authorized_dag to check if user has permission to read DAGs (#36279): @hussein-awala @vincbeck
  Linked issues:
  • Check DAG read permission before accessing DAG code (#36257)
• Add support of Pendulum 3 (#36281): @Taragolis
• Remove common.io from chicken-egg providers. (#36284): @potiuk
• Straighten typing in workday timetable (#36296): @uranusjr
• Fix AirflowSkipException message raised by BashOperator (#36354): @dolfinus
• Allow PythonVirtualenvOperator.skip_on_exit_code to be zero (#36361): @dolfinus
• fix datetime reference in DAG.is_fixed_time_schedule (#36370): @joshowen
• Stop serializing timezone-naive datetime as a timezone-aware datetime with UTC tz (#36379): @Lee-W @hussein-awala
  Linked issues:
  • handle tzinfo in S3Hook.is_keys_unchanged_async (#36363)
• Fix the required access for get_variable endpoint (#36396): @hussein-awala
• Add flask config: MAX_CONTENT_LENGTH (#36401): @yo1956
• Make kubernetes decorator type annotation consistent with operator (#36405): @Taragolis
• Remove redundant docker decorator type annotations (#36406): @Taragolis
• Follow BaseHook connection fields method signature in FSHook (#36444): @Taragolis
• Fix get_leaves calculation for teardown in nested group (#36456): @dstandish @ardubev16
  Linked issues:
  • Setup/Teardown in nested Task Groups creating unwanted dependencies with downstream tasks (#36345)
• Remove pypitest from the release process (#36466): @potiuk
• fix: Repair incorrect display of the deferrable operators in the documentation (#36476): @kacpermuda
• Add zlib1g-dev package to Airflow images (#36493): @potiuk
• Add pre-commit check to check integrations table. (#36497): @timonviola
• Refactor _manage_executor_state by refreshing TIs in batch (#36502): @hussein-awala
• Announce MSSQL support end in Airflow 2.9.0, add migration script hints (#36509): @jscheffl
  Linked issues:
  • Remove MSSQL support from Airflow core (#35868)
• Raise error when DagRun fails while running dag test (#36517): @kaxil
• Remove option ot set a task instance to running state in UI (#36518): @jscheffl
• Fix details tab not showing when using dynamic task mapping (#36522): @shohamy7 @csp33
  Linked issues:
  • Empty Details tab after upgrading to Airflow 2.8.0 (Dynamic Task Mapping) (#36481)
• feat: K8S 1.29 support (#36527): @raphaelauv
• Standardize airflow build process and switch to Hatchling build backend (#36537): @potiuk
• Avoid using dict as default value in call_regular_interval (#36608): @hussein-awala
• Tracking airflow-github changelog activities using progress bar (#36610): @amoghrajesh
• Fix run type icon alignment with run type text (#36616): @shohamy7 @amoghrajesh
  Linked issues:
  • Align the run type icon in line with text in grid view (#36574)
• Redirect to index when user does not have permission to access a page (#36623): @vincbeck
• Make graphviz dependency optional (#36647): @potiuk
  Linked issues:
  • Standardize airflow build process and switch to Hatchling build backend (#36537)
• Make sure multiple_outputs is inferred correctly even when using TypedDict (#36652): @noamcohen97
• Fix the type hint for tis_query in _process_executor_events (#36655): @hussein-awala
• Provide the logger_name param to base hook in order to override the logger name (#36674): @hussein-awala
• Change in dag details from concurrency to max active tasks (#36691): @romsharon98
• Fix Callback exception when a removed task is the last one in the task instance list (#36693): @joaopamaral
• Get rid of pyarrow-hotfix for CVE-2023-47248 (#36697): @potiuk @ephraimbuddy
  Linked issues:
  • Use pyarrow-hotfix to mitigate CVE-2023-47248 (#35650)
  • Use pyarrow-hotfix to mitigate CVE-2023-47248 (#35650)
• Set min pandas dependency to 1.2.5 for all providers and airflow (#36698): @potiuk
  Linked issues:
  • Standardize airflow build process and switch to Hatchling build backend (#36537)
• Fix AttributeError while using the function update_state (#36712): @fuatcakici
  Linked issues:
  • AttributeError while using the function update_state (#34604)
• Use mariadb by default when build final prod image (#36716): @ant-bl @Taragolis
  Linked issues:
  • Docker libmariadb.so.3 is missing from official Docker image (#36715)
• Add back FAB constant in legacy security manager (#36719): @zachliu @vincbeck
  Linked issues:
  • default_webserver_config.py isn't compatible with the latest changes (#36702)
• Fix check on subclass for typing.Union in _infer_multiple_outputs for Python 3.10+ (#36728): @Taragolis
• Fix tests to adopt changes in Jinja 3.1.3 (#36731): @Taragolis
• Remove usused index on task instance (#36737): @dstandish @babinos87
  Linked issues:
  • Index on task instance (#30762)
• Allow anoymous user edit/show resource when set AUTH_ROLE_PUBLIC (#36750): @BobDu @eliskovets
  Linked issues:
  • Webserver returns 500 for POST requests to api/dag/*/dagrun from anonymous user (#36110)
• Restore function scoped httpx import in file_task_handler for perf (#36753): @vchiapaikeo
• explicit string cast required to force integer-type run_ids to be passed as strings instead of integers (#36756): @evgenyslab
• Disable PytestDeprecationWarning when create warnings_recorder (#36759): @Taragolis
• Generate Python client in reproducible way (#36763): @potiuk
• Better error message when sqlite URL uses relative path (#36774): @potiuk
• Remove wrong changelog about default Python version in the image (#36777): @potiuk
  Linked issues:
  • Switch "latest" image to point to newest supported Python version (#36003)
  • Apache Airflow "latest" image is still Python 3.8 in Airflow 2.8.0 (#36740)
• Add unit test for AUTH_ROLE_PUBLIC=Admin (#36787): @BobDu
  Linked issues:
  • Allow anoymous user edit/show resource when set AUTH_ROLE_PUBLIC (#36750)
• Avoid auto-detection of Airflow Sources during breeze installation (#36792): @potiuk
• Fix airflow-scheduler exiting with code 0 on exceptions (#36800): @joaopamaral

Thanks to all who contributed to the release (probably not a complete list!):
@joaopamaral @Lee-W @hussein-awala @amoghrajesh @fuatcakici @zachliu @llamageddon83 @dirrao @avkirilishin @uranusjr @dolfinus @yo1956 @ephraimbuddy @internetcoffeephone @BobDu @romsharon98 @Kache @kaxil @timonviola @potiuk @pateash @babinos87 @shohamy7 @tullis @HurSungYun @raphaelauv @ant-bl @csp33 @Taragolis @evgenyslab @joshowen @dstandish @noamcohen97 @ardubev16 @eliskovets @crabio @jscheffl @alephmelo @kacpermuda @vchiapaikeo @vincbeck

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[BobDu]

sorry, where can i pull airflow 2.8.1rc1 docker image?
not found in dockerhub

https://hub.docker.com/r/apache/airflow/tags?page=1&name=2.8.1rc1

[ephraimbuddy]

sorry, where can i pull airflow 2.8.1rc1 docker image? not found in dockerhub

https://hub.docker.com/r/apache/airflow/tags?page=1&name=2.8.1rc1

Sorry, I created this before the images were done pushing. Can you try again

[BobDu]

#36275 #36750 #36787 working as expected.

[amoghrajesh]

Just checked my changes:

• Tracking airflow-github changelog activities using progress bar #36610
• Fix run type icon alignment with run type text #36616 (not my change exactly, but tested it and works)

Both of these work as expected 🚢

[Taragolis]

• Use mariadb by default when build final prod image #36716

Validate all non slim X86_64 images, the command python -c 'import MySQLdb; print("OK")' do not throw any error

In addition validate custom docker build from the tag without specifying INSTALL_MYSQL_CLIENT_TYPE build arg, everything works fine

❯ git checkout 2.8.1rc1
Note: switching to '2.8.1rc1'.

❯ docker build . \
    --build-arg DOCKER_CONTEXT_FILES=./docker-context-files \
    --build-arg INSTALL_MYSQL_CLIENT="true" \
    --build-arg INSTALL_MSSQL_CLIENT="true" \
    --build-arg INSTALL_POSTGRES_CLIENT="true" \
    --tag "airflow-image:latest"
[+] Building 171.4s (68/68) FINISHED                                                                                                           docker:default

❯ docker run -it --rm airflow-image:latest bash

airflow@ab694be6c26d:/opt/airflow$ apt list *maria*
Listing... Done
libmariadb3-compat/now 1:10.11.6+maria~deb12 amd64 [installed,local]
libmariadb3/now 1:10.11.6+maria~deb12 amd64 [installed,local]
mariadb-client-core/now 1:10.11.6+maria~deb12 amd64 [installed,local]
mariadb-client/now 1:10.11.6+maria~deb12 amd64 [installed,local]
mariadb-common/now 1:10.11.6+maria~deb12 all [installed,local]

airflow@ab694be6c26d:/opt/airflow$ python -c 'import MySQLdb; print("OK")'
OK

Changes in tests, should not affect codebase

• Disable PytestDeprecationWarning when create warnings_recorder #36759
• Fix tests to adopt changes in Jinja 3.1.3 #36731

Taskflow's multiple_outputs

• Make sure multiple_outputs is inferred correctly even when using TypedDict #36652
• Fix check on subclass for typing.Union in _infer_multiple_outputs for Python 3.10+ #36728

Tested, work as expected. No errors in case of Union typing annotation in Python 3.10+

Decorators typing annotations

• Make kubernetes decorator type annotation consistent with operator #36405
• Remove redundant docker decorator type annotations #36406

Looks consistent to the latest docker and k8s providers

[xpgcrx]

I checked my change:

• flask config: max_content_length (Add flask config: MAX_CONTENT_LENGTH #36401)

It worked as expected.
If a request larger than the "allowed_payload_size" specified in airflow.cfg is sent to the webserver, it returns the error page: "Request Entity Too Large".

[joaopamaral]

• Fix airflow-scheduler exiting with code 0 on exceptions (#36800):
  Confirmed that the scheduler is exiting with code 1 on exception:

• Fix Callback exception when a removed task is the last one in the task instance list (#36693):
  Confirmed the callback ran without any issue when a removed task is the last one in the task list:

[2024-01-16, 13:55:20 UTC] {logging_mixin.py:188} INFO - Success callback triggered!
[2024-01-16, 13:55:20 UTC] {logging_mixin.py:188} INFO - Task Instance: [<TaskInstance: example_removed_task_dag.success_task scheduled__2024-01-15T00:00:00+00:00 [running]>, <TaskInstance: example_removed_task_dag.removed_task_2 scheduled__2024-01-15T00:00:00+00:00 [removed]>, <TaskInstance: example_removed_task_dag.removed_task scheduled__2024-01-15T00:00:00+00:00 [removed]>, <TaskInstance: example_removed_task_dag.failure_task scheduled__2024-01-15T00:00:00+00:00 [removed]>, <TaskInstance: example_removed_task_dag.start_task scheduled__2024-01-15T00:00:00+00:00 [removed]>, <TaskInstance: example_removed_task_dag.end_task scheduled__2024-01-15T00:00:00+00:00 [removed]>]

[csp33]

All good with #36522

[potiuk]

All my changes are in and good !

Particularly happy with that one:

airflow@e6cddae1d57a:/opt/airflow$ AIRFLOW_HOME=./test airflow db init
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 68, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 538, in initialize
    configure_vars()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 198, in configure_vars
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Cannot use relative path: `sqlite:///./test/airflow.db` to connect to sqlite. Please use absolute path such as `sqlite:////tmp/airflow.db`.

[joshowen]

I've confirmed fix datetime reference in DAG.is_fixed_time_schedule (#36370) is good.

[shohamy7]

#36522 & #36616 looks good to me

[Taragolis]

• Add support of Pendulum 3 #36281

Check different component under pendulum 3 (default in constraints) and pendulum 2.1.2, can't find any noticeable side effects

[jscheffl]

Validated #36509 and #36518 - all looks good!

[Lee-W]

I tried to test #36379, but it did not fix the issue as expected. I still need #36363 to get it to work properly. It did not break anything, either, so it won't be a blocker. @hussein-awala, do you have the chance to test it on your side as well?