Skip to content

Do not dilute task failures with finalization errors during execution#62070

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:dont-dilute-original-errors
Feb 18, 2026
Merged

Do not dilute task failures with finalization errors during execution#62070
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:dont-dilute-original-errors

Conversation

@amoghrajesh
Copy link
Copy Markdown
Contributor


Was generative AI tooling used to co-author this PR?
  • no

Why and Fix

When a task with overwrite_rtif_after_execution=True fails, the finalize() function attempts to update RTIF via the execution API. If this API call fails (e.g., 404 Not Found due to timing issues or whatever reasons), the error propagates and masks the original task failure error.

This causes users to see "AirflowRuntimeError: API_SERVER_ERROR" instead of their actual task error , making it confusing to narrow down the issue while debugging.

The fix is to wrap the SetRenderedFields call in a error handling block, consistent with how operator extra links xcoms pushes are handled in the same function. Errors are now logged but don't prevent finalization from completing or
mask the original task error.

If you undo changes in the task_runner file, you can see the error like this:

/Users/amoghdesai/Documents/OSS/repos/airflow/.venv/bin/python /Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pycharm/_jb_pytest_runner.py --target task_sdk/execution_time/test_task_runner.py::TestRuntimeTaskInstance.test_overwrite_rtif_after_execution_handles_errors_gracefully 
Testing started at 4:28pm ...
Launching pytest with arguments task_sdk/execution_time/test_task_runner.py::TestRuntimeTaskInstance::test_overwrite_rtif_after_execution_handles_errors_gracefully --no-header --no-summary -q in /Users/amoghdesai/Documents/OSS/repos/airflow/task-sdk/tests

============================= test session starts ==============================
collecting ... collected 1 item

task_sdk/execution_time/test_task_runner.py::TestRuntimeTaskInstance::test_overwrite_rtif_after_execution_handles_errors_gracefully 

========================= 1 failed, 1 warning in 4.61s =========================

========================= AIRFLOW ==========================
Home of the user: /Users/amoghdesai
Airflow home /Users/amoghdesai/airflow
2026-02-17T10:58:13.964423Z [warning  ] The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html [py.warnings] category=RemovedInAirflow4Warning filename=/Users/amoghdesai/Documents/OSS/repos/airflow/devel-common/src/tests_common/test_utils/db.py lineno=47
FAILED [100%]
tests/task_sdk/execution_time/test_task_runner.py:2181 (TestRuntimeTaskInstance.test_overwrite_rtif_after_execution_handles_errors_gracefully)
task_sdk/execution_time/test_task_runner.py:2215: in test_overwrite_rtif_after_execution_handles_errors_gracefully
    finalize(
../src/airflow/sdk/execution_time/task_runner.py:1705: in finalize
    SUPERVISOR_COMMS.send(SetRenderedFields(rendered_fields=_serialize_rendered_fields(ti.task)))
/opt/homebrew/Cellar/python@3.13/3.13.3_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/unittest/mock.py:1169: in __call__
    return self._mock_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/homebrew/Cellar/python@3.13/3.13.3_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/unittest/mock.py:1173: in _mock_call
    return self._execute_mock_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/homebrew/Cellar/python@3.13/3.13.3_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/unittest/mock.py:1228: in _execute_mock_call
    raise effect
E   airflow.sdk.exceptions.AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 404, 'message': 'Not Found', 'detail': {'detail': 'Not Found'}}

Process finished with exit code 1

But that error doesn't show up with the fix which is TLDR of the PR.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Comment thread task-sdk/tests/task_sdk/execution_time/test_task_runner.py Outdated
Copy link
Copy Markdown
Member

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks a lot @amoghrajesh for fixing this so quickly 🙇🏽

@amoghrajesh amoghrajesh merged commit 5af913d into apache:main Feb 18, 2026
101 checks passed
@amoghrajesh amoghrajesh deleted the dont-dilute-original-errors branch February 18, 2026 07:45
@github-actions
Copy link
Copy Markdown
Contributor

Backport failed to create: v3-1-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 5af913d v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

amoghrajesh added a commit that referenced this pull request Feb 18, 2026
@amoghrajesh
Copy link
Copy Markdown
Contributor Author

Manual cherry pick here: #62113

amoghrajesh added a commit that referenced this pull request Feb 18, 2026
vatsrahul1001 pushed a commit that referenced this pull request Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants