Ensure that TI.id is unique per try. #48749
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ashb
force-pushed
the
remove-try_id-from-ti
branch
4 times, most recently
from
cf203cb to
80a8071
Compare
amoghrajesh
reviewed
Apr 9, 2025
airflow-core/src/airflow/migrations/versions/0067_3_0_0_change_ti_table_to_have_.py
Show resolved
Hide resolved
ashb
force-pushed
the
remove-try_id-from-ti
branch
2 times, most recently
from
655ae65 to
67f83c4
Compare
ashb
added
the
full tests needed
Member
Author
|
Needs a pr desc etc, but I want to see tests |
ashb
force-pushed
the
remove-try_id-from-ti
branch
from
67f83c4 to
55e522a
Compare
kaxil
reviewed
Apr 10, 2025
airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py
Outdated
Show resolved
Hide resolved
kaxil
reviewed
Apr 10, 2025
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py
Show resolved
Hide resolved
kaxil
reviewed
Apr 10, 2025
kaxil
approved these changes
Apr 10, 2025
ashb
force-pushed
the
remove-try_id-from-ti
branch
from
55e522a to
cdfbbdc
Compare
ashb
changed the title
try_id+id on TaskInstance table.
ashb
commented
Apr 10, 2025
airflow-core/src/airflow/migrations/versions/0041_3_0_0_rename_dataset_as_asset.py
Outdated
Show resolved
Hide resolved
ashb
commented
Apr 10, 2025
airflow-core/src/airflow/migrations/versions/0041_3_0_0_rename_dataset_as_asset.py
Outdated
Show resolved
Hide resolved
ashb
force-pushed
the
remove-try_id-from-ti
branch
from
035bf49 to
bd8f902
Compare
We want to update the task instance and ti history tables to have a unique id per attempt (this row id will be used in the strong identity token issued to reach TI attempt.) While this was possible with the previous implementation of having `id` and `try_id` columns on the TaskInstance table it introduces confusion in the API, and breaks some fundamental assumptions in the Execution API. The only real changes needed to make this work (outside of some changes in tests) as to set `ON UPDATE CASCADE` on task note (as notes are by design across all task retries), and to delete from the TaskReschedule table when retrying a task, which is good anyway as it stops that table growing without bound. Closes apache#44975
ashb
force-pushed
the
remove-try_id-from-ti
branch
from
bd8f902 to
fc20c71
Compare
56 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to update the task instance and ti history tables to have a unique id
per attempt (this row id will be used in the strong identity token issued to
reach TI attempt.)
While this was possible with the previous implementation of having
idandtry_idcolumns on the TaskInstance table it introduces confusion in the API,and breaks some fundamental assumptions in the Execution API.
The only real changes needed to make this work (outside of some changes in
tests) as to set
ON UPDATE CASCADEon task note (as notes are by designacross all task retries), and to delete from the TaskReschedule table when
retrying a task, which is good anyway as it stops that table growing without
bound.
Closes #44975
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.