Fix scheduler heartbeat misses caused by slow reschedule dependency check#61983
Merged
ephraimbuddy merged 1 commit intoFeb 17, 2026
Merged
Conversation
boring-cyborg
Bot
added
area:db-migrations
…heck When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
ephraimbuddy
force-pushed
the
add-composite-index-task-reschedule
branch
from
7beb11c to
c8f33d8
Compare
kaxil
approved these changes
Feb 16, 2026
Contributor
Backport failed to create: v3-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 9880716 v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
OscarLigthart
pushed a commit
to OscarLigthart/airflow
that referenced
this pull request
Feb 17, 2026
…heck (apache#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
ephraimbuddy
added a commit
that referenced
this pull request
Feb 17, 2026
…heck (#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index. (cherry picked from commit 9880716)
ephraimbuddy
added a commit
that referenced
this pull request
Feb 17, 2026
…dependency check (#61983) (#62068) * Add index on task_reschedule ti_id (#60931) (cherry picked from commit 14e811c) * Fix scheduler heartbeat misses caused by slow reschedule dependency check (#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index. (cherry picked from commit 9880716) --------- Co-authored-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
choo121600
pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
…heck (apache#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
Subham-KRLX
pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
…heck (apache#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
vatsrahul1001
pushed a commit
that referenced
this pull request
Mar 4, 2026
…dependency check (#61983) (#62068) * Add index on task_reschedule ti_id (#60931) (cherry picked from commit 14e811c) * Fix scheduler heartbeat misses caused by slow reschedule dependency check (#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index. (cherry picked from commit 9880716) --------- Co-authored-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
dominikhei
pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
…heck (apache#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
Ankurdeewan
pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
…heck (apache#61983) When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats. Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table. Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
|
Thank you very much for this contribution, I was having headaches understanding why my scheduler was stuck for 30s and was slowing down my whole pipeline. This PR solves everything |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When many task instances enter UP_FOR_RESCHEDULE state, the query to fetch the latest reschedule date becomes slow due to a missing composite index. This causes the scheduler to miss heartbeats.
Previously only sensors used reschedule mode, but since fddf4a7, non-sensor tasks can also be rescheduled, significantly increasing the number of rows per task instance in the task_reschedule table.
Add a composite (ti_id, id DESC) index to the task_reschedule table, replacing the single-column (ti_id) index.
The reschedule query:
airflow/airflow-core/src/airflow/ti_deps/deps/ready_to_reschedule.py
Line 73 in baa8c72
Other places this can benefit:
airflow/airflow-core/src/airflow/models/taskinstance.py
Line 1171 in baa8c72
airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_reschedules.py
Line 40 in baa8c72