Skip to content

Schedule backfills in the scheduler #42686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dstandish merged 13 commits into from
Oct 5, 2024
Merged

Schedule backfills in the scheduler #42686

dstandish merged 13 commits into from
Oct 5, 2024

Conversation

@dstandish
Copy link
Contributor

@dstandish dstandish commented Oct 2, 2024

(cherry picked from commit 2a581ab)

@boring-cyborg boring-cyborg bot added area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation labels Oct 2, 2024
@dstandish dstandish mentioned this pull request Oct 2, 2024
Merged
@dstandish dstandish force-pushed the aip-78-schedule-backfills-in-the-scheduler branch from f05eca4 to 5763191 Compare October 3, 2024 02:53
@dstandish dstandish marked this pull request as ready for review October 3, 2024 16:29
jedcunningham
jedcunningham reviewed Oct 3, 2024
View reviewed changes
dstandish and others added 3 commits October 3, 2024 15:56
@dstandish @jedcunningham
remove comments from migrations
45d61e4 
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@dstandish
remove unrelated test
fc83368
@dstandish
erd update
d1bbe1b
@dstandish dstandish merged commit 1b4c6d5 into apache:main Oct 5, 2024
@dstandish dstandish deleted the aip-78-schedule-backfills-in-the-scheduler branch October 5, 2024 13:34
@jscheffl
Copy link
Contributor

jscheffl commented Oct 6, 2024

It seems that this PR broke the "Non DB" tests for Always and CLI in https://github.com/apache/airflow/actions/runs/11200846409/job/31135117858?pr=42773 and on main.

Any idea on how to fix this (easily) w/o reversion?

@jscheffl jscheffl mentioned this pull request Oct 6, 2024
Merged
@dstandish
Copy link
Contributor Author

dstandish commented Oct 6, 2024

It seems that this PR broke the "Non DB" tests for Always and CLI in https://github.com/apache/airflow/actions/runs/11200846409/job/31135117858?pr=42773 and on main.

Any idea on how to fix this (easily) w/o reversion?

I took a quick look but cannot reproduce locally.

I am unavailable most of today.

@dirrao dirrao mentioned this pull request Oct 6, 2024
Merged
@jscheffl
Copy link
Contributor

jscheffl commented Oct 6, 2024 •
edited
Loading

It seems that this PR broke the "Non DB" tests for Always and CLI in https://github.com/apache/airflow/actions/runs/11200846409/job/31135117858?pr=42773 and on main.
Any idea on how to fix this (easily) w/o reversion?

I took a quick look but cannot reproduce locally.

I am unavailable most of today.

Thanks for checking. I saw it on multiple PRs and might be it is related to non DB tests and threading?
On my Laptop I can re-produce via:

git checkout main
git pull
breeze ci-image build --python 3.8  # ensure this is up-to-date
breeze testing non-db-tests --parallel-test-types "Always CLI"

Whereas interesting that it does not show-up in canary tests :-( I don't understand why, checked with different parallelism and not able to get this green locally.

@dstandish
Copy link
Contributor Author

dstandish commented Oct 6, 2024

i'll look now

kunaljubce pushed a commit to kunaljubce/airflow that referenced this pull request Oct 13, 2024
@dstandish @jedcunningham @kunaljubce
Schedule backfills in the scheduler (apache#42686)
2fd1068 
This is part of AIP-78.  Here we make it so for dag runs queued as part of a backfill, the scheduler will set them running in accordance with concurrency controls and schedule their tasks etc.  Currently, backfill's concurrency is evaluated independently of DAG.max_active_runs.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
@dstandish @jedcunningham @joaopamaral
Schedule backfills in the scheduler (apache#42686)
724ff32 
This is part of AIP-78.  Here we make it so for dag runs queued as part of a backfill, the scheduler will set them running in accordance with concurrency controls and schedule their tasks etc.  Currently, backfill's concurrency is evaluated independently of DAG.max_active_runs.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
@dstandish @jedcunningham @harjeevanmaan
Schedule backfills in the scheduler (apache#42686)
900e2fa 
This is part of AIP-78.  Here we make it so for dag runs queued as part of a backfill, the scheduler will set them running in accordance with concurrency controls and schedule their tasks etc.  Currently, backfill's concurrency is evaluated independently of DAG.max_active_runs.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
@dstandish @jedcunningham @ellisms
Schedule backfills in the scheduler (apache#42686)
dbc0fb6 
This is part of AIP-78.  Here we make it so for dag runs queued as part of a backfill, the scheduler will set them running in accordance with concurrency controls and schedule their tasks etc.  Currently, backfill's concurrency is evaluated independently of DAG.max_active_runs.

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jedcunningham jedcunningham jedcunningham approved these changes

@potiuk potiuk Awaiting requested review from potiuk

@kaxil kaxil Awaiting requested review from kaxil

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@ashb ashb Awaiting requested review from ashb

Assignees

No one assigned

Labels

area:db-migrations PRs with DB migration area:Scheduler including HA (high availability) scheduler kind:documentation

Projects

No open projects
AIP-78 backfills in scheduler
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dstandish @jscheffl @jedcunningham