Add ability to disable a running DAG only after after it's in a finished state

[SangwanP]

Description

I'd like the ability to "drain" a DAG from the UI. This would mean:

• If a DAG is in a running state - disable the DAG after the run reaches a finished state (success/failed)
• If a DAG is in an idle state - disable immediately (same as what disable does right now)

Use case/motivation

Currently, if you disable a DAG, it stops the DAG midway - the running tasks are completed, and queued tasks remain on the queue. I would like a way to disable the DAG, but if there is an active Dag Run it should continue running until it reaches a finished state.

For us, this would be especially helpful when upgrading our Airflow images/environments. To my understanding, Airflow doesn't have a failure-free zero-downtime upgrade (please point me to documentation if its possible). When upgrading, we don't want to fail/kill running DAGs, so we let them reach a finished state. Once none of the DAGs are in a running state, we perform the upgrades. Precisely timing the DAG disable operation such that a subsequent run doesn't start, is not a practical/viable solution.
For upgrades in the past, we have changed all the DAG schedules to None, which allows a running DAG to complete but doesn't start a subsequent run. Once upgrades are done, we update the DAG schedules back to their original value.
We would like to move away from this solution, and have some built-in functionality to deal with such a case.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Good idea. Would you like to work on maybe ? this is the best way to make sure it happens :)

[bbovenzi]

I wonder if this should be added as a configuration for pausing/unpausing DAGs? Having a separate button for this could be confusing.

[eladkal]

The use case as described is "Don't schedule new runs but let the current running dags to finish" - This sounds more like a scheduler feature rather than UI.

[potiuk]

Very much so @eladkal

[SangwanP]

As a user, I'd like to have functionality in the UI that allows me to either "disable and stop the DAG immediately", or "disable DAG once finished". How to implement such a feature in the UI, such that it's not confusing is of course up for discussion - button, switch, etc.
The underlying functionality to perform the "drain" will likely be need to be developed in the scheduler. So, this feature I think, falls under both scheduler, and UI development areas.

Also, @potiuk - I can try to pick this task up in a couple of weeks. Let's leave it unassigned for now, and I can let you know once I am ready to pick it up.

[potiuk]

Sure!

[eladkal]

I think this involves some changes to the DB schema as probably it means that is_paused is no longer just bool of True / False but actually it has several states.

[bbovenzi]

I think this involves some changes to the DB schema as probably it means that is_paused is no longer just bool of True / False but actually it has several states.

This is why I am partial to it being a config for the scheduler because then we can't represent is_paused as an on/off switch.

[potiuk]

I think this involves some changes to the DB schema as probably it means that is_paused is no longer just bool of True / False but actually it has several states.

Maybe a chance to solve the "Activated" "Paused" conundrum: #14459