Multiple DAG processors with separate DAG directories keep deactivating each other's DAGs

[smphhh]

Apache Airflow version

2.6.3

What happened

When running multiple standalone DAG processors with separate DAG directories using the --subdir argument the processors keep deactivating each other's DAGs (and reactivating their own).

After stepping through the code with a debugger I think the issue is that the calls here and here have no awareness of the DAG directories.

What you think should happen instead

The DAG processors should not touch each other's DAGs in the metadata DB.

How to reproduce

Start two or more standalone DAG processors with separate DAG directories and observe (e.g. via the UI) how the list of active DAGs keeps changing constantly.

Operating System

Linux 94b223524983 6.1.32-0-virt #1-Alpine SMP PREEMPT_DYNAMIC Mon, 05 Jun 2023 09:39:09 +0000 x86_64 x86_64 x86_64 GNU/Linux

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[pierrejeambrun]

Hello @smphhh,

Thank you for reporting.

I couldn't reproduce on main with breeze. Basically started 2 dag-processors on 2 different subdirs, activated one dag from each folder and nothing unexpected happened. I only had 2 'active' dags and none of them were deactivated by each other processor loop.

Can you try on main branch and this if you still experience the issue, or give me extra steps to reproduce that I might have missed.

Confirmed. (My initial tries didn't show the issue because of fortunate 'good' sync between the two processor)