Dag processor gets SIGKILL signal and all DAGs are removed from UI

[krisgeus]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Dag processor got killed. DAGs disappeared, The UI shows the dag-processor healthy still, Restarting the dag-processor service solves this but I do not think this will be the way to go.

What you think should happen instead?

DAG processor should not be killed or recover correctly from this if this is correct behaviour

How to reproduce

Use the provided docker compose with a couple of dag in the mounted dag directory. After a while (computer went to sleep sometimes) the dags disapear

Operating System

Docker compose on OSX

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

Log extract:

[2025-04-23T15:21:19.334+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:21:24.409+0000] {manager.py:748} INFO - 
================================================================================
DAG File Processing Stats

Bundle       File Path                          PID    Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-----------  ---------------------------------  -----  ------------------  --------  ----------  ---------------  -------------------
dags-folder  04_PythonOperator_get_pictures.py                                    1           0  0.74s            2025-04-23T15:20:55
dags-folder  03_BashOperator.py                                                   1           0  0.74s            2025-04-23T15:20:55
dags-folder  02_download_rocket_launches.py                                       1           0  0.74s            2025-04-23T15:20:57
dags-folder  01_download_rocket_launches.py                                       1           0  0.76s            2025-04-23T15:20:54
dags-folder  05_download_rocket_launches.py                                       1           0  0.76s            2025-04-23T15:20:54
================================================================================
[2025-04-23T15:21:24.411+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:21:24.413+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:21:24.420+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:21:25.478+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:25.481+0000] {dag.py:2517} INFO - Setting next_dagrun for 01_download_rocket_launches to None, run_after=None
[2025-04-23T15:21:25.487+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:25.491+0000] {dag.py:2517} INFO - Setting next_dagrun for 05_download_rocket_launches to 2025-04-24 00:00:00+00:00, run_after=2025-04-24 00:00:00+00:00
[2025-04-23T15:21:26.558+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:26.561+0000] {dag.py:2517} INFO - Setting next_dagrun for 03_BashOperator to None, run_after=None
[2025-04-23T15:21:26.569+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:26.571+0000] {dag.py:2517} INFO - Setting next_dagrun for 04_PythonOperator_get_pictures to None, run_after=None
[2025-04-23T15:21:27.626+0000] {dag.py:1894} INFO - Sync 1 DAGs
[2025-04-23T15:21:27.629+0000] {dag.py:2517} INFO - Setting next_dagrun for 02_download_rocket_launches to None, run_after=None
[2025-04-23T15:21:29.661+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:21:34.713+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:24:45.683+0000] {job.py:232} INFO - Heartbeat recovered after 186.91 seconds
[2025-04-23T15:24:45.689+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:24:45.690+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:24:45.696+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:24:45.721+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 794 started 186 ago killing it.
2025-04-23 15:24:45 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=794 signal_sent=SIGKILL
[2025-04-23T15:24:45.725+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 795 started 186 ago killing it.
2025-04-23 15:24:45 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=795 signal_sent=SIGKILL
[2025-04-23T15:24:45.763+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 796 started 186 ago killing it.
2025-04-23 15:24:45 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=796 signal_sent=SIGKILL
[2025-04-23T15:24:45.767+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 797 started 186 ago killing it.
2025-04-23 15:24:45 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=797 signal_sent=SIGKILL
[2025-04-23T15:24:45.796+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 798 started 186 ago killing it.
2025-04-23 15:24:45 [info     ] Process exited                 [supervisor] exit_code=0 pid=798 signal_sent=SIGKILL
[2025-04-23T15:24:50.836+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:24:55.899+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:00.974+0000] {manager.py:748} INFO - 
================================================================================
DAG File Processing Stats

Bundle       File Path                          PID    Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-----------  ---------------------------------  -----  ------------------  --------  ----------  ---------------  -------------------
dags-folder  04_PythonOperator_get_pictures.py                                    0           1  186.62s          2025-04-23T15:24:45
dags-folder  03_BashOperator.py                                                   0           1  186.62s          2025-04-23T15:24:45
dags-folder  02_download_rocket_launches.py                                       0           1  186.62s          2025-04-23T15:24:45
dags-folder  01_download_rocket_launches.py                                       0           1  186.61s          2025-04-23T15:24:45
dags-folder  05_download_rocket_launches.py                                       0           1  186.60s          2025-04-23T15:24:45
================================================================================
[2025-04-23T15:25:00.983+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:06.048+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:11.141+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:25:12.155+0000] {manager.py:303} INFO - DAG 01_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.155+0000] {manager.py:303} INFO - DAG 05_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 03_BashOperator is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 04_PythonOperator_get_pictures is missing and will be deactivated.
[2025-04-23T15:25:12.156+0000] {manager.py:303} INFO - DAG 02_download_rocket_launches is missing and will be deactivated.
[2025-04-23T15:25:12.160+0000] {manager.py:315} INFO - Deactivated 5 DAGs which are no longer present in file.
[2025-04-23T15:26:54.257+0000] {job.py:232} INFO - Heartbeat recovered after 103.12 seconds
[2025-04-23T15:26:54.277+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 807 started 285 ago killing it.
2025-04-23 15:26:54 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=807 signal_sent=SIGKILL
[2025-04-23T15:26:54.280+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 808 started 285 ago killing it.
2025-04-23 15:26:54 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=808 signal_sent=SIGKILL
[2025-04-23T15:26:54.314+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 809 started 285 ago killing it.
2025-04-23 15:26:54 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=809 signal_sent=SIGKILL
[2025-04-23T15:26:54.317+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 810 started 285 ago killing it.
2025-04-23 15:26:54 [info     ] Process exited                 [supervisor] exit_code=0 pid=810 signal_sent=SIGKILL
[2025-04-23T15:26:54.346+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 811 started 285 ago killing it.
2025-04-23 15:26:54 [info     ] Process exited                 [supervisor] exit_code=0 pid=811 signal_sent=SIGKILL
[2025-04-23T15:26:55.329+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:26:55.332+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:26:55.340+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:27:00.378+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:05.447+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:10.535+0000] {manager.py:748} INFO - 
================================================================================
DAG File Processing Stats

Bundle       File Path                          PID    Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-----------  ---------------------------------  -----  ------------------  --------  ----------  ---------------  -------------------
dags-folder  04_PythonOperator_get_pictures.py                                    0           1  285.64s          2025-04-23T15:26:54
dags-folder  03_BashOperator.py                                                   0           1  285.64s          2025-04-23T15:26:54
dags-folder  02_download_rocket_launches.py                                       0           1  285.65s          2025-04-23T15:26:54
dags-folder  01_download_rocket_launches.py                                       0           1  285.63s          2025-04-23T15:26:54
dags-folder  05_download_rocket_launches.py                                       0           1  285.63s          2025-04-23T15:26:54
================================================================================
[2025-04-23T15:27:10.538+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:15.621+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:20.697+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:24.791+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 820 started 285 ago killing it.
2025-04-23 15:27:24 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=820 signal_sent=SIGKILL
[2025-04-23T15:27:24.793+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 821 started 285 ago killing it.
2025-04-23 15:27:24 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=821 signal_sent=SIGKILL
[2025-04-23T15:27:24.831+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 822 started 285 ago killing it.
2025-04-23 15:27:24 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=822 signal_sent=SIGKILL
[2025-04-23T15:27:24.834+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 823 started 285 ago killing it.
2025-04-23 15:27:24 [info     ] Process exited                 [supervisor] exit_code=0 pid=823 signal_sent=SIGKILL
[2025-04-23T15:27:24.863+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 824 started 285 ago killing it.
2025-04-23 15:27:24 [info     ] Process exited                 [supervisor] exit_code=0 pid=824 signal_sent=SIGKILL
[2025-04-23T15:27:25.849+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:30.897+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:35.937+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:41.018+0000] {manager.py:748} INFO - 
================================================================================
DAG File Processing Stats

Bundle       File Path                          PID    Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-----------  ---------------------------------  -----  ------------------  --------  ----------  ---------------  -------------------
dags-folder  04_PythonOperator_get_pictures.py                                    0           1  285.65s          2025-04-23T15:27:24
dags-folder  03_BashOperator.py                                                   0           1  285.65s          2025-04-23T15:27:24
dags-folder  02_download_rocket_launches.py                                       0           1  285.64s          2025-04-23T15:27:24
dags-folder  01_download_rocket_launches.py                                       0           1  285.63s          2025-04-23T15:27:24
dags-folder  05_download_rocket_launches.py                                       0           1  285.63s          2025-04-23T15:27:24
================================================================================
[2025-04-23T15:27:41.020+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:46.093+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:51.184+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:27:55.373+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('01_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 833 started 285 ago killing it.
2025-04-23 15:27:55 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=833 signal_sent=SIGKILL
[2025-04-23T15:27:55.377+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('05_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 834 started 285 ago killing it.
2025-04-23 15:27:55 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=834 signal_sent=SIGKILL
[2025-04-23T15:27:55.417+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('04_PythonOperator_get_pictures.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 835 started 285 ago killing it.
2025-04-23 15:27:55 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=835 signal_sent=SIGKILL
[2025-04-23T15:27:55.420+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('03_BashOperator.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 836 started 285 ago killing it.
2025-04-23 15:27:55 [info     ] Process exited                 [supervisor] exit_code=0 pid=836 signal_sent=SIGKILL
[2025-04-23T15:27:55.449+0000] {manager.py:1000} ERROR - Processor for DagFileInfo(rel_path=PosixPath('02_download_rocket_launches.py'), bundle_name='dags-folder', bundle_path=PosixPath('/opt/airflow/dags'), bundle_version=None) with PID 837 started 285 ago killing it.
2025-04-23 15:27:55 [info     ] Process exited                 [supervisor] exit_code=0 pid=837 signal_sent=SIGKILL
[2025-04-23T15:27:56.433+0000] {manager.py:526} INFO - Refreshing bundle dags-folder
[2025-04-23T15:27:56.435+0000] {manager.py:577} INFO - Searching for files in dags-folder at /opt/airflow/dags
[2025-04-23T15:27:56.444+0000] {manager.py:579} INFO - Found 5 files for bundle dags-folder
[2025-04-23T15:28:01.501+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:28:06.571+0000] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-23T15:28:11.648+0000] {manager.py:748} INFO - 

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashb]

Got a fix coming for this.

[amoghrajesh]

Marking as high priority as this is an issue related to user experience and from my analysis, it will mostly happen only in dev environments, in any case, worth fixing for 3.0.1

[amoghrajesh]

@krisgeus I merged a fix for this, please try with the fix and let me know if it works

[AmeyKu1Karni]

@amoghrajesh I am also using Airflow Version 3.0.0. Can you guide me what to do exactly. I always have to restart the Airflow UI to get the DAGs back after making some changes in the DAG python file. Nothing happens even if i refresh the Airflow UI page. It just says No DAGs found.