Airflow dag processor exits with too many open files after sometime

[asasisekar]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Upgraded Airflow from 2.10 to 3.0.0 and DagProcessorJob getting failed after few hours. with below exception
OSError: [Errno 24] Too many open files

[2025-04-25T21:30:41.765+0100] {dag_processor_job_runner.py:63} ERROR - Exception when executing DagProcessorJob
Traceback (most recent call last):
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
    self.processor.run()
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 262, in run
    return self._run_parsing_loop()
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 347, in _run_parsing_loop
    self._start_new_processes()
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 894, in _start_new_processes
    processor = self._create_process(file)
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/dag_processing/manager.py", line 876, in _create_process
    return DagFileProcessorProcess.start(
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/dag_processing/processor.py", line 245, in start
    proc: Self = super().start(target=target, **kwargs)
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py", line 446, in start
    child_comms, read_msgs = mkpipe()
  File "/var/opt/rdos-airflow/venv/lib/python3.10/site-packages/airflow/sdk/execution_time/supervisor.py", line 177, in mkpipe
    rsock, wsock = socketpair()
  File "/var/opt/icetools/python/ICEpythonvenv310/python-3.10.1/lib/python3.10/socket.py", line 607, in socketpair
    a, b = _socket.socketpair(family, type, proto)
OSError: [Errno 24] Too many open files

What you think should happen instead?

No response

How to reproduce

nohup airflow dag-processor

DAG Processor Config

export AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL=10
export AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL=120
export AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=1
export AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL=900

Operating System

RHEL 8.8

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

This problem occurs always for every few hours once.
Number of DAGs less than 10.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[vatsrahul1001]

@tirkarthi looks like same issue as #46048

[tirkarthi]

@asasisekar Can you please run this command in the dag-processor instance watch "pgrep -f dag-processor | xargs -I{} ls -l /proc/{}/fd/ >> /tmp/fd.log" to see if there are any open files that keep increasing in /tmp/fd.log ? I don't see any open files increasing but since this is present after a few hours the fd.log persisted to a location will help in debugging.

https://unix.stackexchange.com/questions/66235/how-to-display-open-file-descriptors-but-not-using-lsof-command

[asasisekar]

@tirkarthi With running with 100 DAGs, DAG processor getting failed with too many open files in few mins.
Around 390 open files

lrwx------ 1 svc svc 64 Apr 28 16:26 390 -> socket:[2348769229]
lrwx------ 1 svc svc 64 Apr 28 16:26 392 -> socket:[2348769231]

DAGs have lot of variables. is that creating too many open files ?

DAG processor have lot of below http calls in log.

[2025-04-28T16:25:37.142+0100] {_client.py:1026} INFO - HTTP Request: GET http://in-process.invalid./variables/xxxx "HTTP/1.1 200 OK"
[2025-04-28T16:25:37.152+0100] {_client.py:1026} INFO - HTTP Request: GET http://in-process.invalid./variables/yyyy "HTTP/1.1 200 OK"

[zachliu]

running watch "pgrep -f dag-processor | xargs -I{} ls -l /proc/{}/fd/ >> fd.log" accumulated 1.3mb of fd.log with 18k rows in 20 seconds, 12k of which are socket: [...]. i have merely ~20 active DAGs + ~300 disabled DAGs.

this might explain why the new dag-processor is both a CPU-hog and a memory-hog
(see my datadog metric screenshot at #49650 (comment))

[uranusjr]

It would be helpful if we have some dags that can produce a similar behaviour.

[vatsrahul1001]

I wasn’t able to reproduce this issue using the Breeze environment, even after executing 100 DAGs with a sleep interval. I also used a variable within the DAG to confirm behavior.

Still continuing to investigate and trying to reproduce the issue.

throughput.txt

[zachliu]

@asasisekar @vatsrahul1001 have you also experienced high cpu/memory usage in the meantime?