Memory leak in dag-processor after upgrade to airflow 3.0

[oleg-savko]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

After upgrade to airflow 3.0 from 2.10.5, dag-processor docker image used alot of memore after some times, seems like some memory leak, and in restarted after oom and airflow feels bad

What you think should happen instead?

No response

How to reproduce

airflow with ~100 dags some of them is huge dynamic dag

Operating System

linu

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Hasan-J]

Indeed, I'm encountering the same issue. I looked further into it at the process level and found that, over time, the DAG processor spawns numerous processes, eventually leading to an out-of-memory (OOM) exception. I resolved it by manually restarting the Docker service running the DAG processor.

The screenshot is taken right after I did the restart. But eventually the process list will grow bigger.

[tirkarthi]

Is there a sample standalone dag that you can share to reproduce this? Is there anything seen in profiling or other tools you use for observability to share?

https://bloomberg.github.io/memray/run.html

memray run --live -m airflow dag-processor

[tirkarthi]

Can you please try this patch to see if it helps with the memory issue?

#49887 (comment)

[kaxil]

oh interesting @tirkarthi , I am tackling this issue too. I have a fix in #50558