Small memory leak from the dag-processor in Airflow 3.x

[zachliu]

Apache Airflow version

3.x

If "Other Airflow 2 version" selected, which one?

N/A

What happened?

There was a significant memory leak (1 Gb/hour depending on DAGs), tracked in #49887 and #50097, which has been addressed by #50558 or via this patch #49887 (comment).

However, one (maybe more) much smaller leak(s) remain—approximately 20 MB/hour—in the dag-processor

What you think should happen instead?

Memory usage of the dag-processor shall be stable.

How to reproduce

Option 1: Run Airflow with only the example DAGs (active or pause, doesn't matter) and observe the growth of memory usage.

Option 2: Run airflow version command in a loop in any airflow container, and then observe the growth of memory usage.

#!/bin/bash

while true; do
  airflow version
  sleep 0.5
done

Operating System

LM 21.3 with docker python:3.12-slim-bookworm

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

Does checking fd give any clue like are these sockets? It will be helpful if it can be reproduced by a standalone simple dag like last one related to Variable/Connection.

watch "pgrep -f dag-processor | xargs -I{} ls -l /proc/{}/fd/ >> fd.log"

Thanks

[zachliu]

@tirkarthi i think i have identified at least two leaks and they don't seem to show up in fd:

1. caused by these two library imports. this leak majorly affects the worker. it has little to no impact in other services unless you're using ["CMD-SHELL", 'airflow ...'] in docker-compose healthcheck

   from importlib import metadata
   from sqlalchemy import create_engine

   to reproduce, run this in any airflow docker container

   #!/bin/bash
   
   while true; do
     # both of the following can show the leak
     # airflow version
     python test.py  # test.py contains the above two imports
     sleep 0.5
   done

2. (possibly) caused by certain sub-process manage mechanism in dag-processor. when there are no dags, you can't see this leak. to reproduce i duplicate this test_bash_operator.py.txt 750 times each with a unique dag name (test_bash_operator_X). then you can see dag-processor memory usage creeping up

right now i have no idea whether these 2 leaks are related or not. here is my test environment:

1. PYTHON_BASE_IMAGE="python:3.12-slim-bookworm" or python:3.9-slim-bookworm
2. airflow 2.10.5 and 3.0.1. local docker compose with CeleryExecutor
3. no example dags no other dags but the 750 test_bash_operator.py.txt DAGs
4. no plugins
5. no custom logging configs
6. no custom webserver_config.py
7. use SimpleAuthManager

[zachliu]

i configured high memory allowance for worker (4Gb) and low allowance (1Gb) for dag-processor
no memory leakage observed on api-server, scheduler, & triggerer

[roks531]

I noticed similar behavior. Using 3.0.1 I get memory leaks. I just run airflow for less then a day and it used 6Gb more as the 2.9.x versions. It was constantly increasing so I restarted the containers to get back to normal.

By memory allowance you mean you configured the container max memory or is there a specific airflow variable?

Did that impact the performance?

[zachliu]

@roks531 yeah it's the container max memory. afaik it doesn't impact the performance as long as you don't set it too low for the memory hogs like the worker.

[rubnogueira]

After upgrading to the latest v3 I'm dealing with the same issues. The memory seems to keep increasing causing major issues on my end. I'm open to help to find the root cause.