Memory leak both scheduler and triggerer

[sawaca96]

I'm experiencing memory leak in scheduler and triggerer, but I don't know reason. I want to know how to fix and debugging method

At same interval, scheduler cache increase 150MB and scheduler logs total 149MB. but scheduler increase 600MB.

Also, I experienced this problem in community helm chart. so I changed to official chart to solve this problem, but still occurring.... Is it a problem caused by parsing the dag? If my dag structure is wrong?

Please help....🙏

[potiuk]

There is no "universal debugging method" - just use whatever you usually use in this case to debug python memory leaks. You have not written which version of Airflow you are using, but we do not see a widespread memory issues from others in the latest version of Airflow so this is indeed likely something in your code.

Ideally - in order to narrow-down the root cause of it, you should run a series of experiements - running a pure scheduler with no dags, no external script, adding ver simple example dags and let them run, adding any customisations you have, adding your DAGs. That's what I'd do. This way you can pin-point the root cause and that might help in narrowing down further investigations. Knowing what causes the problem, you can then either use more sophisticate techniques (See below) or gather more evidences - logs, more traces (actually afrer knowing what causes it will help you to decide what are the right evidences).

This is what I woudl do. And you are the only one who can make such investigation I am afraid, No-one will help you without having access to your system.

Re - more sophisticated tracing - I googled and found for example this link on how to trace memory issues, but I am not sure if it is good for you - you can generreally start looking there.

https://stackoverflow.com/questions/1435415/python-memory-leaks

[sawaca96]

Thanks for reply. I will try variaous experiments.

I use airflow versions 2.2.5 and helm chart 1.6.0. also tried at airflow 2.3.2 but not working

[sawaca96]

@potiuk
For the test, I deploy the airflow with below values

defaultAirflowTag: "2.3.2-python3.8"

airflowVersion: "2.3.2"


fernetKeySecretName: airflow-fernet-key

webserverSecretKeySecretName: airflow-webserver-secret

executor: "KubernetesExecutor"

There is No DAG But memory still increasing

Now I use eks. Do I have to investigate the eks cluster?

[potiuk]

For the reference - you can read about all kinds of memory, all kinds of investigations and trials in this issue #14924. There are lots of links and explanations there discussing various memory allocation phenomena happening on Linux/Kubernetes, so you can use it as a guide in investigation. There are also a number of links to various detailed explanations of different resources.

The problem is that without doing detailed analysis (following those links there) you do not know if you are looking at the right memory and whether it is a problem or not.

And reasons for memory growth might be different - from deployment issue, logging, sidecars you might have, libraries you are using. It also might be, this is perfectly OK because some part of the deployment is filling the Unix Cach memory for files. Unix Cache memory growing for files is perfectly OK and if you are looking at the system as a whole, you might observe continuous growth of "memory" but this memory will be freed at the very instant it is needed and it used by the operating system to speed up file operations.

Looking at individual memory is also not always indicative of problems. Unix uses copy-on-write mechanism which means that it's perfectly fine for two processes to take 4GB each on 6GB system - because for example 3GB of the memory is shared between the process. So if you really want to investigate memory issues, you need to go the rabbit hole and understand what's going on.

I do not want to re-type again what has been said written in #14924 so the easiest way to follow the investigation for you is to go through the discussion there and apply what's relevant for you.

[potiuk]

Yoy need to see what process is leaking the memory. Overal container memory metrics is not enough.

[potiuk]

For example - if you are using shared filesystem, it could be the shared filesystem process that is leaking it.... But that's only one of the examples.

[potiuk]

It could be logging library you use. or faulty database driver,.... you name it.

[potiuk]

Once you know which process is leaking memory - if it's python, you can use for example tracemalloc to see the reason: https://www.fugue.co/blog/diagnosing-and-fixing-memory-leaks-in-python.html

[vladiceanu]

+1 here, we're also seeing memory leakage on Triggerer, Airflow 2.2.5. Fresh setup based on configs from previous Airflow 1.10.13.

[potiuk]

Still - more details are needed to investigate. I think without those who experience it tracking the root cause of it, there is not much anyone can do. Any kind of clues, suggestions and observations is helpful - but without it we are pretty blind here and the only thing we can recommend is to restart scheduler periodically.

We really need help of those who manage their platforms and their expertise in tracking similar memory issues to be able to do anything about it.

Going the pin-pointing route described above to investigate which of the customisations and configurations caused it - starting from teh "baseline" airflow with no modifications whatsoever is a very good route (that we cannot do without access to your systems, I am afraid).

[ejk43]

Hi there!

We are running airflow in EKS using the community supported airflow helm chart v8.6.1. We saw this memory leak on airflow 2.3.2, and just updated to airflow v2.4.3. Very similar behavior as reported above: linear memory increase for both airflow-triggerer and airflow-scheduler. We're running two replicas and see this behavior on both pods across two separate environments (dev vs prod)...

airflow-triggerer pods over the last 16 hours (blue = CPU usage in nanocores, red = Memory usage in bytes)

airflow-scheduler pods

I also realize we're just another data point who has not investigated yet :) Has anyone had any luck tracing down the issue? Wanted to reach out before we look deeper. Thanks!

[potiuk]

I think the first thing to look at is to see what kind of memory is leaked. As discussed before - not all memory is the same. Any deeper looks and investiation would be useful.

[ahipp13]

Just to comment in here so everybody knows, we are having the same problem as well on Airflow 2.5. I will make a more in depth post once I am done completely debugging. We are currently trying to go back to a default Airflow 2.5 image to see if we see the same leaks there, and then adding on stuff custom to our envs.

[potiuk]

Ok. Cool. Please let us know.

[ahipp13]

So wanted to give an update on my testing so far. After doing this I don't know if it is a problem or not, but want to get your thoughts @potiuk.

Before I went to as default of an airflow image as I can get, I first wanted to get some information on what it was doing currently in my setup. Here are some pictures of the scheduler and triggerer working set and cache memory before I went to the default image, as well as processes on the pods:

Scheduler:

Scheduler Pictures

Triggerer:

Triggerer Pictures

As you can see, they both get up to around 2 or 3 GBs and then that is where they sit, and it looks like the cache is fine. When I saw this, I had a feeling it might not be a problem, but I still wanted to do some more testing just to see what would happen.

So, what I did was went to this image (I installed procps so I could see the processes on the pods):

FROM docker.repo1.uhc.com/apache/airflow:2.5.0-python3.8
USER root
RUN apt-get update
RUN apt-get -y install procps

I also took out my OAuth and went back to the default config, and I also removed remote logging. I let this run for 2 days, and made sure no DAGs were running. The results are below:

Scheduler:

Default Scheduler Pictures

working set mem: cache: processes:

Triggerer:

Default Triggerer Pictures

working set mem: cache: processes:

As you can see by these pictures, cache stays normal, and the scheduler and triggerer both get up to about a GB in working set mem and then stabilize.

For one more test, what I wanted to do was enable our DAGs to run over the weekend to see what it did to the memory We have 8 DAGs in dev, but only have 5 enabled to run on a daily basis. Our DAGs require the install of different libraries, so this is what I changed the Dockerfile to:

FROM docker.repo1.uhc.com/apache/airflow:2.5.0-python3.8
USER root

# installing for DAG (These are for accessing Hive)
RUN apt-get update
RUN apt-get -y install python-dev libsasl2-dev gcc g++ vim procps
# switching to airflow user in order to use pip
USER airflow
RUN pip install --upgrade pip
# For Hive Connection
RUN pip install thrift
RUN pip install sasl
RUN pip install thrift_sasl
RUN pip install pyhive
# For Snowflake Connection
RUN pip install snowflake-connector-python
RUN pip install snowflake-sqlalchemy
# to upgrade this to latest version (as it was getting errors when trying to read logs)
RUN pip install --upgrade apache-airflow-providers-microsoft-azure

I let this run over the weekend, and these are the results:

Scheduler:

Default DAG Working Scheduler Pictures

working set mem: cache: processes:

Triggerer:

Default DAG Working Triggerer Pictures

working set mem: cache: processes:

As you can see, it looks like the Triggerer spiked back up to around that 3GB mark and then stabilized, while the Scheduler stayed at around 1GB.

So I guess my question is, is this really a problem for us? The stabilization of the mem is what is making me think it really isn't a problem. I guess my bigger question is is having the scheduler and triggerer running at this high of working set mem normal? Is it dependent on the DAGs we are running and libraries we are installing? Should I look into the libraries? Does it depend on how much resources we have available to us?

Sorry for the flurry of questions, I am just trying to discern what the next steps are to take (if any).

[potiuk]

This is soooo cool analysis. Thanks for doing it @ahipp13 !

So I guess my question is, is this really a problem for us? The stabilization of the mem is what is making me think it really isn't a problem.

I agree that if we see stabilisation of the memory used in all cases, this means we likely do not have a problem.

I guess my bigger question is is having the scheduler and triggerer running at this high of working set mem normal?

Yes. I think it is not particularly High. When you look at the min requirements of our "docker-compose" documentation - 4GB is recommended as minimum memory: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html and we based it on observation that the default 2GB of memory that Docker engine gets with Docker Desktop on MacOS (2GB) is next to imposible to run Airflow docker-compose.

Is it dependent on the DAGs we are running and libraries we are installing?

Absolutely yes. Python interpreter loading libraries and parsing DAGS will absolutely need more memory to import libraries. This is reallly the side-effect of how importing works. Just loading modules during import take heap memory - and especially when you have c-compiled libraries that have .so files might allocate and use even more memory outside of regular Python's heap.

Short experiment:

Running Python command:

Importing airflow in it:

Further Importing pandas in it:

And this is just a beginning - because many of those libraries will allocate more memory for caching and other purposes while running. If you see that just importing airflow and pandas (without running anything) takes 90 MB, suddenly 4GB seems not that far off, taking into account that Airflow currently has 650 (!) dependencies - when you install all providers. That's A LOT. And just using Google, AWS and few other libraries and importing DAGs might easily go into GBs. And if you add webserver with flask and multiple levelst of caching, DB caching internally, etc. etc. 4GB will not seem like a lot suddenly.

Should I look into the libraries?

I think it is a bit futile effort. You have very little control over which library uses how much memory. You can at most limit which libraries are used, but if you want to use some of the operators/sensors/plugins.

Does it depend on how much resources we have available to us?

Well. I think you need to simply live with the fact that if you want to use airflow for serious work, serious memory amount is needed. If the memory reaches certain level and stabilizes, then this should be the indication that this is the amount of memory you need. You might try to limit some of the library memory usages, I guess but this is hard to track and specific per-library really, so I am not sure if any general advice can be given here.

[ahipp13]

This is exactly what I was looking for. Thank you! It makes sense how all of it adds up, especially the importing of other libraries in DAGs on top of everything else Airflow already has in it. Since it was stabilizing I had a feeling it wasn't a problem but I just wanted to make sure that the memory level was normal. Thanks again!

[aj-cloete]

Super late reply, but I suspect I know what's going on here. It's got to do with the log retention settings, e.g. triggerer.logGroomerSidecar.retentionDays which is set to 15 days in the default Values.yaml. It would explain why it goes up almost linearly and then stabilises eventually. I'm taking the setting down to 3 days to see if I can confirm my suspicions. 🤔

[aj-cloete]

Turns out it wasn't as simple as I thought. The linear increase in memory usage persists, even after reducing the triggerer.logGroomerSidecar.retentionDays to 3

[manugarri]

not sure if the issue is related, but in my company we just updated to 2.7.2 and we are observing an abnormal number of error code -9 due to the MWAA workers running out of memory.

[potiuk]

Did you raise it MWAA support ? What do they say?

[manugarri]

unfortunately my company has the Basic Support plan, so we cant raise support tickets.

[potiuk]

Maybe good idea to buy a better plan. We have no idea what is in MWAA so seeking help here for problems seen in managed instance is not something we can really help with. We have completely no idea what -9 in MWAA is - for the context. So you won't find here information that will help you with anything MWAA specific.