Memory not released once the execution completes

[giriannamalai]

HI,
I'm using this. The queueing and call back are perfectly working fine.
But after its completed. It becomes a zombie process.

Any way to close clear it. Or it is related to add_done_callback() feature?

[giriannamalai]

App context we copied were not released If we use add_done_callback()
any fast solution is there>?

[dchevell]

Hey @giriannamalai, can you provide a basic example to reproduce and observe the problem?

[giriannamalai]

Ok. I have
app.config['EXECUTOR_TYPE'] = 'process'
app.config['EXECUTOR_MAX_WORKERS'] = 4

executor created. I did a job that runs in background. Added a callback too.

def handler(future):
    import traceback
    import logging
    logging.basicConfig()
    logger = logging.getLogger('logger')
    try:
        res = future.result()
    except Exception as e:
        logger.exception('Failed: ' + str(e))
        traceback.print_exc()    
    return ""

executor.submit(fn, args)
executor.add_done_callback(handler)

The task perfectly completed but the created process in sleep state with memory of 120-250MB. If I run another and its stacking. fn is a Network IO task. Will take around 10 mins to complete and consume 100 MB at run time. I did a memory profiler too to find the leak. If I run the task without executor, Its running successfully memory clears as well.

[dchevell]

Alright @giriannamalai I've done some testing around this and found that I can reproduce this same behaviour when bypassing Flask-Executor and using the built in concurrent.futures library directly. With that being the case I'm going to forget about Flask-Executor for a moment and just focus on the behaviour of the standard library.

Subprocesses created by a concurrent.futures.ProcessPoolExecutor will hold onto the result of the last task they ran until a new task is received. Once a new task is received, the previous values will be dereferenced and garbage collected. You can see this in the following example (watch memory usage with a memory profiler):

import concurrent.futures
import time
executor = concurrent.futures.ProcessPoolExecutor(max_workers=1)

def big_val():
    return [{1:1} for i in range(1, 1000000)]

def donothing():
    pass

executor.submit(big_val)

# check the memory usage of your subprocess during the sleep interval
time.sleep(5)


# submit a dummy task:
executor.submit(donothing)

# check memory usage of your subprocess again, it will go back down
time.sleep(5)

If you add a callback handler, you're setting up a task that will be run in the original thread or process that submitted the task - in this case, your parent process. That means your parent process will also get a full copy of the return value, and this will stay in memory until the future object goes away. In a Flask app, that should happen automatically at the end of a request, so you might not even be seeing this side of the memory usage.

I'm assuming you're focused on the memory usage of the child processes here based on the fact that Flask requests should be garbage collected as soon as they end as mentioned above. In that case, the only workaround I know of is to submit a dummy task, however having said that I'm going to investigate this further. As I mentioned, this isn't Flask-Executor behaviour, it's the behaviour of the standard library's concurrent.futures subprocesses (which is in turn a wrapper around the multiprocessing module - all the same I'll leave this open a few more days and let you know what I find out about how to deal with this in general.

[giriannamalai]

Thanks for the reply and for the excellent library.
I get it. Its the concurrent.futures behavior.
I'm going to temporarily disable the handler, that will save memory. As well as i can kill the spawned process within it, once it reaches the output. So It will not be in the memory or list.

If you found a fix for this, Please post there...!

[giriannamalai]

hi,
I killed the background process using "kill -9 pid". But there are zombies in the process list. The concurrent futures will spawn parent and child? Am I killed the child only?

[dchevell]

Hey @giriannamalai , I have a few more findings for you:

If you're concerned about memory being held by the child/worker processes, the only workaround is to submit a new task. I've tracked down where the return value of worker processes is held in the concurrent.futures library, and have submitted a bug report and pull request to python/cpython to have worker processes to dereference return values once they're done with and thereby allow worker processes to discard the return value once it's returned it. I've compiled cpython locally and tested a fix for this and can verify that it works; no guarantees on when (or if!) it will ever be merged.

https://bugs.python.org/issue35715
python/cpython#11514

If you're concerned about memory being held by the parent process when you add a callback handler, this can be resolved by removing any held references to the Future object so that Python can garbage collect the Future once it's done. There are two ways you can do this:

1. Delete the variable holding your Future immediately after you add your done callback. The callback will still run, at which point there will no longer be any remaining references to the Future and it will be garbage collected:

   future = executor.submit(fn, args)
   future.add_done_callback(handler)
   del future
   

2. Never assign your Future to a variable in the first place. You would therefore add your handler like this:

   executor.submit(fn, args).add_done_callback(handler)
   

I've tested both these approaches and in each case the handler runs, then the parent process garbage collects the Future result.

Let me know if I can do anything else on this one before closing it out @giriannamalai !

[giriannamalai]

Ok. I'm gonna implement and get back to you...!

[giriannamalai]

Working... Issue fixed

[dchevell]

For reference @giriannamalai, I was successfully able to merge a fix into cpython so that child processes will release the return value of their task from memory once it's been sent to the parent application: python/cpython@962bdea

Not sure when this will make it in, but presumably a future version of 3.8 should have the fix!