bpo-35715: Liberate return value of _process_worker

[dchevell]

ProcessPoolExecutor workers will hold the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's Future or else effectively discarded by this point, the memory can be safely released.

Simple case to reproduce:

import concurrent.futures
import time

executor = concurrent.futures.ProcessPoolExecutor(max_workers=1)

def big_val():
    return [{1:1} for i in range(1, 1000000)]

executor.submit(big_val)

# Observe the memory usage of the process worker during the sleep interval
time.sleep(10)

This should be easily addressed by having the worker explicitly del r after calling _sendback_result as it already does this for call_item.

https://bugs.python.org/issue35715

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

[dchevell]

Signed the PSF contributor agreement

[pablogsal]

This looks incorrect as this will raise a NameError if the first call to call_item fails because the symbol was never bound.

[dchevell]

@pablogsal thanks for catching that. I've modified to handle this case and do nothing - although I'm not generally a fan of passing in except clauses it seems appropriate here.

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[pablogsal]

You just need to delete the result in the else clause of the previous try block unless I am missing something.

[dchevell]

Fair point, thanks @pablogsal !

[dchevell]

@pablogsal any other feedback on this one?

[dchevell]

Hey @pablogsal, any other feedback or changes needed here?

[pablogsal]

Sorry for the delay. Please, ping me if I have not landed this this week :)

[dchevell]

@pablogsal did you mean to ping you about merging, or did I misunderstand?

No need to apologise for the delay, appreciate you taking the time :)