Limit workers in ProcessPoolExecutor #1747
Conversation
|
I'm not entirely sure why more than one build backend has pools executing simultaneously. If that's a necessary feature and not something we can fix, maybe having a single shared pool for all backends would help this? Or switching to using just a |
|
Could be the same issue as when using pytest-xdist: sometimes the number of processor detected is really high, leading to too many concurrent jobs and crash / timeout issues with PyPy. I agree with you that this "feature" does not seem necessary. |
As a mitigation for pypa#1730, this commit limits the number of workers in the ProcessPoolExecutor to 1 (default is the number of CPUs). On PyPy, having a higher number of available workers dramatically increases the number of concurrent processes, leading to some resource exhaustion issues. This does not address the root issue, but should improve the situation until the root issue is addressed.
|
It seems like #1746 has mitigated the problem on Travis, but I think we should still merge this until we figure out why it's spawning even as many processes as it is, because I'm still seeing pypy failures when I run the test suite locally without this change. |
As a mitigation for #1730, this commit limits the number of workers in the
ProcessPoolExecutorto 1 (default is the number of CPUs). On PyPy, having a higher number of available workers dramatically increases the number of concurrent processes, leading to some resource exhaustion issues.This does not address the root issue, but should improve the situation until the root issue is addressed.