Possible race condition with psutil

[guillermo-carrasco]

Hi again,

I think that I may have found a possible race condition when counting the memory with psutil of a process using the include_children option. The problem (I think) is in this piece of code in _get_memory:

if include_children:
    for p in process.get_children(recursive=True):
        mem += p.get_memory_info()[0] / _TWO_20

The method get_childrenreturns a list that is used to iterate over and calculate the total memory. It may happen though that one of the child processes dies or finishes before the sum has finished, resulting on an error like this:

Reading configuration from '/pica/h1/guilc/repos/facs/tests/data/bin/fastq_screen.conf'
Using 1 threads for searches
Adding database phiX
Processing /pica/h1/guilc/repos/facs/tests/data/synthetic_fastq/simngs_phiX_100.fastq
Output file /pica/h1/guilc/repos/facs/tests/data/tmp/simngs_phiX_100_screen.txt already exists - skipping
Processing complete
Process MemTimer-2:
Traceback (most recent call last):
  File "/sw/comp/python/2.7_kalkyl/lib/python2.7/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 124, in run
    include_children=self.include_children)
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 52, in _get_memory
    mem += p.get_memory_info()[0] / _TWO_20
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/__init__.py", line 758, in get_memory_info
    return self._platform_impl.get_memory_info()
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/_pslinux.py", line 470, in wrapper
    raise NoSuchProcess(self.pid, self._process_name)
NoSuchProcess: process no longer exists (pid=17442)

It happens randomly, and can be solved encapsulating the sum on a try except statement:

if include_children:
    for p in process.get_children(recursive=True):
        try:
            mem += p.get_memory_info()[0] / _TWO_20
        except NoSuchProcess:
            pass

I'm not sure that this is the best solution though... any comments/ideas? @fabianp @brainstorm

Thanks!

[brainstorm]

That clearly biases the measurement since the information is lost (together with @guillermo-carrasco we saw some strange numbers after this tweak).

I guess the next step would be reproducing the error (in tests?), which might prove tricky. @fabianp, we are having this issue while we run this particular test:

https://github.com/SciLifeLab/facs/blob/master/tests/test_fastqscreen.py#L112

If you see something clearly wrong in the way we call it, please, let us know.

[brainstorm]

@guillermo-carrasco I guess you meant OSError as the exception, since:

  File "/home/roman/.pyenv/versions/2.7.6/lib/python2.7/site-packages/memory_profiler.py", line 53, in _get_memory
    except NoSuchProcess:
NameError: global name 'NoSuchProcess' is not defined

[guillermo-carrasco]

Not really, I meant NoSuchProcess, you have to import it from psutil

[kblin]

This just happened to me on a profiling run. Seeing how the uncaught exception causes the profiling to stop completely, I'd prefer to just miss out on the short-running job's memory usage.

[fabianp]

I pushed a fix in a9e5a02 . Feel free to reopen/send a pull request if it doesn't solve the issue.

[kblin]

Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch.

[fabianp]

It would indeed be great if you can send a pull request for that

…

On Tue, Jan 10, 2017 at 3:10 PM, Kai Blin ***@***.***> wrote: Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/issues/71#issuecomment-271584350>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8hwdisHB0f2IK6Sajep0HT9qZuhW4ks5rQ5E6gaJpZM4Bfo5e> .

I ran into a similar problem on macOS, which I assume is also caused by this issue. I'm trying to track a process which creates tens or hundreds of children which live for a very short time, and occasionally I get the following error:

Traceback (most recent call last):
  File "/Users/sasa/work/proj/env/bin/mprof", line 554, in <module>
    actions[get_action()]()
  File "/Users/sasa/work/proj/env/bin/mprof", line 244, in run_action
    multiprocess=options.multiprocess, stream=f)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/memory_profiler.py", line 358, in memory_usage
    for idx, chldmem in enumerate(_get_child_memory(proc.pid)):
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/memory_profiler.py", line 115, in _get_child_memory
    yield getattr(child, meminfo_attr)()[0] / _TWO_20
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/__init__.py", line 1119, in memory_info
    return self._proc.memory_info()
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 295, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 421, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 352, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sasa/work/proj/env/lib/python2.7/site-packages/psutil/_psosx.py", line 326, in catch_zombie
    raise AccessDenied(proc.pid, proc._name)
psutil.AccessDenied: psutil.AccessDenied (pid=43916)

I'll send a PR to ignore psutil.AccessDenied in addition to psutil.NoSuchProcess shortly.