psutil is never installed on Windows

[gvanrossum]

On Windows, it turns out pyperf (and hence pyperformance) will always print a warning when psutil is not installed. The warning is:

WARNING: unable to increase process priority

It is printed by _process_priority(). (Note the if not MS_WINDOWS: return right before.)

The function set_priority() silently returns None, but the caller prints the warning in that case.

So now for every benchmark run by pyperformance it calls this _process_priority() function for each run, printing the warning each time (maybe 10 times per run).

The solution of course is to install psutil. But no matter what I do, because the way venvs used to run the benchmark are created, those venvs do not get psutil installed, even if it is installed in site-packages. (At least, when running from a dev environment.)

• I tried adding "psutil" to the install_requires line in pyperformance's setup.py, but that doesn't seem to help.
• psutil is already listed in requirements.in, so that doesn't seem to make a difference.
• It is also listed in pyperformance/data-files/requirements.txt, apparently with no effect.

So how do I add this dependency to every package? Surely there's a more elegant way than adding it to every benchmark-specific requirements.txt file in pyperformance/data-files/benchmarks?

[corona10]

@vstinner cc @gvanrossum @ericsnowcurrently
IMO, it should be solved from pyperf side instead of pyperformance project.
WDYT?

[gvanrossum]

No, I think this has to do with how pyperformance manages venvs. @ericsnowcurrently

[vstinner]

pyperformance does install psutil in VirtualEnvironment._install_req():

        # install optional requirements
        for req in requirements.optional:
            cmd = pip_program + ['install', '-U', req]
            exitcode = self.run_cmd_nocheck(cmd)
            if exitcode:
                print("WARNING: failed to install %s" % req)
                print()

But if the installation fails, it only logs a warning.

psutil is marked as an "optional" dependency because I failed issues sometimes to install it on psutil or on development versions of Python:

        # parse requirements
        filename = os.path.join(PERFORMANCE_ROOT, 'requirements.txt')
        requirements = Requirements(filename,
                                    # FIXME: don't hardcode requirements
                                    ['psutil', 'dulwich'])

It's listed in pyperformance/requirements.in:

# Optional dependencies
# ---------------------
#
# The list of optional dependencies is hardcoded in pyperformance/venv.py

psutil
dulwich                            # bm_dulwich_log

[vstinner]

I tried adding "psutil" to the install_requires line in pyperformance's setup.py, but that doesn't seem to help.

Currently, psutil is hardcoded as an "optional dependency" in venv.py: it tries to install it, but if pip fails, it only logs a warning.

Do you see this warning when you just create a venv using the python3 -m pyperformance venv create command?

[gvanrossum]

Victor, it looks like you're looking at an earlier version of pyperformance. I suspect that @ericsnowcurrently accidentally broke this (perhaps only for Windows).

[gvanrossum]

Do you see this warning when you just create a venv using the python3 -m pyperformance venv create command?

No, I don't see that warning, and psutil is not present in the created venv.

[ericsnowcurrently]

#170 should fix it. I suspect that I originally thought psutil was in pyperf's requirements.txt (it isn't), but didn't check.