Improve Python support to adhere to Python versions

[fviernau]

For Python projects it is hard (impossible) to determine to which Python version they apply.
ORT has some mechanism which tries (1) Python 2.x and (2) Python 3. In particular, for the latter Python 3.6 is used.

When installing dependencies via pip install -r requirements.txt from pypi.org, pip considers only the dependencies which are compatible with the used Python version, 3.6.

For example in a Python 3.7 project this can lead to the following two issues:

1. A dependency cannot be resolved by ORT, while it is actually resolvable using the appropriate Python version. This
   happens if none of the versions allowed by the constraints is compatible with Python 3.7.
2. Even worse, the wrong dependency version is resolved. This happens e.g. if the youngest 3.6 compatible version allowed by
   the constraints does not equal the youngest 3.7 compatible version. In the worst case it can happen that the project uses the
   latest version of a dependency while ORT resolves a really old (Python 3.6 compatible) version of that dependency.

[sschuberth]

At least for setup.py-based projects, there might be hints which Python version to use. For example, the classifiers may contain something like

classifiers=[
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
],

This at least tells you the minimum version to use. Similarly, a line like

python_requires=">=3.6",

might be contained. However, both types of entries are a bit inconvenient to parse.

But even if the Python version to use would be known, another problem then is that you actually require that version to be installed. At least for our Dockerfile it's impractical to install all sorts of Python versions beforehand, just in case they might be needed. Maybe bootstrapping the right Python version on the fly would be an option, but doing that right for all supported platform is a lot of effort.

A completely different idea that I have in mind is to generally switch over to using GraalVM for ORT, and leverage its Python support to run pip programmatically, and somehow "pretending" to be the right Python version.

[sschuberth]

Ping @pombredanne, being the Python guy here, do you have any good idea how to solve this problem?

[pombredanne]

@sschuberth a way is to rely on python_requires=">=3.6" in modern setup.cg or setup.py. Classifiers are weak information otherwise.

If you want to go the dynamic pip route in anycase pip would resolve based on it current interpreter version and not some other version of Python. Note that pip now uses this bundled resolver https://github.com/sarugaku/resolvelib which you could also invoke directly and be in control of which os/python version/python/arch combo you are emulating.

With that said there is no absolute right answer as each of these combos may resolve to different deps versions and different packages, unless you work from a lockfile, as each dep may be tagged like here

• https://github.com/nexB/scancode-toolkit/blob/5174a3b21758d0d6d5c1db3c74494d56e49a8e74/setup.cfg#L52
• and here https://github.com/nexB/scancode-toolkit/blob/5174a3b21758d0d6d5c1db3c74494d56e49a8e74/setup.cfg#L96

So to recap: each combo of

• Python interpreter (CPython, Pypy)
• Python version (3.6 to 3.9 at least today)
• OS (Windows, Linux, Mac, etc)
• Architecture (x68, x86_64, ARM, etc)
  ... may yield a different set of:
• built/bundled dependencies (included in a built "wheel")
• external dependencies (resolved and fetched from PyPI)
  ... which furthermore can change over time, unless pinned/locked down.

That said this is nothing really specific to Python AFAIK, just made ore visible here.

However, both types of entries are a bit inconvenient to parse.

ScanCode knows how to parse these alright AFAIK.

[sschuberth]

Note that pip now uses this bundled resolver https://github.com/sarugaku/resolvelib

Does pip really use that library as-is? I thought I read somewhere that it's rather the other way around, with resolvelib being a stand-alone reimplementation of pip's (new) resolver mechanism.

However, both types of entries are a bit inconvenient to parse.

ScanCode knows how to parse these alright AFAIK.

Sure, because ScanCode is written in Python itself 😋

[pombredanne]

Does pip really use that library as-is?

See https://github.com/pypa/pip/tree/0ffff034f376feef189cf32cfba56ddd3a472c70/src/pip/_vendor/resolvelib

The resolvelib resolver library is "vendored" as-is in pip from what I can see (and I reckon I did study the code ;) ). FWIW, the vendoring is because pip cannot have external dependencies itself and needs things bundled to be able to bootstrap standalone.

Sure, because ScanCode is written in Python itself

Actually that's not really an advantage when we do static analysis. There is of course a clear and obvious advantage to use Python overall ;) but in the case of Python manifests proper, setup.cfg is an .ini format, and except for setup.py that needs some code lexing and finicking, (as does Ruby and as would Groovy and Kotlin need it) most everything else is either RFC822-, Toml- or JSON-formatted, so reasonably easy to get to.

[sschuberth]

Actually that's not really an advantage when we do static analysis.

Except that you can more easily use the same Python libraries / functions as pip itself when parsing setup.py.

There is of course a clear and obvious advantage to use Python overall ;)

You came to the wrong place for that statement 😄 When we meet in person again, I'll explain you why I basically like Python's syntax, but its ecosystem (and esp. dependency management) just sucks. I mean, there obviously are not even means to just query the transitive dependency tree without actually downloading all the binaries (that you don't need nor are interested in) and pretending to do a build (which is not what you want to do). But let's continue that discussion elsewhere 😉

[deeplook]

Any progress on this?

[deeplook]

Why is this "inconvenient to parse"?

classifiers=[
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
]

There is always the nice ast package you can use without executing the parsed code...

[nicorikken]

I ran into this issue with the analyzer which was running python 3.6, but the project was developed for a newer python version. It had requirements like numpy~=1.20.3 which at least requires python 3.7:

07:54:22.146 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'virtualenv --version' in '/'...
07:54:22.202 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.analyzer.managers.Pip - Resolving PIP dependencies for '/project/requirements.txt'...
07:54:22.203 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.analyzer.managers.Pip - Creating a virtualenv for the 'project' project directory...
07:54:22.206 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'python3 /tmp/python_compatibility9712707363750178.py -d /project' in '/'...
07:54:29.849 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.analyzer.managers.Pip - Trying to install dependencies using Python 3...
07:54:29.852 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'virtualenv /tmp/ort-project-virtualenv122357601095579656 -p /usr/bin/python3' in '/project'...
07:54:39.758 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running '/tmp/ort-project-virtualenv122357601095579656/bin/pip --trusted-host pypi.org --trusted-host pypi.python.org install pip==18.0' in '/project'...
07:54:41.992 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running '/tmp/ort-project-virtualenv122357601095579656/bin/pip --trusted-host pypi.org --trusted-host pypi.python.org install pipdeptree==0.13.2' in '/project'...
07:54:42.769 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running '/tmp/ort-project-virtualenv122357601095579656/bin/pip --trusted-host pypi.org --trusted-host pypi.python.org install --no-warn-conflicts --prefer-binary -r requirements.txt' in '/project'...
07:54:45.662 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.utils.ProcessCapture - Running '/tmp/ort-project-virtualenv122357601095579656/bin/pip --trusted-host pypi.org --trusted-host pypi.python.org install --no-warn-conflicts --prefer-binary -r requirements.txt' in '/project' failed with exit code 1:
  Could not find a version that satisfies the requirement numpy~=1.20.3 (from -r requirements.txt (line 13)) (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0rc1, 1.13.0rc2, 1.13.0, 1.13.1, 1.13.3, 1.14.0rc1, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0rc1, 1.15.0rc2, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0rc1, 1.16.0rc2, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0rc1, 1.17.0rc2, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0rc1, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0rc1, 1.19.0rc2, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5)
No matching distribution found for numpy~=1.20.3 (from -r requirements.txt (line 13))
You are using pip version 18.0, however version 21.1.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

07:54:45.663 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.analyzer.managers.Pip - Falling back to trying to install dependencies using Python 2...
07:54:45.665 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.ProcessCapture - Running 'virtualenv /tmp/ort-project-virtualenv1856848556700345242 -p /usr/bin/python2' in '/project'...

Considering that the analyzer is creating a virtual environment, it would be nice if it could select a Python version.

For my case I'll try running the analyzer with a newer Python version and running the Scanner with Python 3.6 as is required by Scancode toolkit. This will make it difficult to have an all-in-one Docker image though.