Feature request: ability to override pep425 checks

[lorencarvalho]

(relevant pypa-dev thread)

Hello! I have a use case where I'd like to download wheels, install them into a specific directory, then zip that directory up and move it to a different machine. More details can be found here & here.

I'm currently unable to "install" wheels that are not supported by my current arch/py-version/etc due to the PEP 425 check done via pip._internal.wheel.Wheel.supported.

Would it be possible to expose a way to override this check? I realize this is a bit of an edge case, but my only current alternative is to manually rename wheel files to "appear" supported so they pass the check. What I'd like to propose is either a command line arg (such as --force or --ignore-pep425) or (more preferably) an environment variable to influence this checking.

Here's a (very) simple example that seems to achieve what I'm proposing:

the diff

(venv) lcarvalh-mn1 ~/src/pip git:master ± ❱❱❱ git diff
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
index 8dfe2d66..ddc31a8c 100644
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -603,6 +603,8 @@ class Wheel(object):

     def supported(self, tags=None):
         """Is this wheel supported on this system?"""
+        if os.environ.get('PIP_IGNORE_PEP425', None):
+            return True
         if tags is None:  # for mock
             tags = pep425tags.get_supported()
         return bool(set(tags).intersection(self.file_tags))

Step 1: download a manylinux wheel of numpy on my Darwin workstation

(venv) lcarvalh-mn1 ~/src/pip git:master ± ❱❱❱ pip download --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: numpy
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/71/90/ca61e203e0080a8cef7ac21eca199829fa8d997f7c4da3e985b49d0a107d/numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
Successfully downloaded numpy

Step 2: try a normal install:

(venv) lcarvalh-mn1 ~/src/pip git:master ± ❱❱❱ pip install --target test_dir numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl is not a supported wheel on this platform.

Step 3: set an environment variable and install again

(venv) lcarvalh-mn1 ~/src/pip git:master ± ❱❱❱ PIP_IGNORE_PEP425=1 pip install --target test_dir numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
Processing ./numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.14.3

Thanks for looking!

-- Loren

[pfmoore]

I'm pretty firmly against simply allowing the user to ignore the PEP 425 mechanism. This is something highly specific, as I understand it, to doing a --target install of a local wheel file. Some thoughts come to mind:

1. It's actually not that hard to simply unpack the wheel to the target location. Wheels are designed so that unpacking them by hand is entirely possible. I'm willing to accept that wanting to use pip for consistency is a reasonable desire, though.
2. Any solution should be explicitly limited to the case where we are doing a --target install. The PEP 425 mechanisms are there precisely to protect users against installing incompatible code into their Python installation, and we should maintain that. It's reasonable to assume that --target installs are not intended for direct use by the local python, and so the rules can be relaxed. (This is not without risk, though, and we should probably issue a warning even in that case).
3. We shouldn't automatically consider incompatible wheels as candidates for installation. The only way to install an incompatible wheel should be if the user explicitly specifies it as a requirement.

[warsaw]

Does it make sense to automatically skip the PEP 425 checks if any of --platform, --abi, or --implementation are given? If you're being explicit about those parameters, then you're taking responsibility that they are what you want.

[pfmoore]

Yes it does, but those are only available on pip download. (I thought of that too, but found that they weren't available on install when I went digging...)

Edit: Adding those options to pip install may be worth considering, but I'm not sure they would be a good idea except in conjunction with --target, and I doubt that it'd be feasible to restrict them to that case only.

[lorencarvalho]

I think adding those options to install would be counter-intuitive, especially given the slim use case. I have no issue with those options being right where they are, in the download subcommand. Having a two-step process is fine imho (pip download --dest wheels/ ... pip install -f ./wheels), but having to hand-extract the 'incompatible' wheels is what I'd like to avoid if possible :)

[techalchemy]

Hi, speaking as a maintainer of pipenv who has some significant experience tweaking pip internals (particularly around ignore_compatibility flags specifically aimed at manipulating the representation of pep425 tags) I would offer some strong caution against this as well. We currently have some tooling that 'hand downloads' some wheels and as the project in question seems to describe, also builds a portable zipfile, and we don't really struggle with it.

On the other hand, when we touch this at all for our dependency resolver's cross platform functionality, we usually get some pretty bad results. We have a PR open right now that fixes an issue we caused around this, even. Simply put, if you're going to be redistributing something, you probably shouldn't be doing it from a 'binary' format anyway, which may not include licenses (as we've all learned recently), etc.

[pfmoore]

Looking at the motivating use case (the shiv application) a few questions come to mind:

1. How does including platform-specific wheels help anyway? If you're building a .pyz archive, loading binary extensions don't work anyway. So I don't see how this would help.
2. Assuming it does help, I'd like to have a clear understanding of what the shiv command line interface for creating a .pyz archive that contains (as an example) platform-specific binaries for manylinux1 and Windows. And how those command line arguments would translate into pip commands to download and unpack the relevant wheels.

[lorencarvalho]

If you're building a .pyz archive, loading binary extensions don't work anyway. So I don't see how this would help.

We don't load directly from the zip, we unpack it to the filesystem prior to launching. Not only for the reason you specify, there's also tons of libraries that build paths off of __file__ etc.

Assuming it does help, I'd like to have a clear understanding of what the shiv command line interface for creating a .pyz archive that contains (as an example) platform-specific binaries for manylinux1 and Windows. And how those command line arguments would translate into pip commands to download and unpack the relevant wheels.

Personally, (and internally at LinkedIn) this isn't a big concern. We have build machines for each of the architectures that we target, so we build an artifact for each platform we distribute to. @dan-blanchard put in a PR specifically aiming to support manylinux wheels. The details of which can be found here. But to sum it up, I think he doesn't necessarily have the luxury of a build-matrix and would like to build a single artifact that can get pushed around to a variety of architectures that fall under the manylinux umbrella.

[lorencarvalho]

Sorry, totally spaced on the 2nd question, shiv just delegates all unprocessed args to pip install. There's a few validation checks (like we don't allow --target or anything that conflicts with --target because internally we require it).

So for the case of building a pyz for a-platform-that-isn't-localhost the workflow would be akin to:

pip download --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy --dest wheelhouse
shiv --output-file env.pyz --find-links wheelhouse/ numpy Cython

So, if the diff I proposed initially were merged, the only difference would be

PIP_IGNORE_PEP425=1 shiv --output-file env.pyz --find-links wheelhouse/ numpy Cython

[pfmoore]

We don't load directly from the zip, we unpack it to the filesystem prior to launching.

OK. That's not technically how a .pyz file is supposed to work. Do you bundle the unpacking process in the archive, and automatically do the unpack when the __main__ file is run? If not, could I respectfully ask that you choose a different file extension, as the .pyz extension is defined by Python to mean a zip file that can be run simply using python xxx.pyz (see PEP 441). If you do unpack automatically (and clean up on completion - which will likely be hard as you can't delete loaded DLLs on Windows) then while you're technically OK, I'd still prefer it if you used a different extension, as the expectation is that pyz files get run in place.

Offtopic digression over. Regardless, thanks for the clarification.

shiv just delegates all unprocessed args to pip install
[...]
PIP_IGNORE_PEP425=1 shiv --output-file env.pyz --find-links wheelhouse/ numpy Cython

OK. That example makes me even more unhappy with your proposed solution. Expecting pip to find a wheel that's not for the current platform as part of its normal search seems like it's going to cause way too many problems for users who don't expect that.

See my previous comment. I'm only really willing to consider a solution that is restricted solely to the case of pip install --target XXX numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl (i.e. --target is specified, and the .whl file is explicitly named). And even then, I'd rather the install was explicitly opt-in via something like a --platform manylinux1 command line argument, not a general "ignore compatibility" flag.

[lorencarvalho]

OK. That's not technically how a .pyz file is supposed to work.

Sorry if I wasn't clear, PEP 441 was the inspiration for shiv, and I think we follow it closely (we even use zipapp to build the pyz files internally, though vendored/backported with the compression fix from 3.7). The "unpacking" of the included dependencies (which are basically a staged site-packages dir) is done in the __main__.py.

lcarvalh-mn1 ~ ❱❱❱ shiv -c http httpie -o http -q
lcarvalh-mn1 ~ ❱❱❱ head -n1 http
#!/export/apps/python/3.6/bin/python3 -sE
lcarvalh-mn1 ~ ❱❱❱ unzip -p http __main__.py
# -*- coding: utf-8 -*-
import _bootstrap
_bootstrap.bootstrap()

We don't delete the unpacked files after the fact though, the bootstrap function can re-use a previously unpacked "cache" if it exists (if not it will re-extract). If you're familiar with pex, our tool behaves much the same wrt the deps on disk. That said, the tool itself makes no assumptions about file extension (we just refer to them casually as pyz files in the README), so file name & extension is up to the user supplied input to the --output-file option. If that makes you uncomfortable I'd be totally fine removing any mention of 'pyz' from the readme.

I'm only really willing to consider a solution that is restricted solely to the case of pip install --target...

Yeah! I don't mean to imply that I'm stuck on the example diff I gave, I'm just using it as a sample. I don't want to push on something that makes y'all uncomfortable :)

I totally get your trepidation as well, a feature like this would break a lot of assumptions about how pip is supposed to be used, and that could mean imposing a lot of additional unforeseen labor on y'all.

[pfmoore]

If that makes you uncomfortable I'd be totally fine removing any mention of 'pyz' from the readme.

It's offtopic for here so I don't want to labour the point. It's not like it's that big a deal. I'll have a think about it, and if I feel strongly enough I'll raise an issue over in the shiv repo.

[warsaw]

Ignoring for the moment the __file__ issue, which we are actively trying to kill off, as soon as you have an extension module in your wheel, I don't see any way to avoid some unpacking. @sixninetynine explained elsewhere how I wrote a .so extractor/loader to only unpack the shared library when needed.

[pfmoore]

@warsaw Agreed. Ultimately it's an OS limitation. Personally I prefer to only put pure Python code in a zipapp, and bundle extension modules alongside it if needed. It's a trade-off at the end of the day (single file distribution vs extra complexity with auto-unbundling).