Enforcing 32/64-bit python

[mrh1997]

Via python_version it is possible to specifiy the exact version of python that shall be used.
But for some Project I need to ensure that only 32bit/64bit Python is used. I.e. if the project uses specific DLLs.

The windows launcher of python allows to specifiy the exact python interpreter by adding "-32" or "-64" to the version. I.e. python_version = "3.6-32". It would be nice if pipenv allows the same interpreter selection.

Alternatively an additional setting in the Pipfile could solve the issue.

[uranusjr]

It is probably not a good idea to combine this to python_version, but I guess a standalone key is possible, e.g.

[requires]
platform_architecture_bits = "32bit"

There is however not a key in PEP 508 denoting this information, which leads me into thinking maybe there are potential technical difficulties. I wonder if @ncoghlan could provide more insights on this.

[techalchemy]

I don’t think we really have any plans to expand the semantics in the Pipfile for this because of the lack of official support (for now). As for the tooling I built it into @sarugaku/pythonfinder with cross platform suppor

[mrh1997]

Could you please explain what you mean with "lack of official support"?
And why could this operation not be cross-platform?

Would you accept a pull-request if you need not to do it yourself?

[techalchemy]

@mrh1997 just because it's not in the pep 508 standard which governs what values we allow in the requires section for python -- There is a pep 508 marker that includes this information which is platform_machine but we don't support it in the pipfile spec currently. This will require some debate probably about the cost/benefit and the challenges of enforcing this especially on non-x86 architectures and I'm not entirely sure what would be required of us to implement this

@ncoghlan -- thoughts?

[uranusjr]

@techalchemy It’s worse. platform_machine specifies the architecture of the machine, not the Python interpreter. A 32-bit interpreter running on 64-bit host will still give you AMD64. I don’t think there is an environment marker in PEP 508 that can tell you the interpreter’s bitness at the current time.

[ncoghlan]

D'oh, we completely missed that platform_machine gave the wrong answer for 32-bit binaries running on 64-bit systems back when PEP 345 was defined (which is where the current environment marker set comes from).

@uranusjr's suggestion of defining a platform_architecture_bits marker would work, but I'm also inclined to ask whether it might make sense to work out a way to directly support PEP 425 compatibility tags in the environment marker syntax, so users don't need to learn two different ways of identifying platforms.

The downside of the latter option is that the way we spell the 32-bit vs 64-bit distinction in platform tags is unfortunately platform dependent (it's always x86_64 on Mac OS X, it's x86_64 vs i686 on Linux, and then it's win32 vs win_amd64 on Windows).

Either way, PEP 508 would need a replacement PEP that gives it the same treatment that PEP 566 gave to the core metadata spec:

• make https://packaging.python.org/specifications/dependency-specifiers/ the reference document for the current dependency specifier format (rather than the PEP itself), such that future PEPs (including that one) only need to cover their additions, and not repeat the whole spec
• add any new environment markers that we think are missing (e.g. we may decide to add platform_architecture_bits)
• amend any environment markers that experience has shown are poorly defined (e.g. despite the name, it might be reasonable to decide that platform_machine should be checking against the compatible ABI of the running Python interpreter, not the bit-width of the underlying CPU)

[techalchemy]

This seems correct. In pep 514 the interpreter architecture lives in the registry, which is nice.

In my python finder library I was doing this to determine architecture on posix systems... is this wrong? architecture, _ = platform.architecture(path.path.as_posix())

[uranusjr]

This is the correct way, from my own understanding, except in some edge cases (macOS universal binary),
but this shouldn’t matter for a Python finder, since bitness is not relevant in macOS anyway. Universal binaries are always native to your current architecture.

FWIW I use this to detect whether the interpreter is 64-bit (and assume 32-bit otherwise). I got this from an old project by Kenneth (forgot the name).

output = subprocess.check_output([str(path), 'c', 'import sys; print(sys.maxsize > 2**32)']).strip()
# 'True' if the interpreter is 64-bit.

[mrh1997]

If I understand you correctly, you propose to check if the interpreter has the correct bitness and raise an error if not so?!?
This is perfect on systems where either a 32bit or a 64bit interpreter are installed.
But at least on 64bit windows there are regular use cases where interpreters for 32bit and 64bit are installed side by side (I do not know if this is possible on linux/macOS). In this case I expect pipenv to select the correct interpreter instead of raising an error when accidentially selecting the wrong one.

When looking into the sourcecode, I saw that the current pipenv implementation relies on py.exe (find_python_from_py()) which prefers 64bit and uses 32bit as fallback.

To ensure that the correct bitness (if specified in Pipfile) is used, find_python_from_py() could be modifed depending on the bitness:

• 32bit: enforce 32bit binary by adding postfix "-32" to version (i.e. "py.exe -3.5-32")
• 64bit: use the default interpreter and assume that it is 64bit if available (the default behaviour of py.exe; pep397). To ensure that this actually is 64bit the code from the uranusjr previous commit could be used.

Since python/cpython@870f6a1 (=python launcher 3.7) the option "-64" to enforce 64bits natively (i.e. py -3.5-64) is also supported. So this variant could be tried first in case of 64bit. If it results in an error, the previous solution could be used as fallback.

[uranusjr]

I think the idea is similar to how python_version works—Use this information to create a virtual environment when initialising a project. If the running environment does not match (e.g. because the environment was created manually, thus not using the specified options), raise a warning message about “pipenv check will surely fail”.

As for finder heuristics, unfortunately Linux does allow 32- and 64-bit Python to be installed side-by-side, but does not have an equivalent to py.exe (and PEP 514 in general). This means we have to provide some other ways to check that, hence the above exchange, since python_version = "3.6-32" only makes sense on Windows.

[mrh1997]

The current concept of pipenv relies on the which command. This command allows not only to find the first executable that matches the version, but could return all executables that matches the version. Together with your "version check code snippet" it would be possible to check all found executables for their bitness and take the matching one (if available).

But I am not sure if there is a "standard" way of installing python side by side on linux (After googling around I have the feeling that multiple python architectures on a single machine is problematic). Thus I could imagine that if somebody installs boths architectures side by side, the 64bit installation is done via the paketmanager, while the 32bit installation is done manually and thus might even be not present in the PATH (in which case pipenv would not find the interpreter at all)

If this is true, it might be enough to simply check if the bitness of the found interpreter is OK on linux and macOS, while on windows (where side by side is a usual case) the discussed solution is used to select the correct one.

[uranusjr]

Well… on Ubuntu for example, pythonX.Y defaults to the one matching your architecture, but you can install pythonX.Y:i386 on 64-bit to have them side-by-side. It is very, very convoluted :(

[mrh1997]

According to this link installing pythonX.Y:i386 will uninstall the corresponding 64bit python => no side by side installation possible.

But i.e. pyenv allows to do side by side installations. Thus one can assume that there are side by side installations (I saw that you wrote a tool named pythonup; but this one seems to be available for windows/mac only?!?).

So what do you think about my proposal two comments ago?

[kennethreitz]

out of scope