wrong python installation when system has multiple versions

[gdt]

I am building recent master under pkgsrc.
The build passes --with-python=/usr/pkg/bin/python3.10. On a system with only 3.10, PyNUT.py and test_nutclient.py are installed correctly with the right #! path.

On a system with all of python2.7 python3.9 and python3.10 installed, I get 3 copies of the python files in site-packages for all 3 python versions. I think this is a bug because I specified the python version to configure, and once specified, others should not be found and used. (It also doesn't play well with packaging; nobody would want a package that depends on 3 python versions!)

Additionally, the #! for the 2.7 and 3.9 versions is: #!/usr/pkg/bin/python3.10.

I think the fix is to find and use a single python version. I am not aware of packages that find multiple and install to multiple; the usual approach if python modules are needed in multiple is to install multiple versioned packages, e.g. py39-ups-nut and py310-ups-nut. Each package invokes the build once and thus each build uses one version.

Perhaps this is resolved by #1788?

[jimklimov]

As with other options, configure tries to enable (or tune) as much as possible for the given platform. If users have it and can technically use it, they get it by default.

Given that python 2/3 are essentially different languages, which can coexist (even if they don't on your systems), there is no practical reason for us to enforce it onto users that one excludes the other. It is a somewhat similar conundrum to "you've got python, so I won't install perl bindings - hey, they are both scripting languages!"

Packagers can use --without-python2 for that, for example. Or deliver sub-packages under a "nut" meta-package with different dependencies spelled out, same as for drivers for different media (e.g. to get the suitable set of files for the Python generation based on what a user does have installed).

The #1788 PR should however resolve a discrepancy of specifying one --with-python parameter explicitly, and guessing a (currently different) version for --with-python2/3. With that PR's code, it would first check if the preferred PYTHON reports a major version 2 or 3, and will try to resolve the realpath location of its binary to best correspond to that version's "site-packages" and interpreter program locations to use in the script shebangs and installation paths.

• Note: I think the shebang in PyNUT module has no practical meaning, since it has no main method or other standalone logic.

[gdt]

I understand the idea of enabling what is found, but installing to multiple python versions is basically unprecedented. Technically each python version is a different language. Packaging systems tend to view them as something one can build for separately, vs a single package producing multiple at once, is what I am trying to say.

I'm not objecting to being able to do this, just that I'm ending up in a bad state. So I can easily add --without-python2, and it sounds like #1788, once merged, should resolve this issue.

I think you are right about the PyNUT.py shebang, but the testclient can be run. However pkgsrc requires that if there is a shebang, it has to be valid. So I'm ok with dropping it, but it can't be env and it can't be other than the pkgsrc-chosen python version. It's fine for it to be the right python version, as it is now for the files installed for the chosen version.

[jimklimov]

Technically it can be env (maybe bad practice, shooting in the foot and all that), and for a self-contained NUT-Monitor app co-bundling a ../modules/PyNUT it does work for any tested Python version (2.7 and 3.x) as picked by run-time PATH settings.

So again, NUT as a portable project is not the right place to impose constraints on users and uses of the things that can work. Constraints are distro packagers' job ("we can only support combo X") and NUT like other projects provides the toggles and levers to help fine-tune all that.

When sitting on several chairs, don't forget to switch hats too ;)

[gdt]

With shebang and env, I'm not complaining about what can be configured, just that it has to be easy to configure to meet packaging norms. So "can't be" was meant to be "in the mode that pkgsrc uses", and also "the standard approach should have the path found at configure time" because it's unsound for a package to have a variable reference, even if some people sometimes find that useful.

[jimklimov]

It really depends on the packaging tech. Some are flexible enough about "Provides" or "Alias" concepts, so may well have a meta-package "python" which is fulfilled by installing at least one of "python2 || python3" which in turn reference a preferred (or one-or-more-of) specific versions.

For this example, if a user has "some python" installed as the explicit request (and that happens to pull a particular version today and another tomorrow, if constraints allow to drop the old one), that may be enough info to install NUT sub-packages that require the specific version and deliver modules to it (and also impose the constraint that it can not be removed).

[gdt]

This seems fixed enough with the merge.

[jimklimov]

Cheers @gdt - did something break?

[gdt]

I'm having trouble again. My system has python3.10 and python3.11 both, because I'm in the middle of upgrading. Despite

CONFIGURE_ARGS+=        --without-python2
CONFIGURE_ARGS+=        --with-python=${PYTHONBIN}

I am getting pyNUT.py and test_nutclient.py installed in the python site-lib for python3.10, despite passing python3.11 as --with-python. While I know the correct answer is that installing into other python versions is Just Wrong :) and you see it differently, it needs to be easy for packagers to control this. I will get my package updated to the most recent git and retest, but I wanted to point out that this behavior is still bad for packaging, and I still believe that almost all people who run any successful package do so via packages, so packaging is the primary target for build systems.

With --with-python3 instead of --with-python, apparently I get files installed only in 3.11. This seems like a bug; it seems obvious to me that --with-python=/usr/pkg/bin/python3.11 means only that python. This degree of magic really needs explaining in INSTALL.nut.

I again note that I have never run into this sort of behavior with any other program. They all find a single python version and use it. Packages expect to build N times for N versions.

[gdt]

I'm having trouble again. My system has python3.10 and python3.11 both, because I'm in the middle of upgrading. Despite

CONFIGURE_ARGS+=        --without-python2
CONFIGURE_ARGS+=        --with-python=${PYTHONBIN}

I am getting pyNUT.py and test_nutclient.py installed in the python site-lib for python3.10, despite passing python3.11 as --with-python. While I know the correct answer is that installing into other python versions is Just Wrong :) and you see it differently, it needs to be easy for packagers to control this. I will get my package updated to the most recent git and retest, but I wanted to point out that this behavior is still bad for packaging, and I still believe that almost all people who run any successful package do so via packages, so packaging is the primary target for build systems.

With --with-python3 instead of --with-python, apparently I get files installed only in 3.11. This seems like a bug; it seems obvious to me that --with-python=/usr/pkg/bin/python3.11 means only that python, and I don't follow why --with-python3 should suppress the behavior while --with-python should not. Actually the help documents --with-python3 as only applying to code that is incompatible with python2:

  --with-python           Use a particular program name of the python
                          interpeter (auto)
  --with-python2          Use a particular program name of the python2
                          interpeter for code that needs that version and is
                          not compatible with python3 (auto)
  --with-python3          Use a particular program name of the python3
                          interpeter for code that needs that version and is
                          not compatible with python2 (auto)

This degree of magic really needs explaining in INSTALL.nut if it's staying, but I think it's too much and isn't working as intended.

I have never run into this sort of behavior with any other program. They all find a single python version and use it. Packages expect to build N times for N versions of pythons, to make separate packages.

(This is with 20230126, which is what is in pkgsrc-wip; I'll update and retest. But this almost certainly applies to 2.8.0.)