pip package upgrade fails to recognise version compatiblity

[ghost]

• Pip version: 7.1.0
• Python version: 2.6.6
• Operating System: CentOS 6.8

Description:

I updated a package using:
$ pip install ftp-cloudfs --upgrade

After the upgrade, I ran my package ftpcloudfs and got the following issue:

Traceback (most recent call last):
  File "/usr/bin/ftpcloudfs", line 14, in <module>
    from ftpcloudfs.main import Main
  File "/usr/lib/python2.6/site-packages/ftpcloudfs/main.py", line 14, in <module>
    import swiftclient
  File "/usr/lib/python2.6/site-packages/swiftclient/__init__.py", line 20, in <module>
    from .client import *  # noqa
  File "/usr/lib/python2.6/site-packages/swiftclient/client.py", line 136
    return {key: safe_value(key, val) for (key, val) in headers}
                                        ^
SyntaxError: invalid syntax

I reported the issue to ftpcloudfs here: https://github.com/cloudfs/ftp-cloudfs/issues/57
But they advised me it was to do with using the wrong version of swiftclient alongside the wrong version of python.
Instead I've had to explicitly install swiftclient v2.7
$ pip install python-swiftclient==2.7.0

I was expecting pip to provide me with the latest compatible version of packages for my specs.

What I've run:

Collecting ftp-cloudfs
  Downloading ftp-cloudfs-0.35.tar.gz
Collecting pyftpdlib>=1.3.0 (from ftp-cloudfs)
  Downloading pyftpdlib-1.5.1.tar.gz (127kB)
    100% |████████████████████████████████| 131kB 1.7MB/s 
Collecting python-swiftclient>=2.1.0 (from ftp-cloudfs)
  Downloading python_swiftclient-3.1.0-py2.py3-none-any.whl (67kB)
    100% |████████████████████████████████| 69kB 3.2MB/s 
Collecting python-daemon>=1.5.5 (from ftp-cloudfs)
  Downloading python_daemon-2.1.1-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 57kB 4.1MB/s 
Collecting python-memcached (from ftp-cloudfs)
  Downloading python-memcached-1.58.tar.gz
Collecting futures>=2.1.3 (from python-swiftclient>=2.1.0->ftp-cloudfs)
  Downloading futures-3.0.5-py2-none-any.whl
Collecting requests>=1.1 (from python-swiftclient>=2.1.0->ftp-cloudfs)
  Downloading requests-2.11.1-py2.py3-none-any.whl (514kB)
    100% |████████████████████████████████| 516kB 482kB/s 
Collecting six>=1.5.2 (from python-swiftclient>=2.1.0->ftp-cloudfs)
  Downloading six-1.10.0-py2.py3-none-any.whl
Requirement already up-to-date: docutils in /usr/lib/python2.6/site-packages (from python-daemon>=1.5.5->ftp-cloudfs)
Collecting setuptools (from python-daemon>=1.5.5->ftp-cloudfs)
  Downloading setuptools-27.3.0-py2.py3-none-any.whl (467kB)
    100% |████████████████████████████████| 471kB 530kB/s 
Collecting lockfile>=0.10 (from python-daemon>=1.5.5->ftp-cloudfs)
  Downloading lockfile-0.12.2-py2.py3-none-any.whl
Installing collected packages: pyftpdlib, futures, requests, six, python-swiftclient, setuptools, lockfile, python-daemon, python-memcached, ftp-cloudfs
  Found existing installation: pyftpdlib 1.4.0
    Uninstalling pyftpdlib-1.4.0:
      Successfully uninstalled pyftpdlib-1.4.0
  Running setup.py install for pyftpdlib
  Found existing installation: futures 2.2.0
    Uninstalling futures-2.2.0:
      Successfully uninstalled futures-2.2.0
  Found existing installation: requests 2.5.1
    Uninstalling requests-2.5.1:
      Successfully uninstalled requests-2.5.1
  Found existing installation: six 1.9.0
    Uninstalling six-1.9.0:
      Successfully uninstalled six-1.9.0
  Found existing installation: python-swiftclient 2.3.1
    Uninstalling python-swiftclient-2.3.1:
      Successfully uninstalled python-swiftclient-2.3.1
  Found existing installation: setuptools 11.3.1
    Uninstalling setuptools-11.3.1:
      Successfully uninstalled setuptools-11.3.1
  Found existing installation: lockfile 0.10.2
    Uninstalling lockfile-0.10.2:
      Successfully uninstalled lockfile-0.10.2
  Found existing installation: python-daemon 2.0.5
    Uninstalling python-daemon-2.0.5:
      Successfully uninstalled python-daemon-2.0.5
  Found existing installation: python-memcached 1.53
    Uninstalling python-memcached-1.53:
      Successfully uninstalled python-memcached-1.53
  Running setup.py install for python-memcached
  Found existing installation: ftp-cloudfs 0.32
    Uninstalling ftp-cloudfs-0.32:
      Successfully uninstalled ftp-cloudfs-0.32
  Running setup.py install for ftp-cloudfs
Successfully installed ftp-cloudfs futures lockfile pyftpdlib python-daemon python-memcached python-swiftclient requests setuptools six

[pfmoore]

From https://pypi.python.org/pypi/python-swiftclient/3.1.0 it looks like python-swiftclient publish a Python 2/3 compatible wheel for that version. And your log shows

Collecting python-swiftclient>=2.1.0 (from ftp-cloudfs)
  Downloading python_swiftclient-3.1.0-py2.py3-none-any.whl (67kB)

which confirms that. If that wheel contains code that isn't valid syntax for a version of Python they claim to support, then that's a packaging bug on their part, and you should report it to them.

[RonnyPfannschmidt]

btw, what is actually the correct/canonical way to publish universal wheels that do not support python 2.6 ?

[pfmoore]

Simplest answer is to publish two - a Python 2.7 wheel and a Python 3 wheel.

[RonnyPfannschmidt]

i see, can we document that somewhere?
i think literally everyone gets that wrong

i presume its also possible to set the implementation tags to py27.py3?

[pfmoore]

Yeah, that would probably also work.

I'm perfectly OK with documenting it somewhere, but I'm not 100% sure where or how. Publication best practices haven't really settled down to that level yet. (The basic rule, "don't tag a wheel with a version it doesn't support" is both glaringly obvious and also really easy to neglect :-()

[RonnyPfannschmidt]

yes, it is, i am guilty of it too - also these days i think we need to get that out to people

i think most package authors do not realize they do in fact not support python2.6 but their package claims it

[pfmoore]

Radical thought, but maybe these days "universal" should mean py27.py3? If people still want to claim 2.6 support, they have to explicitly request it. That might avoid the worst cases.

[RonnyPfannschmidt]

@dholth any opinion on updating wheel to have universal mean that?

[RonnyPfannschmidt]

@pfmoore another problem is - that a py3 wheel also doesnt tell about python versions not supported
i have a lot of python 3 packages that will not run on 30-3.4 by now

[RonnyPfannschmidt]

@dholth perhapps wheel should deprecate the universal flag and tell people to explicitly specify it

[dholth]

@RonnyPfannschmidt This is a common misunderstanding. py2.py3 does not mean "supports Python 2.6". It means "supports some version of Python 2, and some version of Python 3". If there is no variant of your package that supports Python 2.6 you still say "py2".

Wheel tags are for choosing between different builds of the same version of a package, for example, separate wheels if you are using 2to3 to produce a Python 3 version with different source code, or more commonly, separate wheels for each OS+architecture. With wheel tags the negative is more important: throw away all the available wheels that definitely do not work on your computer, then try to install the higest-ranked one that might.

The requires-python metadata does what you want, it is a way to say exactly which Python versions are supported: #3846

[dstufft]

Wheel tags are for choosing between different builds of the same version of a package

This is important, because otherwise if you don't have a wheel otherwise available for Python 2.6 and someone attempts to install on Python 2.6, it'll just download the sdist and install that anyways. IF you make the setup.py error on 2.6, then you legitimately have different build output on 2.6 and 2.7 and it makes sense to use py27 instead of py2.

[RonnyPfannschmidt]

@dholth at first glance the whole combination is a mess

as for wheels, unless someone puts in the metadata (which at first glance is documented so bad that i have no idea how to specify entries for different major python versions) the wheel that means exactly that

[pfmoore]

OK, I'd misunderstood this too (but as @dstufft points out, sdists are not tagged to Python version, so it's never possible to avoid there being an applicable download for every Python version simply based on what you publish).

So the options for package authors are:

1. Accept that people shouldn't use your code on platforms you don't support, and if they do they should expect breakages. That's the default, and the current status quo. And honestly, it's pretty OK. People should be happy with "we don't support that version" as a response to reports of issues arising on Python versions the project doesn't claim support for.
2. To enforce a failure on particular versions needs the Requires-Python metadata. But this should be used with extreme care, as it's really easy to over-specify. This should not be seen as a way of getting a nicer message for users, but as a way of deliberately locking out particular platforms (for whatever reason).

In the OP's case, none of this is relevant as the project documents that it supports Python 2.6, but it uses constructs that don't work in 2.6. The project should either fix their code, drop support for Python 2.6, or publish a 2.6-only version of the wheel which will be picked up in preference to the universal wheel.

[dholth]

Stuff doesn't work on Python 2.6, and it doesn't work on Python 3.6 maybe, and it definitely doesn't work on Python 3.0 or 3.2. Sometimes the publisher knows this and could possibly include it in some metadata, but often the publisher accidentally breaks support for Python 2.6 or other Python versions they are not using.

IMO this is the basic price you pay for using free software dependencies where you are not paying for the software at all, and are not paying for compatibility guarantees. Any update to any dependency can break you for any reason. Compatibility with interpreter versions is only a tiny part. After that you have the much larger category of API changes and bugs.

[pfmoore]

often the publisher accidentally breaks support for Python 2.6 or other Python versions they are not using

And sometimes the publisher simply doesn't care about those versions, but doesn't want to actively disallow users from using them. That's very commonly the case with newer versions for instance - there are likely very few projects that have actively tried to ensure Python 3.6 support works yet, but I'd be very sad if they included metadata that said "Requires Python 2.7 or 3.3-3.5". It's bad enough having to wait for binary builds for new versions, let's not encourage pure Python projects to do similar (eggs did this, btw, and it was a pain).

[RonnyPfannschmidt]

i would like a mechanism that makes it easy to mark eol'd and broken python versions as not installable there while not excluding new versions (but it might be nice to warn that the package doesn't include explicit support)

[dstufft]

@RonnyPfannschmidt Requires-Python (once we land it) is that mechanism. We don't support OR yet, so the 2.x/3.x makes it a bit funky, but you'd need to do something like >=2.7,!=3.0.*,!=3.1.*,!=3.2.*. If we get OR support than that could be something like `~=2.7 or ~=3.3``.

[RonnyPfannschmidt]

yikes ^^ @dstufft please lets get an or - the second example so much better

[pfmoore]

If we get OR support than that could be something like ~=2.7 or ~=3.3.

Presumably ~=2.7 or >=3.3? The risk of typos is one of the reasons I stand by my statement that you should only use Requires-Python when you really need to.

The way @RonnyPfannschmidt describes his use case, it sounds like he would find a Disallow-Python specifier more useful (which of course nicely sidesteps the need for "or" ;-)). Instead of saying "We only support precisely these versions" that would say "the following versions are unsupported and known not to work, so don't even try".

[dstufft]

@pfmoore ~= will work for any version of Python 3 after 3.3, it won't allow 3.2 or a hypothetical 4.x version. If you want to allow 4.x then yea you'd use >=3.3.

>>> from packaging.specifiers import SpecifierSet as S
>>> S("~=3.3").contains("3.3.0")
True
>>> S("~=3.3").contains("3.3.5")
True
>>> S("~=3.3").contains("3.4.0")
True
>>> S("~=3.3").contains("3.4")
True
>>> S("~=3.3").contains("3.4.7")
True
>>> S("~=3.3").contains("3.2")
False
>>> S("~=3.3").contains("4.0")
False

I'm not sure that a Disallow-Python would be more useful here though. We already have !=, the only thing that's missing is the ability disallow ranges, but that's something that would be useful outside of this narrow use case so is something we should probably add to specifiers anyways.

[pfmoore]

@dstufft ... oh. I knew there were complexities in the version spec stuff that I didn't fully understand, but OK. That seems weird, in the sense that "I don't know how to read ~=3.3 in a way that makes intuitive sense to me". But never mind, the spec trumps my intuition, so fair enough.

I will just say that >=2.7,!=3.0.*,!=3.1.*,!=3.2.* isn't the same as ~=2.7 or ~=3.3 because 4.0 matches the former. Which is of little relevance in practice, certainly. Although there's no reason to assume that 4.0 will be incompatible with 3.x, and we're back to my comment about not over-specifying...

[dstufft]

The ~= operator is basically >=, but staying within the same overall series, it's translated as >=V.N,==V.*. So saying ~=3.3 is saying "it has to be at least 3.3, but still within the 3.x series", whereas saying ~=3.3.0 is saying "it has to be at least 3.3.0 but still within the 3.3.x series".

I honestly accidentally underspecified the more complex string because I intended it to match 2.7 and 3.x but it didn't occur to me to account for 4.x at all :). Whether it makes sense to under-specify or over-specify really depends on the compatibility promises of the thing you're depending on... which I don't actually know if Python is ever planning to release another break-the-world as 4.x or what they're going to do there. If 4.0 is just going to be whatever comes after 3.9 then yea it makes sense to do >=3.3. If 4.0 is going to be another decade long slog of projects needing to update themselves, then ~=3.3 makes more sense.

[dholth]

The over-specified version requirement thing is important. It's way more irritating to have trouble installing a package that would work except it has incorrect metadata than to get a successful install of something that does not quite execute correctly. Maybe you would rather just install, and then edit the 2 lines of code required to get it to run on Python 3.next or 2.previous. This was a problem with egg, which is why pure Python wheels are automatically compatible with all newer versions of Python. This is why I am not that excited about amazing metadata dependency resolution etc. features - they work if the metadata is perfect, but it isn't.

[pfmoore]

@dholth My feelings precisely.

@dstufft Guido's on record (somewhere) as saying "never again" for a big break-the-world release like 2.x->3.x. But either way, you're guessing about the future.

[dstufft]

Closing this, there's nothing for us to do here and the Requires-Python and data-requires-python work exists and is released now.