April 2021 Release

[pradyunsg]

Hiya @pypa/pip-committers!

Do we want to do a pip 21.1 this month? If yes, who wants to be the RM? :)

[pfmoore]

We should, but I'm not sure if I'll have the bandwidth to be RM.

[sbidoul]

If no-one with prior experience has the bandwidth to do it I can take it.
I would do the release around the 18th or 25th of April, to be confirmed.

I've had a quick look at the procedure, which seems very clear (kudos). I have a couple of questions, though:

• I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?
• At what moment should we update the vendored dependencies ? I'd rather do it not too close to the release date.
• Besides discuss.python.org, what are the places where the release needs to be announced ?

[pradyunsg]

@sbidoul Awesome! Feel free to "assign" this issue to yourself. :)

If no-one with prior experience has the bandwidth to do it I can take it.

I can do it, but I'd prefer to start spreading the knowledge of the release process among more of us.

At what moment should we update the vendored dependencies ? I'd rather do it not too close to the release date.

Do it whenever you want honestly. Closer is better IMO, but as your intuition tells you, too close is bad. :)

I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?

Yea, it's usually a good idea to. It's our way of making sure that CPython's vendored wheel does not lag too far behind PyPI.

Besides discuss.python.org, what are the places where the release needs to be announced ?

This is subjective. I've usually just dropped a discuss.python.org post for most releases.

OTOH, the releases during the resolver rollout were significantly more publicised.

[pfmoore]

I see most PR to CPython (ensurepip) that update the bundled pip also update setuptools : is that something we should do this time too ?

Yes we should. With the 21.0 release, there was a problem backporting to Python 3.8 and 3.9, so 21.0 never got into those releases. The root issue is #9540, which is still open. To be honest, I don't know what to do about this, there's a mess of dependencies here with setuptools and packaging, and I don't know what (if any) movement there has been.

It's bad that we can't get the latest version of pip into Python 3.9, but we're not in control of any of the parts causing the issue. I suggested a "quick fix" here but people weren't enthusiastic, preferring to go for the more correct solution.

I'd consider it a RM call whether we want to push to get 21.1 into CPython. Personally, I'd apply the quick fix if the no-one can confirm that the "proper" fix is going to be in place in time. But I guess it's your call. I'll try to prepare a PR today with a test that pip runs with warnings as errors, and with the "quick fix" (assuming that the test is still failing!) so at least you have something to make a decision on 🙂

[pradyunsg]

We updated/can update the setuptools version, which should fix the breakage? Unless I'm missing something here, this should be fine.

But yea, that's a call for the RM to make. Lemme know if you want someone to take a closer look. I'll be happy to.

[sbidoul]

I admit I'd rather not poke this specific hornet nest myself :) So if you want to have a closer look I'll be happy if you do @pradyunsg

[pfmoore]

We updated/can update the setuptools version, which should fix the breakage?

Possibly.

I just tried to write a test for this (test_warning.py):

def test_pip_works_with_warnings_as_errors(script):
    script.environ['PYTHONWARNINGS'] = 'error'
    script.pip("--version", expect_stderr=False)

I think it should work, but right now I'm getting random test suite failures locally and I don't know why. I've just done the branch rename thing, maybe something happened there, or maybe there are test failures on main at the moment (I saw some PRs about fixing CI against main, but I think they were lint). Anyway, I don't have time to get my dev environment fixed up at the moment, so if someone else wants to add that test, feel free.

I do think that we should have the test, if only to ensure that the ensurepip PR doesn't break (because that's nearly always a last minute or after-the-fact part of the release in my experience, at which point finding out that the CPython tests fail is, shall we say, "unwelcome" 🙂)

If the test runs fine now, then I guess we might have already fixed the issue, but a quick manual test suggests not:

❯ pip --version                 
pip 21.1.dev0 from d:\work\projects\pip\src\pip (python 3.9)
PS 14:02 (.venv) [Git: main] 00:02.474 D:\Work\Projects\pip
❯ python -We -m pip --version
Traceback (most recent call last):
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 57, in parse
    return Version(version)
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
pip._vendor.packaging.version.InvalidVersion: Invalid version: 'setuptools'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Gustav\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Gustav\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\work\projects\pip\src\pip\__main__.py", line 21, in <module>
    from pip._internal.cli.main import main as _main
  File "d:\work\projects\pip\src\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "d:\work\projects\pip\src\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "d:\work\projects\pip\src\pip\_internal\cli\main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "d:\work\projects\pip\src\pip\_internal\cli\cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "d:\work\projects\pip\src\pip\_internal\cli\parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "d:\work\projects\pip\src\pip\_internal\configuration.py", line 21, in <module>
    from pip._internal.exceptions import (
  File "d:\work\projects\pip\src\pip\_internal\exceptions.py", line 7, in <module>
    from pip._vendor.pkg_resources import Distribution
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3252, in <module>
    def _initialize_master_working_set():
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 574, in _build_master
    ws = cls()
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2061, in find_on_path
    path_item_entries = _by_version_descending(filtered)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2034, in _by_version_descending
    return sorted(names, key=_by_version, reverse=True)
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2032, in _by_version
    return [packaging.version.parse(part) for part in parts]
  File "d:\work\projects\pip\src\pip\_vendor\pkg_resources\__init__.py", line 2032, in <listcomp>
    return [packaging.version.parse(part) for part in parts]
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 59, in parse
    return LegacyVersion(version)
  File "d:\work\projects\pip\src\pip\_vendor\packaging\version.py", line 127, in __init__
    warnings.warn(
DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release

Sorry, that really is all I have time to do on this for now.

[pradyunsg]

pip._vendor.packaging.version.InvalidVersion: Invalid version: 'setuptools'

Whelp.

[pfmoore]

Yep. Setuptools is checking random bits of the filename to see if they are versions, relying on the version parser accepting anything without complaint. See here.

[pradyunsg]

I think we should do what you suggested in #9761 (comment), to unblock the CPython updates.

warnings.filterwarnings("ignore", category=DeprecationWarning, module=".*packaging\\.version")

[pfmoore]

Cool. I can't do this right now, as the pip test suite is failing all over the place on my machine. I've just spent ages tracking down an issue that makes virtualenv fail on my PC (fix incoming) and now I'm getting failures in the unit tests:

FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__has_non_yanked - assert 2 == 0
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__all_yanked - assert 3 == 1
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__yanked_reason[-<none given>] - assert 3 == 1
FAILED tests/unit/test_resolution_legacy_resolver.py::TestYankedWarning::test_sort_best_candidate__yanked_reason[curly quote: \u2018-curly quote: \u2018] - assert 3 == 1

It looks like someone introduced some additional output, and mucked up testing somehow - one of the outputs that was expected to get one line is:

DEBUG    pip._internal.index.package_finder:package_finder.py:255 Given no hashes to check 1 links for project 'pkg': discarding no candidates
DEBUG    pip._internal.index.package_finder:package_finder.py:957 Using version 1.0 (newest of versions: 1.0)
WARNING  pip._internal.resolution.legacy.resolver:resolver.py:278 The candidate selected for download or install is a yanked version: 'mypackage' candidate (version 1.0 at https://example.com/pkg-1.0.tar.gz)

Those debug lines look suspicious 🙁

I think this actions run is the latest CI against main, but it seems to be broken. So I can't be sure if main is passing the tests in CI right now...

I've spent about as much time as I'm willing to for today tracking down test suite bugs and existing problems - so if anyone has a quick suggestion, please let me know. Otherwise, I'll get back to looking at adding that warning (and a test to check it does the job!) when things settle down, or I have more time/energy.

[pfmoore]

#9779 created for the "warnings as errors" problem, and #9780 for the test failures noted above.