Black install via pip fails on embedded Python on Windows using latest pip version (20.3)

[robintw]

Describe the bug
A new version of pip has recently been released, which is a lot stricter in many ways. When installing the latest version of Black on a Windows machine using the embedded Python distribution, I get an error:

ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

The package it is trying to download is the latest source release (20.8b1), and from what I can gather, pip finds that the version in the filename (20.8b1) conflicts with the version in the metadata inside the package - which it says is 0.0.0.

To Reproduce

1. Install the embedded Python interpreter on Windows (https://docs.python.org/3/using/windows.html#the-embeddable-package)
2. Install pip in the embedded Python interpreter (download https://bootstrap.pypa.io/get-pip.py and run it using the embedded Python interpreter's python.exe)
3. Run python.exe -m pip install black --no-cache-dir
4. You will see the following output:

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Requested black from https://files.pythonhosted.org/packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz#sha256=1c02557aa099101b9d21496f8a914e9ed2222ef70336404eeeac8edba836fbea has different version in metadata: '0.0.0'

If the latest version which has a wheel release published to PyPI is specified instead, then it works fine:

c:\pepys-deploy\pepys-import>python -m pip install black==19.10b0 --no-cache-dir
Collecting black==19.10b0
  Downloading black-19.10b0-py36-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.1 MB/s
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black==19.10b0) (1.4.1)
Requirement already satisfied: toml>=0.9.4 in .\python\lib\site-packages (from black==19.10b0) (0.10.2)
Requirement already satisfied: regex in .\python\lib\site-packages (from black==19.10b0) (2020.11.13)
Requirement already satisfied: attrs>=18.1.0 in .\python\lib\site-packages (from black==19.10b0) (20.3.0)
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black==19.10b0) (1.4.4)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black==19.10b0) (0.8.1)
Requirement already satisfied: click>=6.5 in .\python\lib\site-packages (from black==19.10b0) (7.1.2)
Installing collected packages: black
  WARNING: The scripts black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-19.10b0

So, it seems to be something to do with the metadata checking before building the wheel which is going wrong.

The latest version of Black worked fine with the previous version of pip which didn't use the new strict metadata checker.

I can't seem to reproduce this in non-embedded Python, but it may be an issue there too. I'm aware that pip isn't officially supported for embedded Python on Windows, but it is widely used and all other packages seem to install fine:

Expected behavior
Expected to install successfully.

Environment (please complete the following information):

• Version: 20.8b1
• OS and Python version: Windows embedded Python 3.7

Does this bug also happen on master?
N/A as related to PyPI releases

[cooperlees]

/me boots Windows VM to debug tonight or some time soon hopefully

[robintw]

Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

[cooperlees]

Just as an additional piece of information: the install works when when you tell pip to use it's legacy resolver which isn't as strict about metadata (see below). However, you can see that it is definitely confused about versions - it states Successfully installed black-0.0.0 at the end.

c:\pepys-deploy\pepys-import>python -m pip install black --no-cache-dir --use-deprecated=legacy-resolver
Collecting black
  Downloading black-20.8b1.tar.gz (1.1 MB)
     |████████████████████████████████| 1.1 MB 3.3 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs in .\python\lib\site-packages (from black) (1.4.4)
Requirement already satisfied: toml>=0.10.1 in .\python\lib\site-packages (from black) (0.10.2)
Requirement already satisfied: pathspec<1,>=0.6 in .\python\lib\site-packages (from black) (0.8.1)
Requirement already satisfied: typed-ast>=1.4.0 in .\python\lib\site-packages (from black) (1.4.1)
Requirement already satisfied: regex>=2020.1.8 in .\python\lib\site-packages (from black) (2020.11.13)
Requirement already satisfied: mypy-extensions>=0.4.3 in .\python\lib\site-packages (from black) (0.4.3)
Requirement already satisfied: click>=7.1.2 in .\python\lib\site-packages (from black) (7.1.2)
Requirement already satisfied: typing-extensions>=3.7.4 in .\python\lib\site-packages (from black) (3.7.4.3)
Building wheels for collected packages: black
  Building wheel for black (PEP 517) ... done
  Created wheel for black: filename=black-0.0.0-py3-none-any.whl size=124180 sha256=9d02c9cda7598e7caf1f73046db0bb8db2a1baf63bb54ac8c606c7ea8b143f82
  Stored in directory: C:\Users\ROBINW~1\AppData\Local\Temp\pip-ephem-wheel-cache-h9rra3jz\wheels\c5\85\79\f3af8daaf8037c0bf14beb3b7a1511a39b6e6902ca2aaf494e
Successfully built black
Installing collected packages: black
  WARNING: The scripts black-primer.exe, black.exe and blackd.exe are installed in 'c:\pepys-deploy\pepys-import\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed black-0.0.0

Note: the legacy resolver will be going away in the next pip release (Jan 2021 apparently), so it isn't a long-term solution.

Yup sure thing. I'll fix and push for a release before then. Thanks for raising.

[ichard26]

I'll also look into this soon. I think this has something to do with setuptools-scm breaking as that's how the version is resolved in our source distributions. Why is it not working as expected? I don't know any answers but I'll do some digging.

[robintw]

I imagine a partial solution would be to publish a wheel for the latest version - installing the latest version that had a wheel (19.x) worked fine for me. Obviously it'd be good to work out why it's actually happening too though!

[ichard26]

We want to do more releases more often so I'm totally +1 for a new release. Need to check that the CHANGELOG is ready though. One thing, I can't do a release. I clearly have some power to push for a release, but no power to create and push one. That power lies with the other collaborators :-)

@cooperlees, what do you think?

[pradyunsg]

Since @cooperlees asked nicely...

You seem to be hitting pypa/setuptools-scm#386. The main solution is listed in pypa/setuptools-scm#386 (comment). I'm not a 100% sure how you're generating release artifacts, (I don't see your release process documented -- it's usually a good idea to do so, to avoid mistakes!) but it does seem that the sdist doesn't have the metadata that setuptools_scm needs to properly version it, likely due to setuptools_scm not being installed/used in the build environment.

Also, even though this is a pure-Python project, I strongly recommend uploading wheels to PyPI. Most users who use pip won't really download/uses the sdist if there's a wheel, so you'd also buy some time to investigate the underlying issue here by uploading a well-formed wheel. I'll leave it up to the maintainers to decide whether they want the 20.8 beta wheel built (checkout the 20.8 tag and build the wheel with pip wheel .?) or if they wanna put out a completely new release. :)