Installing from git repo breaks when upgrading from version 21.2.4 to 21.3

[SigmundVik]

Description

Installing the Python.NET package from GitHub works fine with pip version 21.2.4, but breaks after upgrading to version 21.3.

This was observed using Python version 3.9.7, but the behavior is the same for version 3.10.0.

The error message is:

Error reading git repository information: Unsupported repository version 1. Only versions up to 0 are supported.

Expected behavior

Installing packages from git repos should not break after upgrading pip.

pip version

21.3

Python version

3.9.7

OS

Windows

How to Reproduce

1. python -m pip install pip==21.3
2. pip install git+https://github.com/pythonnet/pythonnet

Output

[BOOTCAMP] [none] Thu 11-11-21 14:21 2.143 23G z:\>python -m pip install pip==21.3
Collecting pip==21.3
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3

[BOOTCAMP] [none] Thu 11-11-21 14:21 5.895 23G z:\>pip install git+https://github.com/pythonnet/pythonnet
Collecting git+https://github.com/pythonnet/pythonnet
  Cloning https://github.com/pythonnet/pythonnet to c:\users\sigmund\appdata\local\temp\pip-req-build-w38ycyw8
  Running command git clone --filter=blob:none -q https://github.com/pythonnet/pythonnet 'C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8'
  Resolved https://github.com/pythonnet/pythonnet to commit ac336a893de14aaf2c7b795568203b48030f9006
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing wheel metadata (pyproject.toml) ... done
Requirement already satisfied: clr-loader in c:\tools\python39\lib\site-packages (from pythonnet==3.0.0.dev1) (0.1.6)
Requirement already satisfied: cffi>=1.13 in c:\tools\python39\lib\site-packages (from clr-loader->pythonnet==3.0.0.dev1) (1.15.0)
Requirement already satisfied: pycparser in c:\tools\python39\lib\site-packages (from cffi>=1.13->clr-loader->pythonnet==3.0.0.dev1) (2.21)
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\tools\Python39\python.exe' 'C:\tools\Python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' build_wheel 'C:\Users\sigmund\AppData\Local\Temp\tmpzyoo_s0u'
       cwd: C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8
  Complete output (31 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  copying clr.py -> build\lib
  creating build\lib\pythonnet
  copying pythonnet\__init__.py -> build\lib\pythonnet
  creating build\lib\pythonnet\find_libpython
  copying pythonnet\find_libpython\__init__.py -> build\lib\pythonnet\find_libpython
  copying pythonnet\find_libpython\__main__.py -> build\lib\pythonnet\find_libpython
  running build_dotnet
  Running dotnet build...
  dotnet build src/runtime/Python.Runtime.csproj --configuration release --output C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\build\lib\pythonnet/runtime
  Microsoft (R) Build Engine version 16.11.1+3e40a09f8 for .NET
  Copyright (C) Microsoft Corporation. All rights reserved.

    Determining projects to restore...
    Restored C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime\Python.Runtime.csproj (in 238 ms).
  C:\Users\sigmund\.nuget\packages\microsoft.build.tasks.git\1.0.0\build\Microsoft.Build.Tasks.Git.targets(24,5): error : Error reading git repository information: Unsupported repository version 1. Only versions up to 0 are supported. [C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime\Python.Runtime.csproj]
  C:\Users\sigmund\.nuget\packages\microsoft.build.tasks.git\1.0.0\build\Microsoft.Build.Tasks.Git.targets(24,5): warning : Unable to locate repository with working directory that contains directory 'C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime'. [C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime\Python.Runtime.csproj]

  Build FAILED.

  C:\Users\sigmund\.nuget\packages\microsoft.build.tasks.git\1.0.0\build\Microsoft.Build.Tasks.Git.targets(24,5): warning : Unable to locate repository with working directory that contains directory 'C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime'. [C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime\Python.Runtime.csproj]
  C:\Users\sigmund\.nuget\packages\microsoft.build.tasks.git\1.0.0\build\Microsoft.Build.Tasks.Git.targets(24,5): error : Error reading git repository information: Unsupported repository version 1. Only versions up to 0 are supported. [C:\Users\sigmund\AppData\Local\Temp\pip-req-build-w38ycyw8\src\runtime\Python.Runtime.csproj]
      1 Warning(s)
      1 Error(s)

  Time Elapsed 00:00:01.33
  error: command 'C:\\Program Files\\dotnet\\dotnet.exe' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for pythonnet
Failed to build pythonnet
ERROR: Could not build wheels for pythonnet, which is required to install pyproject.toml-based projects
WARNING: You are using pip version 21.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\tools\Python39\python.exe -m pip install --upgrade pip' command.

Output when downgrading pip to version 21.2.4:
[BOOTCAMP] [none] Thu 11-11-21 14:34 5.739 23G z:\>pip list
Package Version
------- -------
pip     21.3
WARNING: You are using pip version 21.3; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\tools\Python39\python.exe -m pip install --upgrade pip' command.

[BOOTCAMP] [none] Thu 11-11-21 14:34 0.989 23G z:\>python -m pip install pip==21.2.4
Collecting pip==21.2.4
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.3
    Uninstalling pip-21.3:
      Successfully uninstalled pip-21.3
Successfully installed pip-21.2.4

[BOOTCAMP] [none] Thu 11-11-21 14:34 6.032 23G z:\>pip install git+https://github.com/pythonnet/pythonnet
Collecting git+https://github.com/pythonnet/pythonnet
  Cloning https://github.com/pythonnet/pythonnet to c:\users\sigmund\appdata\local\temp\pip-req-build-l5x8zpue
  Running command git clone -q https://github.com/pythonnet/pythonnet 'C:\Users\sigmund\AppData\Local\Temp\pip-req-build-l5x8zpue'
  Resolved https://github.com/pythonnet/pythonnet to commit ac336a893de14aaf2c7b795568203b48030f9006
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting clr-loader
  Using cached clr_loader-0.1.6-py3-none-any.whl (45 kB)
Collecting cffi>=1.13
  Using cached cffi-1.15.0-cp39-cp39-win_amd64.whl (180 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (PEP 517) ... done
  Created wheel for pythonnet: filename=pythonnet-3.0.0.dev1-py3-none-any.whl size=248473 sha256=d97da7bd344da51ea577f4d7e2907b9d6d85ace6878742e750e2468a35f85ba1
  Stored in directory: C:\Users\sigmund\AppData\Local\Temp\pip-ephem-wheel-cache-kd0r2mf4\wheels\8d\f2\7a\76977e88048edad6338951680c5f603bce38ea13b92255546a
Successfully built pythonnet
Installing collected packages: pycparser, cffi, clr-loader, pythonnet
Successfully installed cffi-1.15.0 clr-loader-0.1.6 pycparser-2.21 pythonnet-3.0.0.dev1
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\tools\Python39\python.exe -m pip install --upgrade pip' command.

[BOOTCAMP] [none] Thu 11-11-21 14:35 15.091 23G z:\>pip list
Package    Version
---------- ----------
cffi       1.15.0
clr-loader 0.1.6
pip        21.2.4
pycparser  2.21
pythonnet  3.0.0.dev1
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the 'C:\tools\Python39\python.exe -m pip install --upgrade pip' command.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

This should be reported to Python.Net, their build system likely has an issue.

[filmor]

@uranusjr FYI, because I believe this could hit other projects as well: The issue appears because of commit 2d35b80, which changes the temporary git repository to repository format 1, where it was 0 with a straight clone that was used before this commit. This is an undocumented change in behaviour.

[pradyunsg]

This is an undocumented change in behaviour.

This is provably false. From pip's documentation site:

When a revision is specified in a Git URL, use git’s partial clone feature to speed up source retrieval. (#9086)

[filmor]

1. This says "when a revision is specified", which is not true (see the OP's issue)
2. I for one don't follow pip's changelog, and even if I had, I wouldn't have known that this was something that might need testing. If something changes in how the temporary checkout looks like, that might be worth a bit more screen time.
3. This was just an FYI in the literal sense, it could be that you (or package maintainers) get more bug reports like this if they are using tooling for VCS builds that is not tested against the newer repo format

[jamesmyatt]

I have almost the same issue. Seems like our company git server (Bitbucket Server) doesn't behave correctly with the partial clone functionality.

Using pip 21.3.1, when I do:

python -m pip install git+https://<git-repo>/scm/blah/blah.git

I get:

Collecting git+https://<git-repo>/scm/blah/blah.git
  Cloning https://<git-repo>/scm/blah/blah.git to c:\users\blah\appdata\local\temp\1\pip-req-build-3p42l03f
  Running command git clone --filter=blob:none -q 'https://<git-repo>/scm/blah/blah.git' 'c:\users\blah\appdata\local\temp\1\pip-req-build-3p42l03f'
  fatal: bad revision 'b35ae9bb5956155a6426c63ce4b3b5566f4c10ee'
  error: https://<git-repo>/scm/blah/blah.git did not send all necessary objects
...

This works fine with pip 21.2.4, which doesn't use the partial clone.

Similarly, I can reproduce the issue at the command prompt using git clone directly (i.e. works with full clone and doesn't with partial). So it's definitely an issue with the partial clone.

Bitbucket Cloud doesn't support partial clones, but it at least handles this gracefully. e.g.

Collecting git+https://bitbucket.org/atlassian/python-bitbucket.git
  Cloning https://bitbucket.org/atlassian/python-bitbucket.git to c:\users\myaj\appdata\local\temp\1\pip-req-build-2_n0fi0t
  Running command git clone --filter=blob:none -q https://bitbucket.org/atlassian/python-bitbucket.git 'C:\Users\MYAJ\AppData\Local\Temp\1\pip-req-build-2_n0fi0t'
  warning: filtering not recognized by server, ignoring
  Resolved https://bitbucket.org/atlassian/python-bitbucket.git to commit c90076b960f8b83b2f9126adcabfef1db5cb89ba
  ...

Git client version is 2.33.1.windows.1. It's possible that this is a git problem but partial clone works for Github.com. e.g.:

git clone --filter=blob:none "https://github.com/pypa/pip.git"

As @filmor says, the first issue is that it's not respecting the documentation (i.e. only partial clone when a revision is specified). The second is that not all git servers behave correctly for this functionality, users often do not have control over those git servers and the workaround to downgrade pip to <21.3 is far from obvious.

For this second issue, I'm not sure what pip can do except provide an option to disable the partial clone functionality.