Git for Windows 2.19.0.windows.1 fails to gc some repository clones

[syphr42]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?
  2.19.0.windows.1 64-bit

$ git --version --build-options
git version 2.19.0.windows.1
cpu: x86_64
built from commit: d96bb8bc6c636a8869140e860e72e7bdf64bd790
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
  Windows 10 Enterprise 64-bit

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.14393]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Nano
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: WinSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Rebase: Disabled
Enable Builtin Stash: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

The repository uses Git LFS to store some large files. However, the problem is reproducible with Git LFS removed, so I don't believe it is having an impact here.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash running inside ConEmu

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

The script below reliably reproduces the problem. However, it targets a private repository on GitHub Enterprise that I cannot share. I have not yet figured out a way to create a repository that exhibits the problem. However, I can say this repository has very few commits (less than 100), but thousands of files and some of them are multiple megabytes in size. Both it and any forks of it on GitHub Enterprise exhibit the same problem with version 2.19, but switching back to Git for Windows 2.18 allows the same commands to work. Running the same set of commands in an Ubuntu 18.04 Docker container with Git 2.19.0 installed from the git-core PPA results in success (expected result, no errors). This is reproducible on several laptops running similar software.

#!/bin/bash
set -eo pipefail
shopt -s nullglob

SOURCE_REPO=https://githubenterprise/org/repo.git
MIRROR_REPO=mirror
CLONE_REPO=clone

DIRS=("${MIRROR_REPO}" "${CLONE_REPO}")
echo "Cleaning up prior to test start..."
for dir in "${DIRS[@]}"; do
    if [ -d "${dir}" ]; then
        rm -rf "${dir:-?}"
    fi
done

echo "Running test..."

# Clone the bare mirror
git clone --progress --mirror "${SOURCE_REPO}" "${MIRROR_REPO}"

# Clone the final repository
git clone --progress --branch foo --reference "${MIRROR_REPO}" --dissociate "${SOURCE_REPO}" "${CLONE_REPO}"

echo "Done."

• What did you expect to occur after running these commands?

Cleaning up prior to test start...
Running test...
Cloning into bare repository 'mirror'...
remote: Counting objects: 120746, done.
remote: Total 120746 (delta 0), reused 0 (delta 0), pack-reused 120746
Receiving objects: 100% (120746/120746), 1.96 GiB | 2.30 MiB/s, done.
Resolving deltas: 100% (87355/87355), done.
Cloning into 'clone'...
Enumerating objects: 120746, done.
Counting objects: 100% (120746/120746), done.
Delta compression using up to 6 threads
Compressing objects: 100% (28316/28316), done.
Writing objects: 100% (120746/120746), done.
Total 120746 (delta 87355), reused 120746 (delta 87355)
Checking out files: 100% (14037/14037), done.
Filtering content: 100% (5/5), 396.93 MiB | 6.86 MiB/s, done.
Done.

• What actually happened instead?

Cleaning up prior to test start...
Running test...
Cloning into bare repository 'mirror'...
remote: Counting objects: 120746, done.
remote: Total 120746 (delta 0), reused 0 (delta 0), pack-reused 120746
Receiving objects: 100% (120746/120746), 1.96 GiB | 2.30 MiB/s, done.
Resolving deltas: 100% (87355/87355), done.
Cloning into 'clone'...
Enumerating objects: 120746, done.
fatal: cannot repack to clean up0746)

One time when I ran the test, I got one extra line:

Cleaning up prior to test start...
Running test...
Cloning into bare repository 'mirror'...
remote: Counting objects: 120746, done.
remote: Total 120746 (delta 0), reused 0 (delta 0), pack-reused 120746
Receiving objects: 100% (120746/120746), 1.96 GiB | 2.30 MiB/s, done.
Resolving deltas: 100% (87355/87355), done.
Cloning into 'clone'...
Enumerating objects: 120746, done.
fatal: cannot repack to clean up0746)
Deletion of directory 'clone' failed. Should I try again? (y/n) y

If I re-run the last clone with GIT_TRACE=1:

11:31:47.995570 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
11:31:47.996571 git.c:415               trace: built-in: git clone --progress --branch foo --reference mirror --dissociate https://githubenterprise/org/repo.git clone
Cloning into 'clone'...
11:31:48.049585 run-command.c:637       trace: run_command: git remote-https origin https://githubenterprise/org/repo.git
11:31:48.082577 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:48.087597 git.c:671               trace: exec: git-remote-https origin https://githubenterprise/org/repo.git
11:31:48.087597 run-command.c:637       trace: run_command: git-remote-https origin https://githubenterprise/org/repo.git
11:31:48.123584 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:48.614939 run-command.c:637       trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://githubenterprise/org/repo.git/
11:31:48.649936 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:48.654936 git.c:415               trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://githubenterprise/org/repo.git/
11:31:48.672934 run-command.c:637       trace: run_command: unset GIT_DIR GIT_PREFIX; git --git-dir=C:/Users/user/AppData/Local/Temp/git-test/mirror for-each-ref '--format=%(objectname) %(refname)'
11:31:48.708934 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:48.712934 git.c:415               trace: built-in: git for-each-ref '--format=%(objectname) %(refname)'
11:31:48.742935 run-command.c:637       trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
11:31:48.779934 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:48.786933 git.c:415               trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
11:31:49.047939 run-command.c:637       trace: run_command: git repack -a -d
11:31:49.078933 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:49.085935 git.c:415               trace: built-in: git repack -a -d
11:31:49.088934 run-command.c:637       trace: run_command: git pack-objects --delta-base-offset C:/Users/user/AppData/Local/Temp/git-test/clone/.git/objects/pack/.tmp-18936-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects
11:31:49.127936 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:31:49.134934 git.c:415               trace: built-in: git pack-objects --delta-base-offset C:/Users/user/AppData/Local/Temp/git-test/clone/.git/objects/pack/.tmp-18936-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects
Enumerating objects: 120746, done.
fatal: cannot repack to clean up0746)

Additionally, if I go to the mirror directory and run git gc I get a similar error:

$ git gc
Enumerating objects: 111621, done.
error: failed to run repack06/111621)

With GIT_TRACE=1:

$ GIT_TRACE=1 git gc
11:29:55.457978 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
11:29:55.463975 git.c:415               trace: built-in: git gc
11:29:55.480976 run-command.c:637       trace: run_command: git pack-refs --all --prune
11:29:55.528982 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:29:55.534000 git.c:415               trace: built-in: git pack-refs --all --prune
11:29:55.550978 run-command.c:637       trace: run_command: git reflog expire --all
11:29:55.585976 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:29:55.593009 git.c:415               trace: built-in: git reflog expire --all
11:29:55.608976 run-command.c:637       trace: run_command: git repack -d -l -A --unpack-unreachable=2.weeks.ago
11:29:55.642978 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:29:55.648012 git.c:415               trace: built-in: git repack -d -l -A --unpack-unreachable=2.weeks.ago
11:29:55.650979 run-command.c:637       trace: run_command: GIT_REF_PARANOIA=1 git pack-objects --local --delta-base-offset objects/pack/.tmp-1124-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
11:29:55.683979 exec-cmd.c:236          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
11:29:55.688980 git.c:415               trace: built-in: git pack-objects --local --delta-base-offset objects/pack/.tmp-1124-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
Enumerating objects: 111621, done.
error: failed to run repack06/111621)

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

N/A

[dscho]

We recently addressed a similar problem; could you test the latest snapshot at https://wingit.blob.core.windows.net/files/index.html?

[joergrosenkranz]

We experienced the same issue on a Windows Server 2012 R2. The latest snapshot solves this for us, gc seems to be much slower than in 2.18 thought.

[syphr42]

Using Git-prerelease-2.19.0.windows.1.32.g6c65e18eb1-64-bit resolved the issue.

[dscho]

The latest snapshot solves this for us,

Good.

gc seems to be much slower than in 2.18 thought.

@joergrosenkranz please note that the runtime of gc is really dependent on the shape of the data. No two gc runs seem to perform the same.

However, if you have a consistent reproducer to show that the current snapshot is indeed a lot slower than v2.18.0, please open a new ticket with detailed instructions (and please try to simplify the reproducer as much as possible).

Using Git-prerelease-2.19.0.windows.1.32.g6c65e18eb1-64-bit resolved the issue.

Okay, good. @syphr42 I'll go ahead and close this ticket for you.