checksum mismatch with index version 4: `invalid data in index - calculated checksum does not match expected; class=Index (10)`

[mfulbright]

Description of the bug

I recently ran
git update-index --index-version 4

On the recommendation of others in an attempt to speed up certain Git operations. Since running that, 'git branchless switch' is panicking:

Message:  A fatal error occurred:
   0: could not read index: invalid data in index - calculated checksum does not match expected; class=Index (10)
   1: invalid data in index - calculated checksum does not match expected; class=Index (10)

Looking at the documentation on 'git update-index' apparently the default index version is ether 2 or 3, so I've tried to reset the index back to what it was via git update-index --index-version 3 etc, but 'switch' is still crashing.

Anyone have any idea what about my index 'git branchless' doesn't like, and how I could resolve it?

Expected behavior

'switch' should not crash

Actual behavior

'switch' crashes

Version of rustc

rustc 1.78.0 (9b00956e5 2024-04-29)

Automated bug report

Software version

git-branchless 0.9.0

Operating system

macOS 14.5 (Darwin 23.5.0)

Command-line

/opt/homebrew/bin/git-branchless bug-report 

Environment variables

SHELL=/opt/homebrew/bin/bash
EDITOR=vim

Git version

> git version 
git version 2.44.0

Hooks

Show 7 hooks

Hook post-applypatch

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook post-applypatch "$@"

## END BRANCHLESS CONFIG

Hook post-checkout

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook post-checkout "$@"

## END BRANCHLESS CONFIG

Hook post-commit

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook post-commit "$@"

## END BRANCHLESS CONFIG

Hook post-merge

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook post-merge "$@"

## END BRANCHLESS CONFIG

Hook post-rewrite

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook post-rewrite "$@"

## END BRANCHLESS CONFIG

Hook pre-auto-gc

#!/bin/sh
## START BRANCHLESS CONFIG

git branchless hook pre-auto-gc "$@"

## END BRANCHLESS CONFIG

Hook reference-transaction

#!/bin/sh
## START BRANCHLESS CONFIG

# Avoid canceling the reference transaction in the case that `branchless` fails
# for whatever reason.
git branchless hook reference-transaction "$@" || (
echo 'branchless: Failed to process reference transaction!'
echo 'branchless: Some events (e.g. branch updates) may have been lost.'
echo 'branchless: This is a bug. Please report it.'
)

## END BRANCHLESS CONFIG

Events

Show 5 events

Event ID: 5562, transaction ID: 18545 (message: checkout)

1. RefUpdateEvent { timestamp: 1720025943.786314, event_tx_id: Id(18545), ref_name: ReferenceName("HEAD"), old_oid: 95c043fc89103c9fb962768ccf7f0a6fc703ea60, new_oid: 95c043fc89103c9fb962768ccf7f0a6fc703ea60, message: None }

:
O 281c0f4 18d xxxx x xxxxxx xxxxxx xxxxxxxxx xxxxx xxxxxxxxx
|\
: o a12f23f 15d (redacted-ref-0) xxxxxxx xxxxxx xxxx xxxxx
:
O 8a45e51 6d xxxxx xxx xxxx xxxxxx xx xxxxxx xxxxx xxxxxxxxx
|\
| o 4d8d39d 6d (redacted-ref-1) xxxxxxx xxxxxx xxxxxxxxxxx xxxx xxx xxxx xxxxxxx
|\
| o 2d888f5 6d (redacted-ref-2) xxxxxxx xxxxxx xxxxxxxxxxx xxxxxx xxxx xxx xxxxxxx
|\
: o e4e506c 3d (redacted-ref-3) xxxx xxxxxxx xxx xxxxxxxxxxx
:
@ 95c043f 1d (> master, redacted-ref-4) xxxxxxxxxxxxxxx xxxxxx xxxxxx xxxx xxxxxx xxxxxxxxx
|\
| o 7dad01c 1d (redacted-ref-5) xxxxxxx xxxx xxxxxxx xxxxxxxxxxx xx xxx xxx xxxx xxxxx xxxxxxx
| |
| o 133c553 1d (redacted-ref-6, redacted-ref-7) xxxxxx xxxxxxxxxx xxx xxxx xxxxxxxxxxx xxx xxxxxxx xx xxxxxxxxxxxxxxxx xxx xxxx
|\
| o fdd1b2d 1d (redacted-ref-8, redacted-ref-9) xxxxxxx xxxx xxx xxxxxxx xxx xxxxxxxxxx xxxx xxxxxxxx xxxx
|\
| o 9c5a14b 1d (redacted-ref-10) xxxxxxx xxxx xxxxxxxxxxx xxxx xxx xxxxxx xxxxx
| |
| o e8bf12e 1d (redacted-ref-11) xxx xxxx
| |
| o 57a56ea 1d (redacted-ref-12, redacted-ref-13) xxxxxxx xxxxxxxx xxxx xxxxxx xxx xxxxxx
|
x 2238a72 1d (manually hidden) xxxxxxx xxxxxxx xxxxxxxx xxxxxxx xx xxxxxxxxxxxxxx
|
x 829d0d2d 1d (manually hidden) xxxxxxxxxxxx xxxxxxx xx xxx
|
x 419f36e 1d (manually hidden) xxxxxxxxxxxxxx xxxxxxx xxxxxxxx xxx xxxxxxxxxxxxx xxxx
|
o 52edaef 1d xxxxxx xxxxxxxxx xx xxxxxxxxx xxxxxx xxxxxxxxxxxxxxxx xx xx xxx xxxxx

Event ID: 5555, transaction ID: 18498 (message: branchless)

1. ObsoleteEvent { timestamp: 1720025582.30605, event_tx_id: Id(18498), commit_oid: NonZeroOid(afe4864333abfbc94bff0d70c4e986e3244221cd) }
2. ObsoleteEvent { timestamp: 1720025582.30605, event_tx_id: Id(18498), commit_oid: NonZeroOid(e282ab4f49927aa0175d296637d4d2c14337d743) }
3. ObsoleteEvent { timestamp: 1720025582.30605, event_tx_id: Id(18498), commit_oid: NonZeroOid(8c2149aa6e610413b873766f8e69fabcfb5cf0cf) }
4. RefUpdateEvent { timestamp: 1720025583.157356, event_tx_id: Id(18498), ref_name: ReferenceName("refs/heads/redacted-ref-14"), old_oid: afe4864333abfbc94bff0d70c4e986e3244221cd, new_oid: 0000000000000000000000000000000000000000, message: None }
5. RefUpdateEvent { timestamp: 1720025583.157356, event_tx_id: Id(18498), ref_name: ReferenceName("refs/heads/redacted-ref-15"), old_oid: 8c2149aa6e610413b873766f8e69fabcfb5cf0cf, new_oid: 0000000000000000000000000000000000000000, message: None }
6. RefUpdateEvent { timestamp: 1720025583.157356, event_tx_id: Id(18498), ref_name: ReferenceName("refs/heads/redacted-ref-16"), old_oid: 8c2149aa6e610413b873766f8e69fabcfb5cf0cf, new_oid: 0000000000000000000000000000000000000000, message: None }
7. RefUpdateEvent { timestamp: 1720025583.157356, event_tx_id: Id(18498), ref_name: ReferenceName("refs/heads/redacted-ref-17"), old_oid: e282ab4f49927aa0175d296637d4d2c14337d743, new_oid: 0000000000000000000000000000000000000000, message: None }

:
O 281c0f4 18d xxxx x xxxxxx xxxxxx xxxxxxxxx xxxxx xxxxxxxxx
|\
: o a12f23f 15d (redacted-ref-0) xxxxxxx xxxxxx xxxx xxxxx
:
O 8a45e51 6d xxxxx xxx xxxx xxxxxx xx xxxxxx xxxxx xxxxxxxxx
|\
| o 4d8d39d 6d (redacted-ref-1) xxxxxxx xxxxxx xxxxxxxxxxx xxxx xxx xxxx xxxxxxx
|\
| o 2d888f5 6d (redacted-ref-2) xxxxxxx xxxxxx xxxxxxxxxxx xxxxxx xxxx xxx xxxxxxx
|\
: o e4e506c 3d (redacted-ref-3) xxxx xxxxxxx xxx xxxxxxxxxxx
:
@ 95c043f 1d (> master, redacted-ref-4) xxxxxxxxxxxxxxx xxxxxx xxxxxx xxxx xxxxxx xxxxxxxxx
|\
| o 7dad01c 1d (redacted-ref-5) xxxxxxx xxxx xxxxxxx xxxxxxxxxxx xx xxx xxx xxxx xxxxx xxxxxxx
| |
| o 133c553 1d (redacted-ref-6, redacted-ref-7) xxxxxx xxxxxxxxxx xxx xxxx xxxxxxxxxxx xxx xxxxxxx xx xxxxxxxxxxxxxxxx xxx xxxx
|\
| o fdd1b2d 1d (redacted-ref-8, redacted-ref-9) xxxxxxx xxxx xxx xxxxxxx xxx xxxxxxxxxx xxxx xxxxxxxx xxxx
|\
| o 9c5a14b 1d (redacted-ref-10) xxxxxxx xxxx xxxxxxxxxxx xxxx xxx xxxxxx xxxxx
| |
| o e8bf12e 1d (redacted-ref-11) xxx xxxx
| |
| o 57a56ea 1d (redacted-ref-12, redacted-ref-13) xxxxxxx xxxxxxxx xxxx xxxxxx xxx xxxxxx
|
x 2238a72 1d (manually hidden) xxxxxxx xxxxxxx xxxxxxxx xxxxxxx xx xxxxxxxxxxxxxx
|
x 829d0d2d 1d (manually hidden) xxxxxxxxxxxx xxxxxxx xx xxx
|
x 419f36e 1d (manually hidden) xxxxxxxxxxxxxx xxxxxxx xxxxxxxx xxx xxxxxxxxxxxxx xxxx
|
o 52edaef 1d xxxxxx xxxxxxxxx xx xxxxxxxxx xxxxxx xxxxxxxxxxxxxxxx xx xx xxx xxxxx

Event ID: 5544, transaction ID: 18479 (message: branchless)

1. ObsoleteEvent { timestamp: 1720025496.389633, event_tx_id: Id(18479), commit_oid: NonZeroOid(2238a7277476657ac2ca605bdfdf8ee8c070528c) }
2. ObsoleteEvent { timestamp: 1720025496.389633, event_tx_id: Id(18479), commit_oid: NonZeroOid(829d0d2d234095c82322e82eccb8d88c25a12f79) }
3. ObsoleteEvent { timestamp: 1720025496.389633, event_tx_id: Id(18479), commit_oid: NonZeroOid(419f36e8a95e0c1c76f568df9ca922657168b47c) }
4. ObsoleteEvent { timestamp: 1720025496.389633, event_tx_id: Id(18479), commit_oid: NonZeroOid(46d75830e79ee27345356035b04123491eec8050) }
5. ObsoleteEvent { timestamp: 1720025496.389633, event_tx_id: Id(18479), commit_oid: NonZeroOid(317415e97c4a65f9ae94e20d6fbc2dff0757506b) }
6. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-18"), old_oid: 317415e97c4a65f9ae94e20d6fbc2dff0757506b, new_oid: 0000000000000000000000000000000000000000, message: None }
7. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-19"), old_oid: 317415e97c4a65f9ae94e20d6fbc2dff0757506b, new_oid: 0000000000000000000000000000000000000000, message: None }
8. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-20"), old_oid: 2238a7277476657ac2ca605bdfdf8ee8c070528c, new_oid: 0000000000000000000000000000000000000000, message: None }
9. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-21"), old_oid: 419f36e8a95e0c1c76f568df9ca922657168b47c, new_oid: 0000000000000000000000000000000000000000, message: None }
10. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-22"), old_oid: 829d0d2d234095c82322e82eccb8d88c25a12f79, new_oid: 0000000000000000000000000000000000000000, message: None }
11. RefUpdateEvent { timestamp: 1720025497.002267, event_tx_id: Id(18479), ref_name: ReferenceName("refs/heads/redacted-ref-23"), old_oid: 46d75830e79ee27345356035b04123491eec8050, new_oid: 0000000000000000000000000000000000000000, message: None }

:
O 281c0f4 18d xxxx x xxxxxx xxxxxx xxxxxxxxx xxxxx xxxxxxxxx
|\
: o a12f23f 15d (redacted-ref-0) xxxxxxx xxxxxx xxxx xxxxx
:
O 8a45e51 6d xxxxx xxx xxxx xxxxxx xx xxxxxx xxxxx xxxxxxxxx
|\
| o 4d8d39d 6d (redacted-ref-1) xxxxxxx xxxxxx xxxxxxxxxxx xxxx xxx xxxx xxxxxxx
|\
| o 2d888f5 6d (redacted-ref-2) xxxxxxx xxxxxx xxxxxxxxxxx xxxxxx xxxx xxx xxxxxxx
|\
: o e4e506c 3d (redacted-ref-3) xxxx xxxxxxx xxx xxxxxxxxxxx
:
@ 95c043f 1d (> master, redacted-ref-4) xxxxxxxxxxxxxxx xxxxxx xxxxxx xxxx xxxxxx xxxxxxxxx
|\
| o 7dad01c 1d (redacted-ref-5) xxxxxxx xxxx xxxxxxx xxxxxxxxxxx xx xxx xxx xxxx xxxxx xxxxxxx
| |
| o 133c553 1d (redacted-ref-6, redacted-ref-7) xxxxxx xxxxxxxxxx xxx xxxx xxxxxxxxxxx xxx xxxxxxx xx xxxxxxxxxxxxxxxx xxx xxxx
|\
| o fdd1b2d 1d (redacted-ref-8, redacted-ref-9) xxxxxxx xxxx xxx xxxxxxx xxx xxxxxxxxxx xxxx xxxxxxxx xxxx
|\
| o 9c5a14b 1d (redacted-ref-10) xxxxxxx xxxx xxxxxxxxxxx xxxx xxx xxxxxx xxxxx
| |
| o e8bf12e 1d (redacted-ref-11) xxx xxxx
| |
| o 57a56ea 1d (redacted-ref-12, redacted-ref-13) xxxxxxx xxxxxxxx xxxx xxxxxx xxx xxxxxx
|
x 2238a72 1d (manually hidden) xxxxxxx xxxxxxx xxxxxxxx xxxxxxx xx xxxxxxxxxxxxxx
|
x 829d0d2d 1d (manually hidden) xxxxxxxxxxxx xxxxxxx xx xxx
|
x 419f36e 1d (manually hidden) xxxxxxxxxxxxxx xxxxxxx xxxxxxxx xxx xxxxxxxxxxxxx xxxx
|
o 52edaef 1d xxxxxx xxxxxxxxx xx xxxxxxxxx xxxxxx xxxxxxxxxxxxxxxx xx xx xxx xxxxx

Event ID: 5543, transaction ID: 18401 (message: hook-post-checkout)

1. RefUpdateEvent { timestamp: 1719946653.943282, event_tx_id: Id(18401), ref_name: ReferenceName("HEAD"), old_oid: 95c043fc89103c9fb962768ccf7f0a6fc703ea60, new_oid: 95c043fc89103c9fb962768ccf7f0a6fc703ea60, message: None }

:
O 281c0f4 18d xxxx x xxxxxx xxxxxx xxxxxxxxx xxxxx xxxxxxxxx
|\
: o a12f23f 15d (redacted-ref-0) xxxxxxx xxxxxx xxxx xxxxx
:
O 8a45e51 6d xxxxx xxx xxxx xxxxxx xx xxxxxx xxxxx xxxxxxxxx
|\
| o 4d8d39d 6d (redacted-ref-1) xxxxxxx xxxxxx xxxxxxxxxxx xxxx xxx xxxx xxxxxxx
|\
| o 2d888f5 6d (redacted-ref-2) xxxxxxx xxxxxx xxxxxxxxxxx xxxxxx xxxx xxx xxxxxxx
|\
: o e4e506c 3d (redacted-ref-3) xxxx xxxxxxx xxx xxxxxxxxxxx
:
@ 95c043f 1d (> master, redacted-ref-4) xxxxxxxxxxxxxxx xxxxxx xxxxxx xxxx xxxxxx xxxxxxxxx
|\
| o 7dad01c 1d (redacted-ref-5) xxxxxxx xxxx xxxxxxx xxxxxxxxxxx xx xxx xxx xxxx xxxxx xxxxxxx
| |
| o 133c553 1d (redacted-ref-6, redacted-ref-7) xxxxxx xxxxxxxxxx xxx xxxx xxxxxxxxxxx xxx xxxxxxx xx xxxxxxxxxxxxxxxx xxx xxxx
|\
| o fdd1b2d 1d (redacted-ref-8, redacted-ref-9) xxxxxxx xxxx xxx xxxxxxx xxx xxxxxxxxxx xxxx xxxxxxxx xxxx
|\
| o 9c5a14b 1d (redacted-ref-10) xxxxxxx xxxx xxxxxxxxxxx xxxx xxx xxxxxx xxxxx
| |
| o e8bf12e 1d (redacted-ref-11) xxx xxxx
| |
| o 57a56ea 1d (redacted-ref-12, redacted-ref-13) xxxxxxx xxxxxxxx xxxx xxxxxx xxx xxxxxx
|
x 2238a72 1d xxxxxxx xxxxxxx xxxxxxxx xxxxxxx xx xxxxxxxxxxxxxx
|
x 829d0d2d 1d xxxxxxxxxxxx xxxxxxx xx xxx
|
x 419f36e 1d xxxxxxxxxxxxxx xxxxxxx xxxxxxxx xxx xxxxxxxxxxxxx xxxx
|
o 52edaef 1d xxxxxx xxxxxxxxx xx xxxxxxxxx xxxxxx xxxxxxxxxxxxxxxx xx xx xxx xxxxx

Event ID: 5542, transaction ID: 18398 (message: reference-transaction)

1. RefUpdateEvent { timestamp: 1719946653.805619, event_tx_id: Id(18398), ref_name: ReferenceName("refs/heads/redacted-ref-4"), old_oid: 0000000000000000000000000000000000000000, new_oid: 95c043fc89103c9fb962768ccf7f0a6fc703ea60, message: None }

:
O 281c0f4 18d xxxx x xxxxxx xxxxxx xxxxxxxxx xxxxx xxxxxxxxx
|\
: o a12f23f 15d (redacted-ref-0) xxxxxxx xxxxxx xxxx xxxxx
:
O 8a45e51 6d xxxxx xxx xxxx xxxxxx xx xxxxxx xxxxx xxxxxxxxx
|\
| o 4d8d39d 6d (redacted-ref-1) xxxxxxx xxxxxx xxxxxxxxxxx xxxx xxx xxxx xxxxxxx
|\
| o 2d888f5 6d (redacted-ref-2) xxxxxxx xxxxxx xxxxxxxxxxx xxxxxx xxxx xxx xxxxxxx
|\
: o e4e506c 3d (redacted-ref-3) xxxx xxxxxxx xxx xxxxxxxxxxx
:
@ 95c043f 1d (> master, redacted-ref-4) xxxxxxxxxxxxxxx xxxxxx xxxxxx xxxx xxxxxx xxxxxxxxx
|\
| o 7dad01c 1d (redacted-ref-5) xxxxxxx xxxx xxxxxxx xxxxxxxxxxx xx xxx xxx xxxx xxxxx xxxxxxx
| |
| o 133c553 1d (redacted-ref-6, redacted-ref-7) xxxxxx xxxxxxxxxx xxx xxxx xxxxxxxxxxx xxx xxxxxxx xx xxxxxxxxxxxxxxxx xxx xxxx
|\
| o fdd1b2d 1d (redacted-ref-8, redacted-ref-9) xxxxxxx xxxx xxx xxxxxxx xxx xxxxxxxxxx xxxx xxxxxxxx xxxx
|\
| o 9c5a14b 1d (redacted-ref-10) xxxxxxx xxxx xxxxxxxxxxx xxxx xxx xxxxxx xxxxx
| |
| o e8bf12e 1d (redacted-ref-11) xxx xxxx
| |
| o 57a56ea 1d (redacted-ref-12, redacted-ref-13) xxxxxxx xxxxxxxx xxxx xxxxxx xxx xxxxxx
|
x 2238a72 1d xxxxxxx xxxxxxx xxxxxxxx xxxxxxx xx xxxxxxxxxxxxxx
|
x 829d0d2d 1d xxxxxxxxxxxx xxxxxxx xx xxx
|
x 419f36e 1d xxxxxxxxxxxxxx xxxxxxx xxxxxxxx xxx xxxxxxxxxxxxx xxxx
|
o 52edaef 1d xxxxxx xxxxxxxxx xx xxxxxxxxx xxxxxx xxxxxxxxxxxxxxxx xx xx xxx xxxxx

Version of git-branchless

git-branchless-opts 0.9.0

Version of git

git version 2.44.0

[mfulbright]

So after further investigation, it might not have been the update-index command that was breaking things. I had also recently set feature.manyFiles = true in my git config. According to this related bug, doing so enables the index.skipHash config option, and that can cause the index errors I was seeing.

#894

I was able to get things working again with the workaround stated in that bug, namely disable/unset the manyFiles config option, and then 'git checkout' some revision so that the index gets rebuilt.

I'll leave this bug open, since it still seems like a bug that these options don't work with git branchless, even though I was able to get things working again. But I understand if the maintainers want to close this bug or merge it with the other bug, etc.

P.S. - For anyone reading this, note that it's possible that the 'update-index' command also broke things. I can't be sure it didn't.

P.P.S. - In my attempts to resolve things, I also uninstalled git branchless (including deleting the .git/branchless directory in my git repo) and then reinstalled it. It's vaguely possible that was part of getting things working again too.

[arxanas]

We rely on libgit2 to handle all index operations, so basically the only thing we can do is hope that libgit2 gains support for it, and then upgrade our dependency version.

There might be something we can do to surface better error messages for a class of index errors, to suggest that users undo whatever index configuration changes they might have just made 😅.

[injust]

Now that libgit2 and the rust bindings have been updated to support index.skipHash, I think this issue is effectively a duplicate of #894?