`object not found` when fetching git dependency

[banool]

Problem

When building a crate that depends on other crates via git, sometimes you get an error like this:

Caused by:
  failed to load source for dependency `processor`

Caused by:
  Unable to update https://github.com/aptos-labs/aptos-indexer-processors.git?rev=d44b2d209f57872ac593299c34751a5531b51352#d44b2d20
  
Caused by:
  object not found - no match for id (d44b2d209f57872ac593299c34751a5531b51352); class=Odb (9); code=NotFound (-3)

You can see an instance of this failure here: https://github.com/Homebrew/homebrew-core/pull/165260/files. Which comes from here in case the error fails to show up on the first link: https://github.com/Homebrew/homebrew-core/actions/runs/8180427193/job/22368538918?pr=165260.

This only happens sometimes. It seems to happen more often in CI environments, I'm not sure I've seen it happen locally.

Steps

Repro is challenging because it occurs only sometimes. If you look at any of the past brew version bumps for the aptos formula you'll see that one of the CI steps fails at least once every time due to this issue: https://github.com/Homebrew/homebrew-core/pulls?q=is%3Apr+aptos+is%3Aclosed.

The dep in question comes from here: https://github.com/aptos-labs/aptos-indexer-processors. That repo itself has git submodules, so that could be a factor.

Possible Solution(s)

I'm not sure how to fix this, mostly for now we've just been adding retries to get around it.

Notes

No response

Version

This is the cargo version we're using for the binary in question.

cargo 1.75.0 (1d8b05cdd 2023-11-20)
release: 1.75.0
commit-hash: 1d8b05cdd1287c64467306cf3ca2c8ac60c11eb0
commit-date: 2023-11-20
host: aarch64-apple-darwin
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.4.0 (sys:0.4.68+curl-8.4.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1u  30 May 2023
os: Mac OS 14.3.1 [64-bit]

[weihanglo]

From my understanding, under the same CI environment, a retry seems to work? That's odd. Are they any custom Git config in the CI environment?

[banool]

I've noticed this happening in GitHub Actions, but notably on a variety of different runners, workflows, etc.

You'll see in the original error btw that the commit of the repo it says it can't find does indeed exist: aptos-labs/aptos-indexer-processors@d44b2d2. But it's an orphan, not sure if that is relevant.

[ehuss]

Indeed orphans can be a problem, but usually only when the rev is not specified. That does not appear to be the case here.

One thing to investigate is that it appears https://github.com/aptos-labs/aptos-indexer-processors.git is specified with three different commit revs. I don't recall how cargo handles fetching those separate revs. If it is randomly fetching one of them, and the others aren't "reachable", then that could be a problem. It could also depend on how github's servers decide what to send, since they don't always send a minimal set, and it might change depending on which server is accessed or the phase of the moon.

[banool]

One thing to investigate is that it appears https://github.com/aptos-labs/aptos-indexer-processors.git is specified with three different commit revs

Where can you see that? I can try fix that to reduce the odds of this problem happening.

[weihanglo]

Here are the three commits:

• https://github.com/aptos-labs/aptos-core/tree/aptos-cli-v3.0.1/ecosystem/indexer-grpc/indexer-grpc-parser -> 6fdc6f31fcc494d4ba77ca3bdc8c9eb6b3fc1acb
• https://github.com/aptos-labs/aptos-core/blob/aptos-cli-v3.0.1/crates/aptos/Cargo.toml#L83 -> d44b2d209f57872ac593299c34751a5531b51352
• https://github.com/aptos-labs/aptos-core/blob/aptos-cli-v3.0.1/Cargo.toml#L458 -> 4801acae7aea30d7e96bbfbe5ec5b04056dfa4cf

I haven't got time creating a minimal reproducer with a similar layout though.

---

@ehuss Would it be the case that git gc removed unreachable objects?

[banool]

Oh I see what you mean. Unfortunately this is intentional, we have something of a... complicated versioning scheme at the moment.

[ehuss]

@ehuss Would it be the case that git gc removed unreachable objects?

Not directly, I don't think. The actions linked above seem to have some caching, but not of the cargo directory that I can see (and would likely be too large for the cache anyways). Cargo won't run git gc for a long time.

GitHub runs git gc internally, and that can affect things. I believe they use some heuristics for how frequently it runs. Referring to orphans runs the risk that GitHub will remove them complete (and that is partially why I said it may contribute to the randomness, since from what I've seen GitHub has mirrors that are not consistent with one another in terms of how they compress and gc their pack files).

[banool]

I believe this orphan has been this way for quite a while, so it seems strange that at this point it'd be in this partially available state. I guess some bad state sharding on their side or something?

So I suppose if we depend on a non orphan commit that'll improve our odds.

On the cargo side, does cargo retry in this situation?