Add support for fetching unadvertised SHA-1 (allow-tip-sha1-in-want capability)

[JustAnotherArchivist]

I'd like to see the allow-tip-sha1-in-want capability added to Gitea's Git server so that objects that aren't advertised in upload-pack can be fetched by a client. The primary use case that I'm aware of is fetching commits that were on branches that have been force-pushed over. In the context of pull requests, these previous commits can be important to understand the review process, for example. GitHub and GitLab.com (haven't tested GitLab CE) support this.

For those who may not be that familiar with how a git clone works, here's a quick summary: the client connects to the server and executes git upload-pack. This returns the server's capabilities and a list of refs. The client then determines which of these refs they want and sends a series of want <SHA-1> lines to the server of the objects (and their history) it wants. The server then returns a packfile with those objects, and the client creates the desired local directory structure.

However, even if the client fetches everything advertised by the server (e.g. with --mirror), this only returns reachable commits. An example of unreachable commits are branches that were force-pushed over. Most notably, that is common in pull requests when major refactors of the PR code are necessary to keep the history clean after the merge. As long as garbage collection hasn't been run on the server, those commits still exist, and they can be fetched if the server supports the allow-tip-sha1-in-want: git fetch remote sha1 will attempt to fetch the commit with hash sha1 from the remote remote.

Example using GitHub and the recent PR #11032:

1. git clone --mirror https://github.com/go-gitea/gitea.git
2. cd gitea.git
3. git show e4c89ba252341b8222dd72389b7407c3a0e05c99 → fatal: bad object e4c89ba252341b8222dd72389b7407c3a0e05c99
4. git fetch origin e4c89ba252341b8222dd72389b7407c3a0e05c99
5. git show e4c89ba252341b8222dd72389b7407c3a0e05c99 → commit e4c89ba252341b8222dd72389b7407c3a0e05c99 etc.

As of version 1.11.6 and the version deployed on gitea.com (see below), Gitea does not support step 4 (error: Server does not allow request for unadvertised object <SHA-1>), which amounts to a want e4c89ba252341b8222dd72389b7407c3a0e05c99 request from the client where that commit is not reachable from any ref.

Example using https://gitea.com/gitea/homebrew-gitea/pulls/94:

1. git clone --mirror https://gitea.com/gitea/homebrew-gitea.git
2. cd homebrew-gitea.git
3. git show 238dfca61957cd2b8b7c690a1ccea6481104c5b3 → bad object (as expected)
4. git fetch origin 238dfca61957cd2b8b7c690a1ccea6481104c5b3 → error: Server does not allow request for unadvertised object 238dfca61957cd2b8b7c690a1ccea6481104c5b3

Since Gitea essentially just delegates the HTTP requests handling to Git (after auth checks etc.) in routers/repo/http.go, I believe the only change that would be needed to support this would be enabling uploadpack.allowTipSHA1InWant in the repositories.

On a related note, supporting allow-reachable-sha1-in-want aka uploadpack.allowReachableSHA1InWant may also be useful.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

[lunny]

This needs git v2.5.0

[zeripath]

This is simply an option in .gitconfig or git config files in the repositories. Git capabilities are passed through by Gitea unaltered.

If you set uploadpack.allowTipSHA1InWant, uploadpack.hideRefs, uploadarchive.allowUnreachable etc. appropriately in git you can do this.

[lyz-code]

I'm seeing this error too in some gitea actions pipelines when the CI tries to run actions/checkout@v3 on a merge commit on the main branch after a pull request merge.

It's interesting because some pipelines work and others don't, it looks there is a race condition.

Any idea on how to solve this?

[wxiaoguang]

You can set your own git config options by : Support changing git config through app.ini #24860

https://docs.gitea.com/administration/config-cheat-sheet#git---config-options-gitconfig

https://github.com/go-gitea/gitea/blob/ec1feedbf582b05b6a5e8c59fb2457f25d053ba2/custom/conf/app.example.ini#L712C6-L712C6

[wxiaoguang]

I'm seeing this error too in some gitea actions pipelines when the CI tries to run actions/checkout@v3 on a merge commit on the main branch after a pull request merge.

It's interesting because some pipelines work and others don't, it looks there is a race condition.

Any idea on how to solve this?

If I understand correctly, the checkout action shouldn't depend on this "unadvertised SHA-1" feature.

What's the difference of these PRs? Maybe branch of the merged PR has been deleted so the CI are not able to checkout.

For this issue, I think it can be closed.

[JustAnotherArchivist]

I'm not sure whether the CI situation is related. But the original issue, i.e. enabling allowTipSHA1InWant (and possibly the other two options mentioned above) by default to allow fetching unadvertised commits, just like GitHub and GitLab do, is not resolved. I understand that it can be done by changing the configuration, but it should be on by default. Please reopen.