Git hooks failing in large repositories due to commit-graph-chain.lock exists.

[lemniskett]

Description

We often encounter Git hooks failing in large repository mirrors (>1 GB), this only happens randomly so we can't reproduce it, Gitea has this error logs when this happens:

2025-01-13T07:01:27.521380693Z Stdout: 
Stderr: fatal: Unable to create '/data/git/repositories/redacted/redacted.git/./objects/info/commit-graphs/commit-graph-chain.lock': File exists.
2025-01-13T07:01:27.521401419Z 
2025-01-13T07:01:27.521408941Z Another git process seems to be running in this repository, e.g.
2025-01-13T07:01:27.521415228Z an editor opened by 'git commit'. Please make sure all processes
2025-01-13T07:01:27.521422223Z are terminated then try again. If it still fails, a git process
2025-01-13T07:01:27.521429451Z may have crashed in this repository earlier:
2025-01-13T07:01:27.521435906Z remove the file manually to continue.
2025-01-13T07:01:27.521448634Z Err: exit status 128

There are no git processes running in the docker container. So we deleted the lockfile and then git hooks run normally.

Gitea Version

1.23.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.47.1

Operating System

Ubuntu 24.04

How are you running Gitea?

Docker

Database

PostgreSQL

[rremer]

Running into this myself, but this bug seems like a dupe of #22578

There's also this workaround you could try: #32485
... although I'm not sure I agree with the approach, probably deferred cleanup check for all git commands wrapped in go should be implemented instead. That PR would work, but would be delayed in catching the orphaned lockfiles. You can't bring that delay down too low, or you risk removing a valid lockfile.

Update: I dug into this further, and the reason we see this more often for large repositories is that gc.auto is enabled by default, will kick in during a mirror sync when the repo has more than 6700 loose objects, and will fail mirroring. When the gc completes, subsequent mirror syncs will succeed.

The 'fix' was to disable gc.auto by modifying the gitconfig the gitea user for the pods source with:

#
[gc]
        # disable auto gc, which by default kicks in full repo gc when there are more than 6700 loose objects
        auto = 0

Note that if you have disabled the git_gc_repos cron (you can see via GET /api/v1/admin/cron) then you will no longer have gc enabled, which probably isn't what you want. Depending on your queue settings, having the cron enabled may mean you'll see the commit graph lockfile error if the git_gc_repos and the update_mirrors cron tasks run simoultaneously, but at least you can configure the schedules accordingly.