Automatically clean up docker images in the registry without a tag pointing to them

[kolaente]

When pushing new docker images for an existing tag, the old image still exists and uses up storage one the server. While you can use images just by pointing to their sha, I've yet to find someone who actively uses that. For my own registry (portus) I have a cron job to automatically remove everything that does not have a tag pointing to it. Docker even has a command for this.

Having a cleanup job like that would allow to keep old versions but still solve the storage space problem.

@KN4CK3R in #21658 (comment):

No, only if it's "older than" or not included in the "keep pattern". But it should be no problem to add a special logic here because there is already the custom Version == "latest" for containers.

Gitlab has an automatic garbage collection process for this: https://docs.gitlab.com/ee/administration/packages/container_registry.html#removing-untagged-manifests-and-unreferenced-layers

I think it's best to discuss this before implementing, mostly regarding these open questions:

1. Should this be enabled automatically?
2. Should this be a repo/org setting or a global config one?

[KN4CK3R]

Just for clarification, a repo has no impact on packages:

Should this be a repouser/org setting or a global config one?

I checked again how I implemented this and currently there are no untagged images in the container registry! (Exception: If you upload a multiarch image, the different arches are untagged images) If you tag and push an image you can later pull that image with the tag and its hash. If a tag gets pushed again the old tag/version gets removed and that deletes the hash reference too. So after that operation there is no untagged image available anymore.

gitea/routers/api/packages/container/manifest.go

Lines 309 to 312 in f17edfa

	if pv, err = packages_model.GetOrInsertVersion(ctx, _pv); err != nil {
	if err == packages_model.ErrDuplicatePackageVersion {
	if err := packages_service.DeletePackageVersionAndReferences(ctx, pv); err != nil {
	return nil, err

So at the moment the cleanup does not need to remove untagged images because there are none. The question should first be "Should Gitea keep untagged version?"

[silverwind]

Use case sounds pretty similar to git gc which we already automatically run as a cron IIRC.

Should this be enabled automatically?

If it's stable, I'd say so.

Should this be a repo/org setting or a global config one?

I think global is sufficient. Ideally it should just be another cron to cleanup orphaned images, like we already do for orphaned git commits via git gc.

[theodiem]

I've came across this issue after experiencing the same effect.
Building multiarch images when only the manifest is tagged, left me with lots of "packages" behind with only the digest (the manifest had only one copy since it was tagged).

Tagging each arch so it gets overwritten makes the "details" tab a bit impractical when you have too much different arch and versions (for matrix builds).

Should this be a repo/org setting or a global config one?

In my case, I would be happy with the exact same global mechanism described (similar to the cron that runs git gc)

[salasrod]

I am also looking for a similar feature, going out of my way to manually prune images is painful.

[lunny]

Doesn't #21658 resolved the issue?

[kolaente]

@lunny I didn't test it but I don't think so. The PR allows to configure rules for removal of tags, I just want to remove every image layer not associated with a tag.

[peiwenxu]

Is this still happening?

[jum]

No, I have 1.20.4 running and it does not happen. Am 19. September 2023 14:06:27 MESZ schrieb Peiwen Xu ***@***.***>:

…

Is this still happening? -- Reply to this email directly or view it on GitHub: #21673 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[silverwind]

No one has implemented this yet, but it's definitely a vital feature to conserve disk space.

Maybe it should be disabled by default to support pulling image by hash, which is a rare, but valid use case.

[c521wy]

Does anyone tried this cleanup rule?

[kolaente]

Does anyone tried this cleanup rule?

Using that and then checking with the preview yields no results, does not look like its working.