-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support compression for Actions logs #31761
Support compression for Actions logs #31761
Conversation
This reverts commit 2d98663.
Why not enable compression by default? Can the log reading code handle a mix of compressed and uncompressed data? If so, I see no reason for not defaulting to it. |
Yes, it works fine with that.
TBH, I also don't see a strong reason for no compression by default. I cannot think of any case where enabling compression would have a negative impact. But I have some concerns:
So my idea is to introduce I opened a issue #31801 in case we forget to do that. |
I almost forget to update https://gitea.com/gitea/docs. So please help review:
|
Follow #31761 --------- Co-authored-by: silverwind <me@silverwind.io>
* giteaofficial/main: Show lock owner instead of repo owner on LFS setting page (go-gitea#31788) Move repository visibility to danger zone in the settings area (go-gitea#31126) [skip ci] Updated translations via Crowdin Add types to various low-level functions (go-gitea#31781) Add warning message in merge instructions when `AutodetectManualMerge` was not enabled (go-gitea#31805) Show latest run when visit /run/latest (go-gitea#31808) Fix typo for `LOG_COMPRESSION` in ini (go-gitea#31809) Add label `docs-update-needed` for PRs that modify `app.example.ini` (go-gitea#31810) Fix `IsObjectExist` with gogit (go-gitea#31790) Support compression for Actions logs (go-gitea#31761) Add issue comment when moving issues from one column to another of the project (go-gitea#29311) [skip ci] Updated translations via Crowdin Fix RPM resource leak (go-gitea#31794)
Support compression for Actions logs to save storage space and bandwidth. Inspired by #24256 (comment)
The biggest challenge is that the compression format should support seekable. So when users are viewing a part of the log lines, Gitea doesn't need to download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too many choices, like bgzip and xz, but I think zstd is the most popular one. It has an implementation in Golang with zstd and zstd-seekable-format-go, and what is better is that it has good compatibility: a seekable format zstd file can be read by a regular zstd reader.
This PR introduces a new package
zstd
to combine and wrap the two packages, to provide a unified and easy-to-use API.And a new setting
LOG_COMPRESSION
is added to the config, although I don't see any reason why not to use compression, I think's it's a good idea to keep the default withnone
to be consistent with old versions.LOG_COMPRESSION
takes effect for only new log files, it adds.zst
as an extension to the file name, so Gitea can determine if it needs decompression according to the file name when reading. Old files will keep the format since it's not worth converting them, as they will be cleared after #31735.