Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve vfsgen to not unzip bindata files but send to browser directly #7109

Merged
merged 8 commits into from
Dec 24, 2020
Merged

Improve vfsgen to not unzip bindata files but send to browser directly #7109

merged 8 commits into from
Dec 24, 2020

Conversation

lunny
Copy link
Member

@lunny lunny commented Jun 3, 2019 •
edited
Loading

Improve the embedded vfsgen, should fix #7107

UPDATE: 2020.12.23

Currently, all bindata files stored on binary with compressed bytes, when serve for http request, Gitea creates a zipReader and unzip them for the web browser. If we enabled gzip, then the file will be compressed again and sent to the browser.

But in fact, it's unnecessary. We can just send the compressed files to browser directly. It will spend less cpu and memory than before.

@codecov-io
Copy link

codecov-io commented Jun 3, 2019 •
edited
Loading

Codecov Report

Merging #7109 (4d27b43) into master (e0c753e) will decrease coverage by 0.01%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7109      +/-   ##
==========================================
- Coverage   42.37%   42.36%   -0.02%     
==========================================
  Files         726      726              
  Lines       77839    77847       +8     
==========================================
- Hits        32982    32976       -6     
- Misses      39447    39460      +13     
- Partials     5410     5411       +1
Impacted Files Coverage Δ
modules/public/dynamic.go 50.00% <0.00%> (-50.00%) ⬇️
modules/public/public.go 54.79% <85.71%> (+4.04%) ⬆️
modules/git/tree_entry_nogogit.go 87.50% <0.00%> (-6.25%) ⬇️
modules/charset/charset.go 68.53% <0.00%> (-4.50%) ⬇️
models/unit.go 46.57% <0.00%> (-2.74%) ⬇️
modules/process/manager.go 72.50% <0.00%> (-2.50%) ⬇️
services/pull/check.go 48.90% <0.00%> (-0.73%) ⬇️
routers/repo/view.go 37.60% <0.00%> (-0.65%) ⬇️
modules/log/file.go 75.20% <0.00%> (+1.60%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0c753e...4d27b43. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 3, 2019
zeripath
zeripath reviewed Jun 3, 2019
View reviewed changes
Copy link
Contributor

@zeripath zeripath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like replication of modules/gzip. How come we can't use that?

Why are we adding another external dependency?

@stale stale bot added the issue/stale label Aug 2, 2019
@lunny lunny added the issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented label Aug 2, 2019
@stale stale bot removed the issue/stale label Aug 2, 2019
@lunny lunny mentioned this pull request Dec 6, 2019
Closed
@lunny lunny added type/enhancement An improvement of existing functionality performance/speed performance issues with slow downs labels Dec 23, 2020
@lunny lunny added this to the 1.14.0 milestone Dec 23, 2020
@go-gitea go-gitea deleted a comment from stale bot Dec 23, 2020
@lunny
Copy link
Member Author

lunny commented Dec 23, 2020

This looks like replication of modules/gzip. How come we can't use that?

Why are we adding another external dependency?

This is different. See my update on PR content.

@lunny lunny changed the title Improve vfsgen Improve vfsgen to not unzip bindata files but send to browser directly Dec 23, 2020
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Dec 23, 2020
@silverwind
Copy link
Member

silverwind commented Dec 23, 2020 •
edited
Loading

What if the client does not support Content-Encoding: gzip? I think this at least needs a check for the Accept-Encoding header like I did here and if the client does not support gzip, send plain.

@silverwind
Copy link
Member

silverwind commented Dec 23, 2020 •
edited
Loading

Also I'd suggest dropping shurcooL/httpgzip because it does not support brotli and implementing Content-Encoding negotiation is like 15 lines of code like I linked. Not worth pulling a dependency in for.

@lunny
Copy link
Member Author

lunny commented Dec 23, 2020 •
edited
Loading

Also I'd suggest dropping shurcooL/httpgzip because it does not support brotli and implementing Content-Encoding negotiation is like 15 lines of code like I linked. Not worth pulling a dependency in for.

Good idea!!! All done except to support br since vfsgen don't support that.

silverwind
silverwind reviewed Dec 23, 2020
View reviewed changes
modules/public/public.go
func parseAcceptEncoding(val string) map[string]bool {
parts := strings.Split(val, ";")
var types = make(map[string]bool)
for _, v := range strings.Split(parts[0], ",") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about whitespace? E.g. Accept-Encoding: gzip, deflate, br? A test would be nice.

@silverwind
Copy link
Member

Maybe you can also copy the docs update regarding ENABLE_GZIP from #12398.

@silverwind silverwind mentioned this pull request Dec 23, 2020
Closed
@lunny
Copy link
Member Author

lunny commented Dec 23, 2020

@silverwind Done.

@silverwind
Copy link
Member

silverwind commented Dec 23, 2020 •
edited
Loading

Does not seem to work for me, HTTP requests are getting stuck unanswered with a bindata build.

image

@lunny
Copy link
Member Author

lunny commented Dec 23, 2020

Fixed. content.Open will still uncompress the file. We have to get compressed file content from *vfsgen۰CompressedFileInfo.GzipBytes.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Dec 23, 2020
@silverwind
Copy link
Member

content.Open will still uncompress the file

That's fine. Most clients will support gzip and will get the fast path without decompression.

@lunny
Copy link
Member Author

lunny commented Dec 24, 2020

make L-G-T-M work.

@lunny lunny merged commit 19ae643 into go-gitea:master Dec 24, 2020
@lunny lunny deleted the lunny/improve_vfsgen branch December 24, 2020 04:25
@go-gitea go-gitea locked and limited conversation to collaborators Feb 11, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@zeripath zeripath zeripath left review comments

@silverwind silverwind silverwind approved these changes

@khmarbaise khmarbaise khmarbaise approved these changes

Assignees
No one assigned
Labels
issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. performance/speed performance issues with slow downs type/enhancement An improvement of existing functionality
Projects
None yet
Milestone
1.14.0
Development

Successfully merging this pull request may close these issues.

Static embed files should not be decompress and compress
6 participants
@lunny @codecov-io @silverwind @khmarbaise @zeripath @GiteaBot