Detect charset and convert non UTF-8 files for display

[lafriks]

Fixes #4879

[lunny]

We already have the same function on templates.ToUTF8WithErr(buf)

[lafriks]

Yes but there are two problems with it. First it does return string this way I would have unneeded casting to string and back to byte array and second it does return error but I need to return default byte array in case of error. But I will rename my function and move to other module than

[lafriks]

@lunny fixed

[codecov-io]

Codecov Report

Merging #4950 into master will increase coverage by 0.01%.
The diff coverage is 63.63%.

@@            Coverage Diff            @@
##           master   #4950      +/-   ##
=========================================
+ Coverage   37.39%   37.4%   +0.01%     
=========================================
  Files         306     306              
  Lines       45343   45373      +30     
=========================================
+ Hits        16956   16974      +18     
- Misses      25933   25945      +12     
  Partials     2454    2454

Impacted Files	Coverage Δ
routers/repo/view.go	45.06% <100%> (+0.17%)	⬆️
modules/templates/helper.go	48.95% <35.71%> (-0.58%)	⬇️
modules/base/tool.go	74.52% <81.25%> (-0.62%)	⬇️
modules/process/manager.go	81.15% <0%> (+4.34%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6780661...92d7a69. Read the comment docs.

[jonasfranz]

@lunny Need your review

[lunny]

When the content is too short, the detection will be wrong randomly. I think maybe copy itself before detection.
See

diff --git a/modules/base/tool.go b/modules/base/tool.go
index 2dfd8ffec..19953c319 100644
--- a/modules/base/tool.go
+++ b/modules/base/tool.go
@@ -59,7 +59,17 @@ func DetectEncoding(content []byte) (string, error) {
                return "UTF-8", nil
        }

-       result, err := chardet.NewTextDetector().DetectBest(content)
+       var detectedContent []byte
+       if len(content) < 1024 {
+               times := 1024 / len(content)
+               detectedContent = make([]byte, 0, times*len(content))
+               for i := 0; i < times; i++ {
+                       detectedContent = append(detectedContent, content...)
+               }
+       } else {
+               detectedContent = content
+       }
+       result, err := chardet.NewTextDetector().DetectBest(detectedContent)
        if err != nil {
                return "", err
        }

[lunny]

I also tested change 1024 to 100. It's still right.

[lafriks]

@lunny I think that should be fixed in other PR

[lunny]

@lafriks but this PR will be back port to release/v1.5, I don't think we need two PRs to fix that. This problem should be a chardet library's bug, my code is a temporary method. And it's not easy to update vendor on release/v1.5.

[lafriks]

@lunny added your requested changes

[lunny]

But CI failed.

[techknowlogick]

typo: loose should be lose

[techknowlogick]

CI is failing because of following test:

gitea/modules/base/tool_test.go

Line 93 in 801843b

assert.Error(t, err)

It fails to return an error detecting encoding of []byte{0xfa}

I think this is related to the code that has the if len(content) < 1024 { conditional.

[lafriks]

Fixed tests and comments

[lunny]

Please send back port to v1.5