Repository browsing is very slow after upgrade to 1.14 on windows

[snoopydo]

• Gitea version (or commit ref): 1.14.0
• Git version: 2.31.1
• Operating system: windows 10
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
• Log gist:

Description

Just upgraded my local instance of gitea to 1.14.0 from 1.13.6. Repository browsing has become much slower after upgrading.
Page and Template render times of gone from
1.13.6 Page: 70ms Template: 30ms
1.14.0 Page: 7184ms Template: 4482ms
These times are consistent across page refreshes and service restarts

repository is 85 MiB, approximately 4500 files in 500 folders.

[lunny]

Have you tried twice?

[techknowlogick]

Closing as dupe of #15401

[snoopydo]

@lunny Yes, multiple times, the slow page times are very consistent in the 5-8 seconds range, previously these were a half second or less. I don't have any timings, more of a feeling, but pushing also seems slower

[zeripath]

One quick thought:

What is your PATH in the environment ?

Have you set [git] PATH in the app.ini ?

On Windows often git is added at the end of an extensive PATH variable. Windows does not hash its path and so multiple executions will result in a large slow down.

This was less of a problem prior to switching to git mostly but could be a source of issues.

We also need logs - including router and likely git logs too. Likely some pprof logs too.

It would also be helpful to know if you have switched off the commit cache or not - and some information about your configuration.

But more seriously - it's EXTREMELY unhelpful to not give us logs. Read https://docs.gitea.io/en-us/logging-configuration/#debugging-problems

[lunny]

@techknowlogick This is for repository browsing but that is PR list. I think it's not duplicate issue.

[snoopydo]

I tried playing with my PATH environment variable and also explicitly setting the [git] path setting in the app.ini none of these affected the page times.

I've attached app.ini and the two log files

app-ini.txt
gitea-log.txt
router-log.txt

[lunny]

Could you add this to your app.ini?

[cache]
ENABLED = true

[cache.last_commit]
ENABLED = true

[snoopydo]

Added to app.ini, restarted the service.
No change to either the initial and any subsequent requests, still taking about 7seconds.

[lunny]

Could you enable pprof and give cpu profile file?

[snoopydo]

How do I do that?
I do have go "go version go1.15.5 windows/amd64" installed, however I have never used go.

[snoopydo]

Ok, think I worked it out.
added to app.ini
[server]
ENABLE_PPROF = true
PPROF_DATA_PATH = c:/gitea/log

and ran "go tool pprof http://localhost:6060"
refreshed the repository page a few times,
pprof then said it saved the attached file, hopefully this is what you're after?

pprof.samples.cpu.001.pb.gz

[zeripath]

I've still not got enough information to even begin figuring out what and where the slowdown is. (Perhaps the pprof will be helpful but I'm not certain.)

We're assuming that the issue is going to be git related but the fact that the cache doesn't seem to change that suggests otherwise.

This may well be due to the algorithm change - and you may have been in a sweet spot for the previous one but now in a bad place for this one. Most usecases have found that the new algorithm is much quicker - at least on Linux.

This is why at least some windows users need to try the RCs and try the devs.

Are you able to build Gitea at all?

[zeripath]

Right because I want to rule out git as the causative problem here is a build of 1.14.0 with the gogit backend:

https://eldritchkitty.com/~andrew/gitea-1.14.0-gogit-windows-4.0-amd64.exe

[zeripath]

The longest time here appears to be in getting single commits - not sure which is responsible for this but I suspect commits need to be cached. One other option is to try to use a common git cat-file --batch instead of loading individual git commits.

However there's actually no good reason why the results of git commit can't be cached - if there is a change their ID will change so they're essentially reasonable to be cached.

@snoopydo are you running AntiVirus? Is there something that your server is doing that would make initiation of programs slower than it should be? git cat-file is supposed to be the workhorse of git and supposed to be extremely quick to run.

[zeripath]

https://eldritchkitty.com/~andrew/gitea-15447-git-windows-4.0-amd64.exe

https://eldritchkitty.com/~andrew/gitea-15447-gogit-windows-4.0-amd64.exe

have the current status of #15447 built on the 1.14-branch

[zeripath]

OK another improvement:

https://eldritchkitty.com/~andrew/gitea-15447-15455-git-windows-4.0-amd64.exe

https://eldritchkitty.com/~andrew/gitea-15447-15455-gogit-windows-4.0-amd64.exe

[snoopydo]

Antivirus - yes, built in Windows 10 security. Disabled, no change in page times.

https://eldritchkitty.com/~andrew/gitea-1.14.0-gogit-windows-4.0-amd64.exe
1st request - 150 ms,
2nd request 20 ms,
more requests remain consistent between 20ms and 30 ms

https://eldritchkitty.com/~andrew/gitea-15447-git-windows-4.0-amd64.exe
1st request 4000 ms
2nd request 5000 ms
more requests 5000 ms range

https://eldritchkitty.com/~andrew/gitea-15447-gogit-windows-4.0-amd64.exe
1st request 150 ms
2nd request 25 ms
more requests between 20 ms and 20 ms

https://eldritchkitty.com/~andrew/gitea-15447-15455-git-windows-4.0-amd64.exe
1st request 4000 ms
2nd request 3500 ms
more requests consistent about 3500 ms

https://eldritchkitty.com/~andrew/gitea-15447-15455-gogit-windows-4.0-amd64.exe
1st request 150 ms
2nd request 48 ms
more requests between 20 ms and 40 ms

[zeripath]

Ok well I'm gonna need some pprofs for the git variants if possible.

Could you confirm that your path has the git directories first?

My suspicion is that we'll just have to tolerate that Windows will need go-git and that we won't be able to get it to be as fast with pure git but you have to be aware that go-git has some memory problems.

[GeyserLaPunk]

I just updated my test install from 1.13.7 to 1.14.0 and it does visibly MUCH slower then previous version.
Specifically any repository browsing.

Opening repo from dashboard: Powered by Gitea Version: 1.14.0 Page: 9330ms Template: 1298ms
Going to branches: Powered by Gitea Version: 1.14.0 Page: 10775ms Template: 2ms
Back to Code: Powered by Gitea Version: 1.14.0 Page: 8628ms Template: 1350ms

Windows 10, MsSql., runs as service, 3 small repos, 37 LDAP users.

[snoopydo]

Yes, the "C:\Program Files\Git\cmd" folder is the first entry in my path

for each exe, i ran "go tool pprof http://localhost:6060"

gitea-15447-git-windows-4.0-amd64.exe
pprof.samples.cpu.001.pb.gz

gitea-15447-15455-git-windows-4.0-amd64.exe
pprof.samples.cpu.002.pb.gz

[zeripath]

@GeyserLaPunk:

• Does the gogit build above actually help?
• Does changing your PATH environment variable help?
• Does setting the [git] PATH setting help?
• Are you able to figure out some way to make windows calls to git faster?
• Can you rebuild gitea with logging to help find the slowdown or provide pprofs?

Ultimately the reason for the change to call git directly was go-git/go-git#104 and testing on Linux showed that the cost of calling git was minimal.

Now in Windows that appears not to be the case but I cannot believe that the calls to git cannot be sped up by your local configuration. Certainly however we can reduce the number of calls to git to a minimum.

It's really a shame that none of this was reported despite making this change many months ago and having multiple release candidates. If we had known that Windows was going to be so badly affected we would have kept go-git builds for it and/or I would have worked harder on the performance impacts for it and/or made the change user selectable. However AFAIK this was if anything better than the go git variants.

[MarTiXcz]

Hi,

we are experiencing the same issue after upgrading to 1.14.0. Loading times are Page: 11296ms Template: 6510ms
Gitea starts this process tree:
C:\Program Files\Gitea\gitea.exe
---- C:\Program Files\Git\cmd\git.exe
---------- C:\Program Files\Git\mingw64\bin\git.exe
-------------C:\Windows\System32\conhost.exe
In combination with Jenkins Gitea starts the git process very frequently. About 37000 times per hour.
I have updated the Git for Windows to git version 2.27.0.windows.1

with go git version:
https://eldritchkitty.com/~andrew/gitea-1.14.0-gogit-windows-4.0-amd64.exe
First request about 1000ms.
Every other request:
Powered by Gitea Version: master Page: 147ms Template: 53ms

I might find some time tonight to provide more info with the system git version. For now we will be using the gogit master build.

[GeyserLaPunk]

• Does the gogit build above actually help?

I'm behind corporate firewall so I had to d/l it on my personal pc and upload to my personal server, so I can get it where I need it. :)
Result is yes, gitea-1.14.0-gogit-windows-4.0-amd64.exe does works about 10X faster.

• Does changing your PATH environment variable help?

makes no difference

• Does setting the [git] PATH setting help?

Git is installed in Program Files, and I wasn't unable to figure proper syntax for the value last night.

• Can you rebuild gitea with logging to help find the slowdown or provide pprofs?

sorry, I have very limited privileges on that pc.

It's really a shame that none of this was reported despite making this change many months ago and having multiple release candidates. If we had known that Windows was going to be so badly affected we would have kept go-git builds for it and/or I would have worked harder on the performance impacts for it and/or made the change user selectable. However AFAIK this was if anything better than the go git variants.

I'm just starting with Gitea and went with a stable build.

[zeripath]

@snoopydo thank you for the pprofs - this is revealing another potential speed-up.

However, I think we're never going to be able to get windows to tolerate these calls to git - so therefore we should just provide the gogit builds.

[snoopydo]

@zeripath , wish I could be of more help, but glad that I could be some help beyond saying its broken.

[SlavekB]

The problem seems not specific to Windows. Yesterday I upgraded Gitea from 1.13.7 to 1.14.0 on Linux server (we use your official binary packages) and today we observe a lot of long delays in accessing practically any page. For example:

Běží na Gitea Verze: 1.14.0 Strana: 496457ms Šablona: 496122ms

[zeripath]

@SlavekB Can you replicate on try.gitea.io or point me to an example repository which could allow us to test?

[lafriks]

@SlavekB could you build gitea v1.14.0 from source with gogit tag and check how that performs?

[SlavekB]

@SlavekB could you build gitea v1.14.0 from source with gogit tag and check how that performs?

As the first test I tried build with Go 1.15.9 instead of 1.16.3. Now I leave several hours for observation.

In any case, the problem did not appear continuously. We could use Gitea for several hours without complications. After some time, it became absolutely unusable for a long time. Therefore, I have to watch it for a long time.

[SlavekB]

@SlavekB could you build gitea v1.14.0 from source with gogit tag and check how that performs?

As the first test I tried build with Go 1.15.9 instead of 1.16.3. Now I leave several hours for observation.

In any case, the problem did not appear continuously. We could use Gitea for several hours without complications. After some time, it became absolutely unusable for a long time. Therefore, I have to watch it for a long time.

Ok, confirmed, build with Go 1.15.9 didn't help – again I see long delays in loading. In the process tree, I see the call "git blame" that is in a state Z because the main process gitea web probably does not respond to signals. This requires really great patience:

Běží na Gitea Verze: 1.14.0 Strana: 705186ms Šablona: 705177ms
Běží na Gitea Verze: 1.14.0 Strana: 995562ms Šablona: 995210ms

…I'm going to make build with gogit tag.

[zeripath]

@lafriks meant with the gogit build tag as in:

TAGS="gogit bindata sqlite sqlite_unlock_notify" make

[zeripath]

it would be really helpful for us to see an example of this repository that causes this problem. We will also need logs and potentially a pprof.

[SlavekB]

I meant with the gogit build tag as in:

TAGS="gogit bindata sqlite sqlite_unlock_notify" make

Yes, I understood that it is in mind build tag, not git tag. Build is finished, the instance is running, so I now have to watch for some time if the behavior will be different…

FYI: https://mirror.git.trinitydesktop.org/gitea

[lalo-mx]

Right because I want to rule out git as the causative problem here is a build of 1.14.0 with the gogit backend:

https://eldritchkitty.com/~andrew/gitea-1.14.0-gogit-windows-4.0-amd64.exe

Viewing a Diff Improved from around 10 minutes to 20 seconds in our local instance. Still slower than 1.13.0.

Running on Windows Server 2019 Standard (10.0.17763.1697)

[SlavekB]

it would be really helpful for us to see an example of this repository that causes this problem. We will also need logs and potentially a pprof.

In the case of our instance, this is on any page – listing activities, browse repositories, opening discussion of specific issue, just everything because the main process "does something".

Now I have to watch it again because now, after restarting an instance, it behaves normally.

[zeripath]

@SlavekB That gitea runs fine then breaks means that you are suffering a different issue.

@lalo-mx the viewing a diff issue will need to be discussed on a different issue - with logs and pprof from the gogit build.

[lalo-mx]

@lalo-mx the viewing a diff issue will need to be discussed on a different issue - with logs and pprof from the gogit build.

OK, but the issue looks the same.

While the diff is loading any another page does not load until the diff finishes loading.

[SlavekB]

@lalo-mx the viewing a diff issue will need to be discussed on a different issue - with logs and pprof from the gogit build.

OK, but the issue looks the same.

While the diff is loading any another page does not load until the diff finishes loading.

Yes, I also see that it shows the same symptoms – when I last time watched the behavior during stucked state, there was a git process in the state Z and the loading of any pages did not work.

[zeripath]

@lalo-mx the viewing a diff issue will need to be discussed on a different issue - with logs and pprof from the gogit build.

OK, but the issue looks the same.

While the diff is loading any another page does not load until the diff finishes loading.

No. It does not look the same. You are talking about diff pages which is not the same route, some apparent deadlock, and the problem does not appear to be fixed by using the gogit build.

Please, open another issue and give me some logs (after reading https://docs.gitea.io/en-us/logging-configuration/#debugging-problems) and preferably a cpuprofile pprof running from when the problem starts.

[lalo-mx]

Understood, just a final comment, the problem does appear to be fixed by using the gogit build

[zeripath]

@SlavekB you're likely hitting the same problem as @lalo-mx - it would be helpful to give me some logs and a pprof.

[GeyserLaPunk]

Gitea 1.14.1 has worse latency than 1.14.0 when browsing any repo related pages
@zeripath
I just replaced gitea-1.14.0-gogit-windows-4.0-amd64.exe with gitea-1.14.1-windows-4.0-amd64.exe and latency is back even worse than 1.14.0:

Repo Powered by Gitea Version: 1.14.1 Page: 9120ms Template: 1278ms
Braches Powered by Gitea Version: 1.14.1 Page: 10410ms Template: 2ms
Commits Powered by Gitea Version: 1.14.1 Page: 7730ms Template: 3ms
Commit Powered by Gitea Version: 1.14.1 Page: 23920ms Template: 12161ms
Repo Powered by Gitea Version: 1.14.1 Page: 9835ms Template: 1299ms

sorry for a bad news.

Should I open new issue for 1.14.1?

[snoopydo]

Upgraded the gitea instance on my synology nas and it too shows a slow down in page times when browsing. Pages times gone from <20ms to >100ms

[techknowlogick]

@GeyserLaPunk you switched to the non-gogit version of 1.14.1, could you switch to https://dl.gitea.io/gitea/1.14.1/gitea-1.14.1-gogit-windows-4.0-amd64.exe ?

[GeyserLaPunk]

@GeyserLaPunk you switched to the non-gogit version of 1.14.1, could you switch to https://dl.gitea.io/gitea/1.14.1/gitea-1.14.1-gogit-windows-4.0-amd64.exe ?

@techknowlogick gogit version is better, somewhat a tick slower than 1.13.7 but acceptable.
The biggest unrelated issue is that it was picked up and quarantined by Symantec Anti-virus as matching WS.Reputation.1 signature.

[JPDohle]

After I have updated our server from gitea-1.13.6-windows-4.0-amd64 to gitea-1.14.1-windows-4.0-amd64 browsing repositories on the web interface was quite slow.
But when creating/merging a pull request for a repository with lots of restructured text files and images (Sphinx documentation) the process got stuck when displaying the changes. It was not possible to complete the pull request. Then I tried gitea-1.14.1-gogit-windows-4.0-amd64 and the problems were solved.

@GeyserLaPunk Older versions of the Gitea executable for Windows have been detected several times by Symantec Endpoint Protection on our server. Usually this happened when creating/merging a pull request. Sometimes also when pushing commits. When this happened I created a ZIP archive of the Gitea executable and uploaded it to Symantec to report a false positive. Currently the link is this: https://symsubmit.symantec.com/false_positive

For gitea-1.14.1-gogit-windows-4.0-amd64.exe I did it last week. So I assume that it should not be detected any longer.

[charlesmorin]

We successfully upgraded from 1.12.4 to 1.14.0. OS is Windows 2016 Server Standard. Anti-virus is McAfee.

We immediately started noticing major and sporadic performance issues with this version. We then saw the blog post about 1.14.1, which was referring to significant performance issues on Windows servers. We upgraded to 1.14.1 immediately. However, the problem still persists.

Anyone facing the same challenges?

Gitea version is:
Gitea version 1.14.1 built with GNU Make 4.1, go1.16.3 : bindata, sqlite, sqlite_unlock_notify

[6543]

@charlesmorin you can switch to a windows specific gogit windows version, as described above :)
in the end it's a workaround but it should do the job

[zeripath]

We've provided multiple improvements in 1.15 - plus fixed several bugs in 1.14. I think we can close this. Please a) upgrade to the latest 1.14 b) try the gogit variant on Windows, c) try main/1.15 with and without gogit.