iowait and memory consumption are skyrocketing regularly

[schmittlauch]

• Gitea version (or commit ref): 1.10.0
• Git version: 2.16.4
• Operating system: openSUSE Leap 15.1
• Database (use [x]):
  • PostgreSQL
  • MySQL (MariaDB)
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

As written earlier in the Discourse forum, it regularly happens that gitea makes my server's iowait and memory consumption skyrocket, causing issues for all other services running within the same VM. Usually only killing the whole gitea service makes the server recover.
While I couldn't find any direct cause by parsing the logs or the PPROF data, I just discovered in the admin web interface (tab Monitoring) that the GetDiffRange operation seems to be responsible. It seems to get stuck at a single repo of just 138MiB size, clogging IO and making gitea consume more and more memory. iotop shows that gitea creates a lot of I/O, although I'm not sure whether swapping operations are allocated to the process causing them as well.
After this issue first occured, I limited the gitea systemd service to 600MiB and gitea is currently using all of it. But apparently this limit isn't working as the memory+swap usage increased by more than 3GiB during the GetDiffRange operation.

What does this GetDiffRange operation do? I wasn't able to find information on that.

Luckily I was able to allocate an additional 1GiB of memory to my VM, giving me the chance to let the operation run. So far it's running for more than 30 minutes though.

Any ideas about that operation, why it's getting stuck or any additional debug information I could provide?

[schmittlauch]

After ~45min the GetDiffRange operation vanished from the Monitoring web UI, but both memory consumption and IOwait remain equally high. I see myself forced to kill gitea after additional 15 minutes.

[schmittlauch]

Killing the gitea service immediately freed up 4.5GiB of memory.

[lunny]

There is a bug about OpenRepository has been fixed on v1.9.6 or v1.10.0(1?). Could you try v1.10.1?

[schmittlauch]

will do

[zeripath]

I think it's unlikely to be fixed by that - the architecture of GetDiffRange keeps the whole diff in memory. (Probably multiple copies of it too.) If you have a huge diff you're going to break Gitea.

[typeless]

References:
https://github.blog/2016-12-06-how-we-made-diff-pages-3x-faster/
https://github.com/go-gitea/gitea/blob/master/services/pull/merge.go#L504-L545

[lunny]

We can also cache the diff sections to cache server or files.

[schmittlauch]

The issue is still happening regularly even with v1.10.1, but I haven't always seen the "GetDiffRange" operation showing up on the Monitoring page. So GetDiffRange isn't necessarily the cause of this issue.

The gitea logs only show a huge amount of mysql queries and I wasn't able to get anything from the PROF data (I'm not a go developer).
If anyone thinks they can get relevant information from there, I'm willing to pass you the logs and prof data.

[lunny]

So that maybe some cron tasks.

[zeripath]

Hmm excessive Perm checks are a possible cause - see the comments on #8540?

There's another possible issue - testPatch was previously reading the whole patch in to memory. That should stop now.

[schmittlauch]

So that maybe some cron tasks.

Judging from the (non?)periodicty-time the issue appears, is at least **none** of the tasks with a periodicty of 10min or 24h listed in the monitoring tab.

Hmm excessive Perm checks are a possible cause - see the comments on #8540?

I don't have any repos with an excessively large number of owners, which seems to be the precondition for this issue.

There's another possible issue - testPatch was previously reading the whole patch in to memory. That should stop now.

Under which circumstances does `testPatch` get triggered?

[zeripath]

Changes to PRs would cause that

[zeripath]

Do you have repo indexer on? Maybe that could be a cause?

[schmittlauch]

I didn't modify any PRs in the meantime and do not have the indexer enabled (if it's disabled by default). git-lfs is enabled though.
Nevertheless after a while my gitea uses 2GiB of memory:

Server Uptime
9 hours, 34 minutes, 53 seconds
Current Goroutines
29
Current Memory Usage
44MB
Total Memory Allocated
17GB
Memory Obtained
2.8GB
Pointer Lookup Times
0
Memory Allocations
103340464
Memory Frees
102837346
Current Heap Usage
44MB
Heap Memory Obtained
2.7GB
Heap Memory Idle
2.6GB
Heap Memory In Use
52MB
Heap Memory Released
2.6GB
Heap Objects
503118
Bootstrap Stack Usage
928KB
Stack Memory Obtained
928KB
MSpan Structures Usage
678KB
MSpan Structures Obtained
8.2MB
MCache Structures Usage
5.1KB
MCache Structures Obtained
16KB
Profiling Bucket Hash Table Obtained
2.3MB
GC Metadata Obtained
97MB
Other System Allocation Obtained
1.6MB
Next GC Recycle
86MB
Since Last GC Time
19.3s
Total GC Pause
0.2s
Last GC Pause
0.000s
GC Times
382

(taken from the admin dashboard)