Gitea Current Goroutines 1500+ and Spiking Slow Server Response Time

[oDinZu]

Description

The Gitea instance jumps around between 60 to 2000+ go routines and was wondering how to limit this? When it jumps to 800, the server load time significantly increases and sometimes the server receives a 502 error.

The server hardware I have Gitea running on has 1CPUs and 2GB of RAM.

Gitea Version

1.23.1 built with GNU Make 4.3, go1.23.4 : bindata, sqlite

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

[W] router: slow GET..
[W] router: failed    GET..
err=runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:785 (0x4776d1)
/source/modules/web/routing/logger_manager.go:116 (0x1fcac88)
/usr/local/go/src/runtime/panic.go:785 (0x4776d1)
/usr/local/go/src/runtime/panic.go:262 (0x479b38)
/usr/local/go/src/runtime/signal_unix.go:917 (0x479b08)

Screenshots

Git Version

2.25.1

Operating System

Ubuntu 22

How are you running Gitea?

Running Gitea with a binary

Database

MySQL

[wxiaoguang]

When it "jump high", what are the requests (the logs could tell)? Do you run the instance publicly and some crawls are heavily accessing various URLs (if so, you can try to limit the public access)?

[oDinZu]

The router slows, then fails and then the gitea binary fails/crashes/restarts. I scheduled a repository sync for a specific time each day.

The gitea server does the GET requests and then eventually has a err=runtime error: invalid memory address or nil pointer dereference.

GET /odinzugiteainstance/gitea/commits/commit/53cf46cae7475befa2dde554bbd9147e436072b9/modules/templates?page=1 for XX.XXX.5.21:0, panic in 59883.8ms @ repo/commit.go:44(repo.RefCommits), err=runtime error: invalid memory address or nil pointer dereference

The server is accessible by the public for cloning/mirroring repositories, but to add repositories and edit, you have to have a username. If what you mean is the gitea instance only accessible to specific set of IP's, then no.

I have fail2ban instance setup and there are no problems in the admin/self-check with the db, including system notices

this error happens first, then the router: slow begins to happen and the go routines begin to stack.

2025/02/12 23:55:29 ...ters/common/serve.go:29:func1() [E] ServeBlob: Close: io: read/write on closed pipe

2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/NvChad/src/commit/f7180602609a17dc76353c99078a886a33729739/lua/core/default_config.lua for 57.141.5.9:0, elapsed 3642.9ms @ repo/view_home.go:314(repo.Home)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/PostgreSQL/src/commit/2b7334d4877ba445003f96b0bb7eed4e7078a39b/src/bin/pg_config/nls.mk for 57.141.5.4:0, elapsed 3203.6ms @ repo/view_home.go:314(repo.Home)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/gitea/blame/commit/9b2536b78fdcd3cf444a2f54857d9871e153858f/services/migrations/gitea_uploader_test.go for 57.141.5.15:0, elapsed 3491.0ms @ repo/blame.go:42(repo.RefBlame)
2025/02/12 23:50:39 ...eb/routing/logger.go:68:func1() [W] router: slow      GET /odinzugiteainstance/liboqs/src/commit/d704da0c5fd0e6d22d2a2f4212910cf383d8c345/scripts/validate_cbom.sh for 57.141.5.18:0, elapsed 3935.8ms @ repo/view_home.go:314(repo.Home)

Maybe related to: #28215

[wxiaoguang]

It seems to be related to the crawls.

• OOM caused by numerous crawls OOM caused by numerous crawls #33520

No idea what kind of crawl is the one you met.

• If it respects "robots.txt", you could set one, or add use a custom template to add "meta": https://stackoverflow.com/questions/67192771/what-is-the-impact-of-a-meta-tag-with-content-noindex-nofollow
• If it doesn't respect "robots.txt", either:
  • block them
  • OR provide more CPU/memory to satisfy them

---

Gitea might also try to optimize some code but I guess the most affects are caused by crawls.

[oDinZu]

That seems to be the case; I will monitor it and update this thread accordingly if more problems arise. At the moment, the gitea instance is running 60 Goroutines.

the solution I made use of is adding custom templates.

I added this line to custom/templates/base/head.tmpl:

<meta name="robots" content="noindex, nofollow, noarchive">

I would recommend this as an official addition to the default head.tmpl

[oDinZu]

Today, I observed the server and the goroutines have dramatically reduced, however, I am unable to migrate data from a git server like GitHub.

I continue to receive similar gitea crash logs and warnings about slow routers from various mirrors.

I temporarily disabled the firewall thinking it may be blocking a server, yet doing so resulted in the same errors.

I haven't changed any nginx settings and have only updated the gitea binary. Yesterday, I was able to migrate a git repository from GH before adding the above fix to my head.tmpl.

I am also having a weird bug about deletion of a repository. I deleted it, yet testing a new migration with the same name results in The repository name is already used.

Update: I had to manually delete the repository from gitea-repositories/username/reponame

Update 2: The repository I was migrating was 3.5GB, so it took much longer than anticipated, however I continue to receive the same errors as previously stated, but everything seems to be speedy and working as intended.