Add a config option to block "expensive" pages for anonymous users

[wxiaoguang]

Fix #33966

Document: https://gitea.com/gitea/docs/pulls/191

;; User must sign in to view anything.
;; It could be set to "expensive" to block anonymous users accessing some pages which consume a lot of resources,
;; for example: block anonymous AI crawlers from accessing repo code pages.
;; The "expensive" mode is experimental and subject to change.
;REQUIRE_SIGNIN_VIEW = false

[wxiaoguang]

At the moment, these paths are considered as "expensive":

	expensivePaths := []string{
		// code related
		"/{username}/{reponame}/archive/",
		"/{username}/{reponame}/blame/",
		"/{username}/{reponame}/commit/",
		"/{username}/{reponame}/commits/",
		"/{username}/{reponame}/media/",
		"/{username}/{reponame}/raw/",
		"/{username}/{reponame}/src/",

		// issue & PR related (no trailing slash)
		"/{username}/{reponame}/issues",
		"/{username}/{reponame}/{type:issues}",
		"/{username}/{reponame}/pulls",
		"/{username}/{reponame}/{type:pulls}",

		// wiki
		"/{username}/{reponame}/wiki/",
	}

[hiifong]

At the moment, these paths are considered as "expensive":

	expensivePaths := []string{
		// code related
		"/{username}/{reponame}/archive/",
		"/{username}/{reponame}/blame/",
		"/{username}/{reponame}/commit/",
		"/{username}/{reponame}/commits/",
		"/{username}/{reponame}/media/",
		"/{username}/{reponame}/raw/",
		"/{username}/{reponame}/src/",

		// issue & PR related (no trailing slash)
		"/{username}/{reponame}/issues",
		"/{username}/{reponame}/{type:issues}",
		"/{username}/{reponame}/pulls",
		"/{username}/{reponame}/{type:pulls}",

		// wiki
		"/{username}/{reponame}/wiki/",
	}

I think graph path should also belong to expensivePaths.

It would be better if these paths under activity can be added.

[wxiaoguang]

I think graph path should also belong to expensivePaths.
It would be better if these paths under activity can be added.

done in 4500686

[wxiaoguang]

Since there are too many AI crawlers, try to backport it. The new code is covered by tests and it does nothing if users don't set it.

• Add a config option to block "expensive" pages for anonymous users (#34024) #34071

[richmahn]

Was there any change to the admin config settings page to reflect this? I turned it on the "expensive", but I still get an X. Could we maybe get it to say if the expensive setting took?

[wxiaoguang]

"admin config settings page" is quite out-dated. A lot of config options are not there or incorrectly displayed there.

To confirm, just try to visit a repo's file anonymously.

---

If we'd like to make the "admin config settings page" work correctly, it needs to completely refactor the setting system.