Error 502 after version upgrade

[plaidshirtakos]

• Gitea version (or commit ref): Gitea version 1.9.2
• Git version: git version 2.17.1
• Operating system: Ubuntu 18.04.3
• Database (use [x]):
  • PostgreSQL
  • [x ] MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • [x ] Not relevant
• Log gist:

Description

I tried to upgrade from version 1.8.3 to 1.9.2. I downloaded it(https://dl.gitea.io/gitea/1.9.2/gitea-1.9.2-linux-amd64) and replaced binary in global location:
sudo cp gitea /usr/local/bin/gitea
Gitea is in running status, but got error 502 Bad Gateway nginx/1.14.0 (Ubuntu) when tried to reach it.

[plaidshirtakos]

That is working, when I downgrade back to Gitea version 1.8.3 built with go1.12.5 : bindata, sqlite, sqlite_unlock_notify from the new Gitea version 1.9.2 built with GNU Make 4.1, go1.12.9 : bindata, sqlite, sqlite_unlock_notify.

[lunny]

Any error logs?

[plaidshirtakos]

I can't see any errors in log folder. Where should I see it?

[lunny]

It should be <gitea_home_dir>/logs .

[benzkji]

I have the same problem. The process is still visible, but getting a 502. Fun fact: When restarting, it may work, for some minutes, or even an hour.

last lines from gitea.log:

2019/08/31 18:03:13 .../xorm/session_get.go:99:nocacheGet() [I] [SQL] SELECT "name" FROM "user" WHERE "id"=$1 LIMIT 1 []interface {}{2}
2019/08/31 18:03:13 .../xorm/session_get.go:99:nocacheGet() [I] [SQL] SELECT "name" FROM "user" WHERE "id"=$1 LIMIT 1 []interface {}{2}
2019/08/31 18:03:13 .../xorm/session_get.go:99:nocacheGet() [I] [SQL] SELECT "name" FROM "user" WHERE "id"=$1 LIMIT 1 []interface {}{4}
2019/08/31 18:03:13 routers/init.go:106:GlobalInit() [I] SQLite3 Supported
2019/08/31 18:03:13 routers/init.go:37:checkRunMode() [I] Run Mode: Production
2019/08/31 18:03:13 ...xorm/session_find.go:199:noCacheFind() [I] [SQL] SELECT "id", "type", "status", "conflicted_files", "issue_id", "index", "head_repo_id", "base_repo_id", "head_user_name", "head_branch", "base_branch", "merge_base", "has_merged", "merged_commit_id", "merger_id", "merged_unix" FROM "pull_request" WHERE (status = $1) []interface {}{1}
2019/08/31 18:03:13 routers/init.go:115:GlobalInit() [I] SSH server started on :61005. Cipher list ([aes128-ctr aes192-ctr aes256-ctr aes128-gcm@openssh.com arcfour256 arcfour128]), key exchange algorithms ([diffie-hellman-group1-sha1 diffie-hellman-group14-sha1 ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 curve25519-sha256@libssh.org]), MACs ([hmac-sha2-256-etm@openssh.com hmac-sha2-256 hmac-sha1 hmac-sha1-96])
2019/08/31 18:03:13 ...xorm/session_find.go:199:noCacheFind() [I] [SQL] SELECT "id", "repo_id", "hook_id", "uuid", "type", "url", "signature", "payload_content", "http_method", "content_type", "event_type", "is_ssl", "is_delivered", "delivered", "is_succeed", "request_content", "response_content" FROM "hook_task" WHERE (is_delivered=$1) []interface {}{false}
2019/08/31 18:03:13 cmd/web.go:151:runWeb() [I] Listen: http://127.0.0.1:62070
2019/08/31 18:03:13 cmd/web.go:154:runWeb() [I] LFS server enabled
2019/08/31 18:03:13 ...ce/gracehttp/http.go:142:Serve() [I] Serving 127.0.0.1:62070 with pid 11277

[benzkji]

I cannot tell if this was like this before, but it seems like it's consuming much more resources:

bnzk     31104  3.5  0.2 2076512 186668 ?      Ssl  20:14   0:03  .../sites/gitea-live/gitea web -c .../sites/gitea-live/custom/conf/app.ini

We are two users on it, so very low traffic. Getting a constant 3.5% CPU usage on a huge server. One reason could be the server upgrade to Debian 10. But really no clue how?

@lunny how could we activate a debug log? I have RUN_MODE = dev, but no additional logs?

[plaidshirtakos]

@lunny : I can't find any relevant log from that period of time.

[benzkji]

will dig into the LOG section, here: https://docs.gitea.io/en-us/config-cheat-sheet/

:-)=

[shimunn]

Same here using the image gitea:1.9 with mysql, http is returning 502 and ssh tells me

fatal: Could not read from remote repository.

and no log or error messages are produced

[benzkji]

it's all very vague. I understand that there is no diagnose possible, this way. all I can tell is that resource usage went quite up. are there any unit tests, that check for such scenarios? in my case, it looks like the problem is gone, after I deactivated another systemd service that has gone wild..still very vague.

[sapk]

@benzkji For high resource usage it may be the bleve index rebuilding for code search if enabled. You can try to disable it and restart or just wait that it end.

[shimunn]

Very strange, when I start gitea via docker & systemd like I for the last year it won't surpass starting the ssh server. But if I then enter the container via docker exec and run /app/gitea/gitea the web interface starts working too.

Looks like gitea web just keeps crashing

   20 root      0:00 s6-supervise gitea
 1180 git       0:00 /app/gitea/gitea web
bash-5.0# ps uax | grep git | grep -v grep
   20 root      0:00 s6-supervise gitea
 1199 git       0:00 /app/gitea/gitea web

[sapk]

@shimunn it doesn't seems to be the same issue to help you on your specific case please start another issue and give us more intel on your problem like:

• previous version
• new version
• some logs
• your docker run config (compose file ?)
• your gitea config (app.ini)

Please remove any confidential informations from those details before sharing them.

The exec context is not the same as the startup one and you may not load the same configuration.

[benzkji]

@sapk is the indexing/code search something that was introduced in 1.9? thanks for the explanations.