Migration of repositories with tags to organisation fails

[oaxiento]

• Gitea version (or commit ref): 1.9.4
• Git version: 1.8.3.1
• Operating system: CentOS 7
• Database (use [x]):
  • PostgreSQL
  • MariaDB
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

We are currently migrating all of our 230 repositories to Gitea. Here's how we do it:

1. git clone --mirror https://old-git.example.org/repo.git
2. git push --mirror https://gitea.example.org/ORG/repo.git

To keep it simple our Gitea instance has just one organisation with around 110 "Owner" team members. When we migrate repositories with more than 50 tags to that organisation, the git client pushes everything without an error, but Gitea fails when creating releases for every tag with the following message:

2019/10/16 11:12:42 models/webhook.go:909:DeliverHooks() [E] Get repository [235] hook tasks: dial tcp 172.xxx.xxx.xxx:3306: connect: cannot assign requested address

In the Gitea logfile I can see that Gitea calls repo_permission.go for every tag and every user of that organisation, which results in a lot of calls and presumably database queries:

2019/10/16 11:12:39 models/update.go:80:PushUpdate() [T] TriggerTask 'TAGNAME' by gituser
2019/10/16 11:12:39 models/pull.go:1088:AddTestPullRequestTask() [T] AddTestPullRequestTask [head_repo_id: xxx, head_branch: xxx]: finding pull requests
2019/10/16 11:12:39 modules/auth/auth.go:97:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2019/10/16 11:12:39 ...s/context/context.go:328:func1() [D] Session ID: xxx
2019/10/16 11:12:39 ...s/context/context.go:329:func1() [D] CSRF Token: xxx
2019/10/16 11:12:39 models/pull.go:1128:AddTestPullRequestTask() [T] AddTestPullRequestTask [base_repo_id: xxx, base_branch: xxx]: finding pull requests
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824725727744:USER1 in 824725727824:ORG/reponame
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824729407712:USER2 in 824729407808:ORG/reponame
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824729900800:USER3 in 824729900880:ORG/reponame
[...]

When migrating that same repository to a "standalone" user account, everything works as expected. All releases get created. In the Gitea logfile I can see that gitea doesn't call repo_permission.go, in this case:

2019/10/16 12:18:27 models/update.go:80:PushUpdate() [T] TriggerTask 'TAG1' by gituser
2019/10/16 12:18:27 models/pull.go:1088:AddTestPullRequestTask() [T] AddTestPullRequestTask [head_repo_id: xxx, head_branch: xxx]: finding pull requests
2019/10/16 12:18:27 modules/auth/auth.go:97:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2019/10/16 12:18:27 ...s/context/context.go:328:func1() [D] Session ID: xxx
2019/10/16 12:18:27 ...s/context/context.go:329:func1() [D] CSRF Token: xxx
2019/10/16 12:18:27 models/pull.go:1128:AddTestPullRequestTask() [T] AddTestPullRequestTask [base_repo_id: xxx, base_branch: xxx]: finding pull requests
2019/10/16 12:18:28 models/webhook.go:898:DeliverHooks() [T] DeliverHooks [repo_id: xxx]
2019/10/16 12:18:28 models/update.go:80:PushUpdate() [T] TriggerTask 'TAG2' by gituser
[...]

We also tried to migrate a very large repository with around 15.000 commits and 7000 tags. Although we get an error from the git client, everything (tags, commits) gets transmitted and all releases get created by gitea when migrating to a user repository.

user@linux:~/repo.git$ git push --mirror https://gitea.example.org/ORG/repo.git
Counting objects: 409616, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (108168/108168), done.
Writing objects: 100% (409616/409616), 1.14 GiB | 25.52 MiB/s, done.
Total 409616 (delta 201291), reused 409611 (delta 201286)
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
Everything up-to-date
user@linux:~/repo.git$ git push --mirror https://gitea.example.org/ORG/repo.git
Everything up-to-date

Long story short, when we migrate repositories with more, or way more than 50 tags to a user account, everything works. When we migrate them to an organizazion with a lot of members, Gitea gets stuck when creating releases from tags.

Workaround

As a workaround we first migrate to a user account and then transmit the ownership of the repository to the organisation.

[guillep2k]

This seems to be related to #8528 (#8273).

I'd recomment changing app.ini and set:

[database]
MAX_IDLE_CONNS = 5
CONN_MAX_LIFE_TIME = 300

(I'd experiment with the values and see if they improve the results)

[zeripath]

CONN_MAX_LIFE_TIME = 5m

Is easier to read.

[oaxiento]

Thanks you both for your quick response. I changed the database settings and tried migrating one of our biggest repositories (>6,000 tags, >14,000 commits) to our Gitea organisation. After creating 854 releases and around 100,000 repo_permission.go calls I see the same error though:

2019/10/17 08:40:54 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824811255664:SOMEUSER in 824811255760:ORG/repo.git: ...
[...]
2019/10/17 09:06:58 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824662319392:SOMEUSER in 824662319520:ORG/repo.git: ...
2019/10/17 09:06:58 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824662792016:SOMEUSER in 824662792224:ORG/repo.git: ...
2019/10/17 09:06:58 ...ters/private/hook.go:165:HookPostReceive() [E] Failed to Update: ORG/repo.git Branch: BRANCHNAME Error: PushUpdate: CommitRepoAction: NotifyWatchers: insert new action: dial tcp 172.xxx.xxx.xxx:3306: connect: cannot assign requested address

I think this is an efficiency issue, as I don't see why there have to be so many repo_permission.go calls when creating releases from tags. When I migrate the same repository to a user account an then move it to the organisation repo_permission.go only runs one for the user transferring the repository.

[lunny]

@mansshardt Thanks for your reports. Could you share your database setting?

[oaxiento]

Sure! Here they are:

[database]
DB_TYPE = mysql
HOST = mariadb-host.example.org:3306
NAME = gitea
USER = gitea
PASSWD = super-secret-password
SSL_MODE = disable
LOG_SQL = False
MAX_IDLE_CONNS = 5
CONN_MAX_LIFE_TIME = 300

[lunny]

300 means 300 nano seconds, NOT 300 seconds, that's too short. Could you change that to 5m and try agin.

[guillep2k]

Sorry, my fault about the nanoseconds.

[zeripath]

My fault too. I should have checked.

[oaxiento]

No problem! I changed the setting to CONN_MAX_LIFE_TIME = 5m and restarted Gitea. Sadly, this didn't make a difference. After creating 766 releases I still get:

2019/10/17 15:14:30 ...ters/private/hook.go:165:HookPostReceive() [E] Failed to Update: ORG/repo.git Branch: BRANCHNAME Error: PushUpdate: CommitRepoAction: NotifyWatchers: insert new action: dial tcp 172.xxx.xxx.xxx:3306: connect: cannot assign requested address

[guillep2k]

172.xxx.xxx.xxx:3306: connect: cannot assign requested address

This is most likely because of attempting too many connections to MariaDB, but I thought this would be solved by the new settings.

As TCP can only create and destroy <64515 socket pairs in a small lapse of time (TIME_WAIT), and this import operation seems to be depleting that, perhaps you can try using UNIX sockets for your connection to MariaDB instead?

[oaxiento]

I just pushed (--mirror) again and checked the connections to the database via netstat -an | grep ':3306' | wc -l. During the creation of releases I have around 25,000 tcp sockets in TIME_WAIT. As mentioned before I'd guess that every permission call results in one database connection which doesn't get reused.

I don't understand why so many connections are nessecary in the first place. In my opinion fiddling with MariaDB properties or database settings to mitigate the issue shouldn't be a solution. I also can't use unix sockets as the database is not on the same machine.

[guillep2k]

@mansshardt I agree that this should not be the solution. I was only offering alternatives. 😄

For some reason MAX_IDLE_CONNS didn't have any effect. AFAIU it should have had the effect of reusing the connections for most transactions. Especially after CONN_MAX_LIFE_TIME = 5m.

[zeripath]

You probably need my pr which allows you to set maxopenconns to prevent too many open connections to the db

[guillep2k]

@zeripath If I'm not mistaken, the problem in this issue is the number of closed connections, not the open ones.