Running gitea across multiple servers

[svarlamov]

Been working with Gitea for a few days now, and I find the project quite intriguing! Now, I'm wondering what it would take to be able to run gitea on a multi-host setup, likely behind a single nginx load balancer... I think that it would be very useful to enable people to host across servers, first and foremost because of storage limitations to a single machine, as well as setups where we don't want to actually keep the files locally, but we want to hold them on S3 or a similar service... In addition, with larger teams that are online 24/7, availability and CPU begins to become a bottleneck...

Overall, it looks like Gitea's architecture would not be too challenging to distribute given it already has a central DB and stateless HTTP APIs (please correct me here if I'm wrong...), however, it seems to have two key things holding it back right now:

1. No distributed task/messaging/scheduling queue
2. No way to store repos beyond the local FS

I am excited to hear the maintainers' thoughts on this, perhaps how it fits into the roadmap for Gitea, and what sort of work this would involve. 😄

[lunny]

@svarlamov Your thoughts are our long-term objective. For sqlite version, it's only suitable for single node. For other databases, you can connect to a database proxy for distribute deployment. And also for reverse proxy load balance and shared session(via Macaron redis session) are both ready.

As you mentioned above, there are two enhancements need to do. I think the second is more difficult than the first one. As I know there are no out-of-box go git library which support S3 or other cloud storage system. So we have to implement it ourselves in https://github.com/go-gitea/git. I have sent a PR go-gitea/git#92 to begin the work.

In fact, we also need some work on split SSH server or hooks with databases. I have added an issue #1023 and serval PRs to do that but not finished. After that, the SSH server could be divided with HTTP server.

If somebody wants to provide public git services based on Gitea, all these things maybe considered.

[kolaente]

And what about just hosting a bunch of dockercontainers with a shared file system over multiple hosts?

[lunny]

@kolaente but you have to mount the shared file system every container

[svarlamov]

@lunny Sounds interesting... So the SSH stuff basically needs to get 'sharded' for lack of a better worded? And what about 'locking' repos so that people don't step over each other?

[svarlamov]

@kolaente That's definitely a possibility, however, I'm wondering if that will still have issues working out of the box as you will still need a distributed queue and some sort of locking method I believe... Also, from a scalability perspective, you will likely need to shard those volumes sooner or later anyways

[svarlamov]

@lunny After doing some more research on this topic, the following items have come to my attention:

1. Using a networked FS (ie., through sharing vols on Docker or an NFS impl) raises quite a few issues, but most crucially, latency... Given git's object store architecture, this is basically impossible.

2. libgit2 (with additional go bindings) could technically be used to store files on a DB (or elsewhere) through implementing the backend interface (https://www.perforce.com/blog/your-git-repository-database-pluggable-backends-libgit2), but this, again for latency reasons, is unlikely to be usable

3. gitea's use of the go-gitea/git wrapper makes using libgit2 problematic...

4. Access through the ssh method requires that files sit on the filesystem (or an equivalent like a FUSE implementation), otherwise the standard git tooling will not work (for obvious reasons)

5. Further to 1, 2, 3, and 4: using anything except for a filesystem (or equivalent setup as far as latency is concerned) is really the only way to serve git. Methods such as NFS, S3, etc., can only be used for backups - not 'hot' storage

My research also highlighted the Github eng team's blog, where they discuss similar issues and echo many of the same concerns:

• https://githubengineering.com/introducing-dgit/
• https://githubengineering.com/building-resilience-in-spokes/

Their conclusion can be most simply described as 'sharding', where the repositories ultimately sit on native filesystems (as Gitea currently stores them) and requests are then proxied to the right servers through a (in their case dynamic) system that knows where each repo is. In their design, the repos are actually stored across 3 replicas for high availability, which is pretty useful, especially if you end up running all of this on commodity hardware...

This sort of sharding logic seems to fit pretty naturally into Gitea and is probably the most feasible solution that could be implemented once Gitea's other services are ported to multi-host.

Separately, LFS would also need to be ported. Based on my current understanding, it is pretty straightforward to migrate the LFS store off of the filesystem and it could either be done through the same sharding method or through a more generic driver-based system where gitea itself doesn't actually bother with the details of how the objects are stored...

I would also like to raise a new topic that is relevant to multi-host deploys, and that is multi-host content search! Looks like this would need to use a 3rd party indexing/search platform such as elastic search...

And as far as the 'locking' issue is concerned, it seems that, as I mention in point i below, that is managed by a neat custom mutex wrapper that controls these 'pools' -- this would definitely have to be re-written for multi-host support and should rely on an existing distributed tool

I'm keen to hear everyone's thoughts on this :)

P.S. Below I've also pasted some notes (they don't aim to be exhaustive by any means) of modules that look to be stateful and would need to be modified to support multi-host deployments...

a. https://github.com/go-gitea/gitea/blob/master/modules/cron/cron.go <- This seems to assume a single-host setup. For multi-host, this sort of stuff is not usually done within the primary application server (except for internal stuff, such as memory-related jobs...). Perhaps this can be optionally turned off and handled by a separate server? Or moved to a generic service? There are lots of wonderful open source options here

b. https://github.com/go-gitea/gitea/blob/master/modules/lfs/content_store.go <- This would need to point to the storage method for LFS, or potentially an LFS driver?

c. https://github.com/go-gitea/gitea/blob/master/modules/ssh/ssh.go <- Assumes single-host setup

d. https://github.com/go-gitea/gitea/blob/master/modules/cache/cache.go <- Looks like this is good to go with built-in support for 3rd party caching systems configured in the options!

e. https://github.com/go-gitea/gitea/blob/master/modules/indexer/repo.go <- Indexing relies on the server's FS...

f. https://github.com/go-gitea/gitea/blob/master/modules/mailer/mailer.go <- Likely needs to use a distributed task queuing service, and should probably support retries beyond a few seconds... There are some good tools for the job here that are widely supported

g. https://github.com/go-gitea/gitea/tree/master/modules/markup <- Looks like this is bound to the local filesystem

h. https://github.com/go-gitea/gitea/blob/master/modules/notification/notification.go <- Seems to be ready to use memcache, so should be fine

i. https://github.com/go-gitea/gitea/blob/master/modules/sync/exclusive_pool.go <- This seems to perform the 'locking' functionality that we've been discussing... This would essentially have to get moved up to a higher-level distributed service that would actually manage the state, with this file becoming a wrapper?

[lunny]

@svarlamov we need a FileSystem or Storage abstract layer.

[svarlamov]

@lunny How would that fit with a sharding approach? I'm thinking we would be reading from the local FS in the end, but it's just a matter of how to get the request onto the right machine

[kolaente]

I've quickly tested a multicontainer with shared volume approach (in rancher). It seems it cannot start the webservice when running with multiple instances, I'll do some tests later today.

@svarlamov Rancher can abstract the FS for you to AWS for example. For the database you could do something with a galera instance.

[svarlamov]

@kolaente Interesting... I will play around with it as well... In regards to the shared volume topic, please have a look at my comments above, namely the discussion of latency, particularly when coupled with Git's object-store architecture. This is what is discussed in Github's engineering blog as well, and they outline why you can't effectively run a git service on top of that sort of a shared FS architecture

[kolaente]

So, after some testing I found the following: when gitea shares the data volume, it cannot start more than one container. But if everything except the gitea/indexers folder (what is this for?) is shared, it works out of the box! With that said, I don't think it is needed to build distributed storage into gitea, it probably is easier to let something like rancher or portworx handle the storage aspect.

But as soon as you start sending forms (for example when creating an issue), you get crfs token errors all over the place... There is some work to do.

I also created a PR to add Gitea to the rancher community catalog: rancher/community-catalog#680

[lafriks]

Did you try changing caching from memory to memcached? Also session storage should be changed from file to something else

[dapperfu]

Some benchmarks and CPU numbers putting Gitea behind nginx.

This is tested on my FreeNAS machine with Gitea & nginx running in a jail.

$ sysctl hw.model hw.ncpu
hw.model: Intel(R) Atom(TM) CPU  C2758  @ 2.40GHz
hw.ncpu: 8

Using bombardier on another machine to drive HTTP load.

Gitea configured to use mysql and memcached. Nginx configured with 8 worker processes.

Baseline idle:
last pid: 88584; load averages: 0.69, 0.78, 0.79 up 8+15:11:08 17:08:20
31 processes: 1 running, 30 sleeping
CPU: 12.3% user, 0.0% nice, 1.2% system, 1.6% interrupt, 84.9% idle
Mem: 417M Active, 85M Inact, 46M Laundry, 14G Wired, 995M Free
ARC: 8561M Total, 5333M MFU, 1059M MRU, 2836K Anon, 146M Header, 2019M Other
5371M Compressed, 6498M Uncompressed, 1.21:1 Ratio
Swap: 6144M Total, 2057M Used, 4087M Free, 33% Inuse

PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
64132 mysql        49  52    0  1970M  5980K select  1   8:49   0.06% mysqld
57145 nobody       10  20    0 37408K  1388K kqread  2   0:17   0.01% memcached
66858 git          39  21    0   118M 21240K uwait   0  27:42   0.00% gitea
69043 www           1  52    0 35604K  1144K kqread  4   6:22   0.00% nginx
69044 www           1  20    0 35604K  1152K kqread  6   6:17   0.00% nginx
69042 www           1  52    0 35604K  1124K kqread  6   6:15   0.00% nginx
69045 www           1  20    0 35604K  1152K kqread  0   6:12   0.00% nginx
69046 www           1  52    0 35604K  1112K kqread  5   6:06   0.00% nginx
69047 www           1  52    0 35604K  1132K kqread  0   5:56   0.00% nginx
69048 www           1  20    0 35604K  1744K kqread  4   5:16   0.00% nginx
69041 www           1  20    0 37652K  1176K kqread  0   5:11   0.00% nginx

Connecting to gitea directly with 200 connections:

last pid: 89104;  load averages: 11.48,  4.97,  2.46                                                                                                                                                                 up 8+15:13:41  17:10:53
31 processes:  2 running, 29 sleeping
CPU: 67.4% user,  0.0% nice, 22.9% system,  5.1% interrupt,  4.6% idle
Mem: 117M Active, 343M Inact, 637M Laundry, 14G Wired, 173M Free
ARC: 8704M Total, 5249M MFU, 1197M MRU, 56M Anon, 147M Header, 2054M Other
     5404M Compressed, 6704M Uncompressed, 1.24:1 Ratio
Swap: 6144M Total, 2012M Used, 4132M Free, 32% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
66858 git          43  80    0   496M   439M RUN     7  32:04 495.09% gitea
64132 mysql        65  52    0  1990M   188M select  2  10:06 161.79% mysqld
57145 nobody       10  20    0 37408K  1388K kqread  2   0:17   0.02% memcached

Latency results:

Bombarding http://172.16.3.104:3000/explore/repos with 10000 requests using 200 connections
 10000 / 10000 [=======================================================================================] 100.00% 18s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       544.74     499.01    6392.01
  Reqs/sec     365.27ms    74.52ms      1.08s
  Latency Distribution
     50%   350.44ms
     75%   444.55ms
     90%   544.68ms
     99%   739.92ms
  HTTP codes:
    1xx - 0, 2xx - 9810, 3xx - 0, 4xx - 0, 5xx - 190
    others - 0
  Throughput:     4.04MB/s

Connecting using nginx as a reverse proxy:

last pid: 90424;  load averages:  2.56,  5.77,  3.85                                                                                                                                                                 up 8+15:18:32  17:15:44
31 processes:  3 running, 28 sleeping
CPU:  5.4% user,  0.0% nice, 26.8% system, 16.1% interrupt, 51.8% idle
Mem: 108M Active, 482M Inact, 517M Laundry, 14G Wired, 397M Free
ARC: 8485M Total, 5114M MFU, 1286M MRU, 5699K Anon, 148M Header, 1933M Other
     5408M Compressed, 6795M Uncompressed, 1.26:1 Ratio
Swap: 6144M Total, 2011M Used, 4133M Free, 32% Inuse
 
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
69043 www           1  48    0 35604K  1452K kqread  3   6:28  44.93% nginx
69046 www           1  47    0 35604K  1424K kqread  3   6:12  40.29% nginx
69044 www           1  47    0 35604K  1456K CPU5    5   6:22  36.64% nginx
69048 www           1  45    0 35604K  2112K kqread  4   5:21  35.97% nginx
69045 www           1  43    0 35604K  1456K kqread  2   6:17  35.21% nginx
69047 www           1  47    0 35604K  1412K kqread  5   6:01  31.80% nginx
69042 www           1  42    0 35604K  1432K kqread  7   6:21  29.14% nginx
69041 www           1  32    0 37652K  1420K RUN     1   5:15  27.52% nginx
66858 git          43  20    0   496M   439M uwait   6  40:56   0.00% gitea
64132 mysql        49  52    0  1997M   195M select  1  13:05   0.00% mysqld
57145 nobody       10  20    0 37408K  1388K kqread  6   0:18   0.00% memcached

Latency results:

Bombarding http://172.16.3.104:80/explore/repos with 10000 requests using 200 connections
 10000 / 10000 [========================================================================================] 100.00% 1s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec     10917.23    1672.91   15680.40
  Reqs/sec      17.78ms     1.22ms    52.06ms
  Latency Distribution
     50%    18.02ms
     75%    18.02ms
     90%    19.02ms
     99%    28.03ms
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   106.99MB/s

Nginx was able to not just keep up with the load but also served the results in less time. What was interesting was when you started doing a lot more connections.

Testing out 2000 connections caused gitea to choke and start returning 5xx codes while nginx kept up (even if it did slow down a bit, latency 90% latency was around 600 ms).