Extend gitea to add custom lfs upload/download adaptors

[praveentiru]

• Gitea version (or commit ref): 9619ccf
• Git version: v1.10.0
• Operating system: Windows
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

This is an enhancement request that I can work on once everyone agrees on spec. This request is inspired from Bitbucket's LFS Media Adapter. Adapter chunks large files and uploads them in parallel.

Proposal:

1. Add a config which would take a list as input.
2. Each item in list will hold an object with two key-value pairs
   1. adapter=mymagicsauce
   2. url=https://www.xxx.com/lfssauce
3. Object link and verify link for upload will be built using url key value as base
4. Gitea will expose an url for adapter service hit to verify upload of files
5. Person extending with custom adapter will also provide client adapter component.

Looking forward to feedback from the team.

[zeripath]

This sounds interesting. How does it compare to tus.io (#1723)?

Gitea recently got some LFS repository management settings pages - would this mean those stopped working?

[praveentiru]

@zeripath tus.io is only one approach. But, there can be lot of different approaches possible. Few options:

• Upload to a cloud bucket in multiple parallel connections
• Manage LFS objects in a local storage while Gitea is hosted on cloud (Sensitive binary data)
• Chunk binary files and upload only the chuncks needed and re-assemble them after upload

So, an extensible approach will allow people to build solutions with different approach. People who use Gitea can select which options to use and this code need not sit in gitea code base.

[zeripath]

Ah, so you're suggesting that instead of offering the Gitea LFS server we can set up the repo to proffer a different LFS server?

[praveentiru]

I am proposing Gitea LFS server would still be primary controller. It will manage everything with exception of Upload/Download of LFS files. That piece will be managed by a separate server-client combination.

[praveentiru]

Here is execution flow that I see:

1. LFS server receives Batch Upload or, Download request
2. LFS server authenticates the request
3. If not authenticated, exit
4. LFS server reads from settings list of transfer adapters supported with preference
5. LFS server identifies most preferred adapter that is supported by current server
6. LFS server sends the Batch request to upload/download adapter server
7. Adapter server responds with fully formed response
8. LFS forwards the response to client
9. Client transfer client and adapter manage upload/download
10. On successful completion, adapter server calls on verify link of Gitea server
11. Transfer cycle closes

Complications with this approach:

1. Adapter control where LFS files are stored. When there is no parity adapters supported in clients using a repo, LFS files would end up spread around.
   1. Restricting exposure to single adapter is brutish way to force that all clients have necessary client side adapter in place.

Looking for feedback on approach before working on the same.

[lunny]

I think this will be easier after #12518 merged.

[lunny]

Let's close this since we have the modules/storage package and the interfaces and just send PRs to implement new adaptors.