Determine file storage solution

[gctucker]

Several options can be considered for storing files to be used with the new API. While kernelci-backend has file storage integrated in the API, it would seem a better approach to make it entirely separate with the new API and the meta-data would just contain URLs to the artifacts. This would make it possible to use different storage mechanisms without changing the API implementation.

Simple solution: direct copy or upload via SSH and serve via nginx #15

The easiest solution for a local setup is probably to have a directory mounted as a volume in a container running nginx. The user can directly copy files to the directory and they will be served by nginx via HTTP. In production, the files could be sent over SSH to the host. Alternatively, another container could be run with the SSH server on a special port number and the same volume mounted in the 2 containers.

API solution: upload via FastAPI and serve via nginx

FastAPI support for uploading files: https://fastapi.tiangolo.com/tutorial/request-files/

This is similar to the simple solution except that instead of using SSH to send files they would be uploaded via FastAPI. The same user token can be used as for other API endpoints. Some meta-data could be kept in Mongo DB about the files, although it's not clear how much value this would add. This breaks the idea of decoupling the API from storage, so it's not great for that reason. It's still technically a possibility.

Advanced option: file / artifact manager

A dedicated service could be set up, a bit like Artifactory but with an open source framework. This would provide an API just for uploading and managing files. Ideally, this could have extra features such as automatically deleting files after a period of time, managing permissions with user accounts, quotas, backups and if possible replication to maintain mirrors in different geographic locations. It's not clear yet which framework could be used for that. There are several file management packages for Django. There's also NextCloud.

Mongo DB: GridFS

Another possibility is to use Mongo DB's GridFS feature, to directly store files in a Mongo database. It would probably also require an API endpoint to upload files. One advantage is that we can directly rely on the Mongo DB replication to distribute the files geographically. Otherwise, it doesn't necessarily seem like a very good fit but it's worth keeping in mind.

[mharyam]

@gctucker
Please assign this to me

[gctucker]

Well it's not really a "good first issue", it's not clear yet what solution we'll be using in the end. However there should be some more concrete issues that you could pick up next week around it, with a better defined scope. I think we'll start with the simple approach and an nginx container.

If you want to investigate the different types of solutions mentioned here and compare them, that would be very useful. But it may not directly lead to some code contributions, which I believe is what is required for Outreachy applicants (as per the FAQ).

[mharyam]

Well it's not really a "good first issue", it's not clear yet what solution we'll be using in the end. However there should be some more concrete issues that you could pick up next week around it, with a better defined scope. I think we'll start with the simple approach and an nginx container.

If you want to investigate the different types of solutions mentioned here and compare them, that would be very useful. But it may not directly lead to some code contributions, which I believe is what is required for Outreachy applicants (as per the FAQ).

Well it's not really a "good first issue", it's not clear yet what solution we'll be using in the end. However there should be some more concrete issues that you could pick up next week around it, with a better defined scope. I think we'll start with the simple approach and an nginx container.

If you want to investigate the different types of solutions mentioned here and compare them, that would be very useful. But it may not directly lead to some code contributions, which I believe is what is required for Outreachy applicants (as per the FAQ).

Okay
Thank you

[gctucker]

@mharyam Please see #15, which should be more like a good first issue.

[fooishbar]

We use MinIO for large-file storage on freedesktop.org, and I think doing something like that is a good idea for KCI too.

Firstly, I think an external storage service is good because it means that you're not putting huge loads through your application server. This makes a big difference as you scale: you're not tying up your application server for large periods of time putting huge transfers through. You can also optimise nginx differently for throughput on bulk transfers vs. latency on API calls, and also balance your servers differently (e.g. file storage wants an emphasis on storage + memory + I/O, whereas app is more CPU-intensive; you can also let the file-storage servers use more memory for disk cache without competing with e.g. Redis).

Secondly, using an S3-compatible service like MinIO gives you a lot of tooling which deals well with large transfers. Stuff like multi-part/resumable/parallel transfers, caching, and wrapping this up in nice easy-to-use interfaces for most languages or from the CLI. When you're talking about 4GB images this is pretty much a non-negotiable since networks are terrible.

Lastly, they already give you things like access control and policy, expiry, etc, without having to reinvent the wheel within an app service which doesn't usually deal with those kinds of things.

I'm not sure I'd recommend MinIO itself though. We've been using MinIO + OPA: the only part of MinIO's access control we use is for manual admin work, and then all the day-to-day bits are handled by a separate part of OPA policy. This is particularly cool because we can use JWTs from GitLab CI to interrogate per-job values in a trusted way, rather than having long-lived access keys which need to stay secret.

Unfortunately MinIO upstream have deprecated OPA integration and have no plans to work on anything similar as far as I can tell. This is a bit of a recurring theme with MinIO, where things get shut down with little explanation, which is especially worrying when it's pretty much a single-person project.

You could use MinIO STS instead since they seem to have more appetite to support that longer-term, or something other than MinIO. The ultra-cloud solution would be Ceph + OPA but that is genuinely terrifyingly complex.

cc @em- who might know some more about this too

[edit: actually, this blog post explains it - As we enter into the next phase of the company, the only metric that we will be measured on is revenue growth. We have built the commercial engine around MinIO subscription. When customers purchase a subscription they get a commercial license to use MinIO without any concern around open source violations or restrictions.]

[gctucker]

Many thanks @fooishbar for the pro tips. I see S3 and Cloud storage in general as part of the "Advanced option: file / artifact manager" approach, which I also think is the best category of solutions for production deployment. I guess the next step is to do some comparisons between the different ways to achieve that. Maybe the aws commands or some Python bindings would work too, and we probably also should consider what Azure has to offer since we already have a KernelCI account there. In fact I've already found some comparison in the Azure docs.

Then for stand-alone instances such as for individual developers or default docker-compose setup, we can stick to ssh and nginx containers like we have now. I guess the other options using the API like on the current backend or Mongo DB to store file objects can be abandoned at this stage.