Resumable video uploads

[kontrollanten]

Description

Make audio/video uploads resumable when the internet connection is flaky. This is done by using node-uploadx and ngx-upload. This family of libraries supports multiple operations modes (TUS v1.0, a Google Resumable v3.0 variant and multipart). We selected the default mode, which is using a variant of Google Resumable (used in Drive and YouTube upload endpoints) and is summarized here: https://github.com/kukhariev/node-uploadx/blob/master/proto.md

/api/v1/videos/upload was left untouched.
/api/v1/videos/upload-resumable was created and dedicated to this new upload method.

The OpenAPI spec was modified to include the new upload API.

Related issues

closes #324
kukhariev/node-uploadx#318

Has this been tested?

• 👍 yes, I added tests to the test suite
  • video upload
  • audio upload

Screenshots

[rigelk]

@kontrollanten thanks for taking care of this issue! I didn't review much the backend, but will check more thoroughly once the PR leaves its draft state.

[rigelk]

@kontrollanten btw if you need help for the implementation (like tests or tmp cleanup as job), I can give you a hand.

[kontrollanten]

@kontrollanten btw if you need help for the implementation (like tests or tmp cleanup as job), I can give you a hand.

Thanks. I'm thinking about what the best solution is for the server tests that are using the shared/extra-utils/videos/uploadVideo function. In one way the new upload process is an implementation detail that are out of scope for our testing purpose, but in the other way we need an easy way to upload videos in the tests. But to rewrite the logic from ngx-uploadx just for testing purpose feels overkill.

[rigelk]

It seems node-uploadx has been updated with the required changes to further this PR: https://github.com/kukhariev/node-uploadx/releases/tag/v4.3.0

PR changed sufficiently to require a re-evaluation

[rigelk]

@kontrollanten I rebased and moved your code to upload-resumable, and restored the upload route. This prevents removing from the API blueprint, and also allows us to not break tests at the same time as we add new ones.

[kontrollanten]

Thanks @rigelk ! What do we got left to be able to merge this? Tests and a cleanup job for /tmp? If you haven't started on the cleanup job I think I can do that. I that it should be a repeat job that runs once an hour.

[rigelk]

Tests and a cleanup job for /tmp?

Yes, and I'd like to cleanup the object passing (file, metadata) which is brittle at the moment. Having well-typed object passing there is important, and especially regarding the plugin hook which receives the object.

As for tests, we need to try not as much as the legacy upload, but at least the audio-only resumable upload, which is handled very differently from the rest within the middleware.

[kontrollanten]

It looks like the cleanup job fails to run in the tests, but I've no idea how to debug.

[Chocobozzz]

Great work!

requested changes covered now refactored code

[Chocobozzz]

PR is ready, waiting for kukhariev/node-uploadx#337

[Chocobozzz]

Thanks @rigelk @kontrollanten and @kukhariev for your work and reactivity