Allow uploads to be resumed

[conferences-gesticulees]

Uploading videos take hours with most connections available today. Waiting for hours to get nothing because the connection breaks for a couple minutes, the server gets restarted, or a power failure at 5am is highly frustrating.

Users should be able to resume their uploads. Ideally, it should be possible to resume a partial upload. At the very least, it should be possible to complete the publishing process after an upload is done. For example if an error happens during the processing stage, users should not have to upload the whole video again.

A checksum of the bytes already uploaded could be compared to ensure the local file has not been modified in the meantime.

Partial uploads should not be kept on the server indefinitely. I guess keeping them for 10 days would be long enough.

[MayeulC]

How about uploading the video with webtorrent as well (with the current way as a fallback)? It could even open up some interesting functionalities down the road, like allowing the video to be watched before it's completely transferred, and maybe helping pave the road to livestreaming :)

[rigelk]

@MayeulC actually uploading the video with WebTorrent would mean uploading the video with a WebRTC data channel to the server and using BitTorrent in it to declare (creating a torrent) and then transfer the file. WebRTC acts as a mere transport layer for a BitTorrent session whose goal is to share a file. The first problem is the fact that torrents are immutable: it prevents livestreaming (see issue #151). In our case the file is a video, but what prevents a client from declaring a torrent for other file types? The server should be able to check the file conforms to what a video is, thus downloading the video in its entirety anyway before forwarding the declared torrent. So I'm afraid using WebTorrent for the upload wouldn't bring anything, except added complexity.

[JigmeDatse]

@rigelk I've been thinking about this feature. The underlying .torrent could potentially be uploaded, but to do so would require the local generation of the .torrent file. Now I'm not sure if this is a possibility or not with how things are currently designed.

From my understanding the only way that would allow that, without someone manually creating the torrent locally than uploading it (and seeding it) would be to try to do it with JavaScript in the browser.

I don't know if that is a feasible way to do things or not either. But if there were the option of using the torrent protocol to upload, that could allow for the creation of clusters. Without that, I think the issues I've had with being able to watch videos will probably continue, as I suspect that the limited numbers of peers is a significant factor.

It is also possible that the peers that do exist for a given video with performance issues are behind a "too slow connection."

[Chocobozzz]

Without that, I think the issues I've had with being able to watch videos will probably continue, as I suspect that the limited numbers of peers is a significant factor.

Please if you have playback problems on peertube create an issue so we can investigate.

[rigelk]

We should not mix up things in this issue. Correct me if I'm wrong, but there are two options considered so far:

1. keep it simple by improving the actual HTTP upload mechanism. Resumable file uploads are not something out of the ordinary, and there seem to be quite a few libraries out there: https://github.com/23/resumable.js/ , https://tus.io/ − and even integrations like https://uppy.io/
2. go a step further, with a client (in the browser or out of it) creating a torrent and seeding it.

Frankly, I think 1. is way more realistic, since 2. complexifies things a lot.

[JigmeDatse]

I will look into it next time. I'm not sure I could really get any idea of how to figure out the ones that were a problem in the past. I have enough problems with other things, that I'm never quite sure what that problem might be.

[JigmeDatse]

There is a third option:

3. allow the uploading of a seeded torrent file.

I don't know the code base at all, so whether that is an option that ends up as "even more complex" or "less complex" I have no idea. The more I'm seeing the community, and the development team, the more I'm leaning toward staying with YouTube and Vimeo...

[rigelk]

@JigmeDatse Don't misunderstand me. I'm merely trying to give hints as to whether or not this is easy, not refusing any option, just so that we can prioritize ways to solve the problem best. Whichever the solution, if you want to get to know the codebase to implement it yourself, we'll be more than willing to help. I don't think you'll ever be given that much by YouTube's.

[JigmeDatse]

I don't believe I am misunderstanding you. It's not merely comments to me, but I have seen repeatedly comments of "we don't want to do this." You're quite right that I'd never be given the right to "rewrite" YouTube, and call it YouTube, but I also wouldn't see bickering between a developer and users ever with YouTube or Vimeo.

Perhaps the problem isn't so much that you disagree with people, but that you present roadblocks with no indication of any effort on your (or any other dev member) interest in helping to find them. To you that may be "merely trying to give hints," to most people who aren't already involved, this is probably a strong indication of "we won't do this, unless you are willing and able to do it yourself."

Which to me means "please feel free to fork, because you're going to basically be doing that anyway."

[JigmeDatse]

I think you've made my decision for me. Thanks for the effort to be "more than willing to help." as you've helped enormously. I'm going to go back to seeing about MediaGoblin where "more than willing to help" was never mentioned, they just proved it by saying, "oh yeah, that's something to fix..." and they went ahead and fixed it. If it was something which required more than that they did work to do that. I don't have the time or energy to try to work with your team...

[Booteille]

Maybe we should continue to discuss about how to fix this issue.
I really think we should have the ability to resume upload and in the first instance make it work simple so we can then take our time to think about a more sophisticated way to do that.

[filmaidykai]

Or maybe is possible to upload file straight to folder without web browser with ftp client like filezila?

[cognition9144]

I would like to help with introducing resumable.js and implement this.