Feature request: Protocol for downloading files from different peers

[master255]

Maybe I don't know something and it's already available?
I need a protocol similar to the torrent protocol (with all the features: dividing files into pieces, checking hashes, receiving torrent handshakes and distributing files). Allowing to download one file from several peers at once. This will allow to refuse torrents at all and organize a decentralized database. Perhaps it will add programmers to this project. And will add many new programs using this library.

[sukunrt]

I think IPFS is close enough to what you want. https://github.com/ipfs/kubo/

[aschmahmann]

You might want to ask about this in one of the IPFS community channels (e.g. forums, chat, etc.) https://docs.ipfs.tech/community/.

However:

1. Protocols are not the same as binaries. Nothing about protocols like https://github.com/ipfs/specs/blob/main/BITSWAP.md or https://specs.ipfs.tech/http-gateways/trustless-gateway/ indicates how the implementation needs to store your data on disk so it can serve it they just express the protocols for requesting and receiving data. This leaves existing (and new) implementations free to store data however they want while still being compatible with each other at the networking layer.
2. kubo as a particular IPFS implementation allows you to, for example, play some video locally while also distributing it to other people. For example, you can load a .mp4 using http://<my-video-cid>.ipfs.localhost while also serving it over a variety of protocols to other peers. Additionally, you can use the filestore to use existing raw files rather than importing the data into the shared datastore.
   • As with 1, if you prefer a torrent-like interface that only deals with files and separately handles storage of file metadata (e.g. hash function to use, chunking, layout, etc.) separately you can do that while still reusing the same network protocols (and of course you can add more if you'd like and leverage libp2p for protocol negotiation).

[marten-seemann]

If you want to download from multiple peers, you need to split them up. Or you need to range requests, but then you need to think about incremental verifiability.

Fully agree with @sukunrt that this is an IPFS question / feature request.

[master255]

@aschmahmann This feature (filestore) is in even one app? Can you give me the name of an app where I can check how it works?

[aschmahmann]

@master255 I'm not really sure what you're getting at (i.e. what your definition of "app" is and what you're asking) since there's a link to a binary that does the things you've asked for (which again are not protocol or even library layer things, but now application layer). As mentioned above I would recommend asking in the IPFS forums or chat channels since there will likely be more people there with the context to address your questions..

If you end up asking elsewhere I'd drop a link here in case anyone wants to follow along.

[master255]

@aschmahmann I realize that this experimental function is in that library, but I need to do a lot of work to test it.
It is strange that such a function that can replace all torrents is not used in any program. Yes?
Most likely this function doesn't work like torrents.
And I wrote to ipfs programmers and they told me that this is not planned to be developed, because file separation is the main thing in ipfs.

So I think it is not fair to close this topic. Since ipfs will never develop it, but an additional, decentralized protocol to libp2p is definitely needed.

[master255]

I found that Libtorrent can disable DHT and use Libp2p DHT. Also with url_seed Libtorrent can use all Libp2p features to download files.
In other words Libtorrent acts as a decentralized protocol for downloading files from Libp2p nodes.

Who knows, maybe this is a transition to new torrents and Ipfs in the future.

Testing Kubo is in my distant plans for the future.

[master255]

I ended up enhancing Libtorrent with Libp2p in my Media Library application. The resulting Libp2p torrents. Nice workaround to torrent blocking in 4G, but not all of them. I'm working on extending the capabilities of Libp2p torrents. There are already two of my Libp2p apps in the Play Market.