🌟 adding Torrent support to IPFS

[daviddias]

I've started working in enabling Torrent support for js-ipfs, very much in the same way that we have support for: dag-pb, dag-cbor, eth-blocks, eth-tx, zcash (go-ipfs only), git (go-ipfs only) and bitcoin (go-ipfs only).

The end goal is to expose two top level commands to add and retrieve files that are Torrents, from the IPFS or BitTorrent network (through a bridge and in the future, by connecting directly). The commands being:

• jsipfs torrent add
• jsipfs torrent cat

However, I stumbled upon a question in which we will have to make a decision and I would like to get feedback before going at full speed. In BitTorrent, torrent files are not referenced by a Cryptographic hash due to their ephemeral and mutable nature (in fact, decoding and encoding is not even always idempotent by spec), the only thing that has a cryptographic identifier is the info field in the torrent file.

I started implementing the IPLD format for a Torrent file, but I'm guessing that most people will want to fetch their torrent through the infoHash of the torrent that they get from a thing like a magnetic URI, the crux is that there is never a file for the info field, as soon as a infoHash query is performed, a Torrent file is retried, rising the question of:

Should dag.get(<infoHash>/somePath) resolve through the retrieved Torrent file or only over the info field?

• Resolve through the Torrent file - This is weird to the IPLD resolver, as it would be resolving an immutable pointer to something that has more fields
• Resolve only within the info field - This would force us to make the info field a full standalone object that can be transferred independently (the solution I'm leaning towards). This option would result in two multicodecs for Torrents, torrent-file and torrent-info.

Thoughts? //cc @jbenet @whyrusleeping @nicola

Resolve only within the info field - This would force us to make the info field a full standalone object that can be transferred independently (the solution I'm leaning towards). This option would result in two multicodecs for Torrents, torrent-file and torrent-info.

This sounds like the pragmatic way for me too -- we'll likely get a better idea of what to do with the whole torrent in the process of working on this.

Given that the torrent file itself is not already content-addressed, it's also the "correct" way I think. Magnet URIs address the info hash anyway.

{
  "infoHash": "d2474e86c95b19b8bcfdb92bc12c9d44667cfa36",
  "infoHashBuffer": {"/": "$infoHashAsCID"},
  "name": "Leaves of Grass by Walt Whitman.epub",
}

[daviddias]

@lgierth I only received your comments after I posted, it seems that we had this chat while you were writing those :)

Notes from a chat with @jbenet and @whyrusleeping

• Instead of having ipld-torrent-file and ipld-torrent-info, we just need ipld-bencode to be able to resolve through bencode encoded objects. After rethinking this, I remembered why we need ipld-torrent-file and ipld-torrent-info, we need them to enable the resolver to resolve through paths inside these objects (i.e. dag.get(infoHash/pieces/0) should return the piece and not the string that is the sha1 of the piece).
• When importing a Torrent, two objects need to be created, one for the info and one for the Torrent file itself.
• New command added: torrent import <torrent-file, magnetic-uri, infohash>
• jsipfs torrent will be available through a module called ipfs-torrent that exposes both a CLI and a module (like ipfs-unixfs-engine).

This leads to the following steps

1. Implement the IPLD Formats to support torrents

• ipld-bencode
• ipld-torrent-file (see 🌟 adding Torrent support to IPFS #779 (comment))
  • make sure to canonicalize them (see 🌟 adding Torrent support to IPFS #779 (comment))
• ipld-torrent-info (see 🌟 adding Torrent support to IPFS #779 (comment))
• integrate in ipld-resolver
• integrate in js-ipfs

2. Implement a blockstore that uses webtorrent as it's storage driver

• torrent-pull-blob-store
• confirm that we can dag.get(<torrentHash or infoHash>) and traverse through those objects

3. Implement the ipfs-torrent service (like ipfs-unixfs-engine)

• module
  • .import (adds the torrent file and creates an infoHash object too)
    • import by magnetic URI
    • import by infoHash
  • .add
    • single files support
    • directories support
  • .cat (single files)
  • .get
• cli
  • spawn a js-ipfs daemon or connect to a remoteDaemon

[jbenet]

\o/

[dignifiedquire]

@diasdavid maybe wait with the torrent blob store for the datastore refactor?

[daviddias]

@dignifiedquire I see the value, but won't block Torrent support because of the datastore refactor, it is not a dependency.

[daviddias]

To keep on log, here is the real structure of both Torrent file and info fields - https://wiki.theory.org/BitTorrentSpecification#Metainfo_File_Structure

[daviddias]

Bringing this one back (🎪 )

Instead of having ipld-torrent-file and ipld-torrent-info, we just need ipld-bencode to be able to resolve through bencode encoded objects. After rethinking this, I remembered why we need ipld-torrent-file and ipld-torrent-info, we need them to enable the resolver to resolve through paths inside these objects (i.e. dag.get(infoHash/pieces/0) should return the piece and not the string that is the sha1 of the piece).

It turns out that we might actually just need to do the bencode, because the format, as described in -- https://wiki.theory.org/BitTorrentSpecification#Metainfo_File_Structure -- prescribes that the SHA1 hashes of the pieces be all concatenated, which means that there won't be any <infohash>/info/pieces/<insert piece number>, unless we apply a transformation to the bencoded data in the first place.

This means that we won't be able to use IPLD resolver to traverse through, without transforming the data, as that pieces field will just be a very long byte array value.

It's pretty ironic, but we can exploit the fact that it prescribes SHA1 and split every 40 bytes.

[daviddias]

20 bytes*, @lgierth we can indeed, that falls into the 'Transformations' category, as IPLD compatible format goes, we are strict about not messing with the data.