Basic spec compliance needed to get a node-ipfs running

[daviddias]

minimum spec

• network
  • NAT Traversal
  • TCP
• routing
  • mDNS
  • DHT
• exchange
  • bitswap
• merkleDAG

validate the usefulness of code already created (deprecate/rebuild/good)

• network
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-network-pipe
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-network
• routing
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-peer-book
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-peer
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-dht-routing
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-dht
  • ~~https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-message~~(deprecated)
• exchange
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-storage
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-bitswap
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-block
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-blocks
• merkleDAG
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-object
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-objects-git
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-path-resolver
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-path
• not a concern for now or didn't understand what they were
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-cli
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-core
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-errors
  • https://github.com/ipfs/node-ipfs/tree/master/submodules/ipfs-mount

The go is to go through these modules, understand what stays, what leaves and what is missing compared to the go-ipfs. (probably will draw some ASCII art after I get this into my head, sometimes helps to think and communicate better)

candidate attack plan:

• Write down the spec of what is necessary to get a working node-ipfs node ( @jbenet )
• Write down (or point out) the spec for how things are put on the wire by go-ipfs, so we can do a tiny node thing that assures that we are using the same. @whyrusleeping @jbenet maybe it would be cool to strip out a tiny test example from go-ipfs just to make sure we get node-ipfs right. note: consider if spdy is going to be a concern of this first impl. - https://github.com/ipfs/specs/blob/wire/protocol/network/wire.md
• Get go-ipfs and node-ipfs talking (based on the previous item) (@diasdavid )
• Get DHT working over local network (@diasdavid )
• Get mDNS working (@diasdavid)
• Get NAT traversal working
• Get bitswap working
• Get MerkleDAG working

[daviddias]

@jbenet "candidate attack plan" from the meeting notes is ready :)

[jbenet]

@diasdavid candidate attack plan looks great to me!

• we should whittle down assignments to one person (a single Directly Responsible Individual)
• I wrote up the wire stuff yesterday (pushing shortly). It's not what go-ipfs does today, but it's what we'll move to (cc @whyrusleeping) for node-ipfs interop.
• still writing module interfaces

[daviddias]

From @jbenet on IRC

• https://github.com/ipfs/specs/blob/wire/protocol/network/wire.md
• https://github.com/ipfs/specs/pull/9/files

we should whittle down assignments to one person (a single Directly Responsible Individual)

Agree, I've put myself doing "Get go-ipfs and node-ipfs talking", it will help pave the path for the remaining tasks and how to partition them.

[jbenet]

Great! 👍

[jbenet]

we can probably assign the rest once we have a clearer scope. also, i think @krl began working on the node merkledag.

[jbenet]

For basic wire protocol interop, all we need is (lots of steps but each one is relatively small):

• spec out multistream and multistream-select (@jbenet)
• write out a wire protocol description (@jbenet) https://github.com/ipfs/specs/blob/wire/protocol/network/wire.md
• write go-multistream and go-multistream-select (may be just one pkg) (@whyrusleeping)
• write node-multistream and node-multistream-select (may be just one pkg)
• make go-ipfs use go-multistream + go-multistream-select instead of the muxer thing it has now.
• make go-ipfs use spdy instead of yamux
• make a build of go-ipfs without secio encryption (for simplicity for now)
• make a node-ipfs program that uses node-multistream and node-multistream-select
• make the node-ipfs program use spdy
• talk to each other!

(could actually talk without spdy first)

[jbenet]

@whyrusleeping have time this week to take on any of the go things above? (part the spdy part?)

[jbenet]

@diasdavid @mappum @krl @travisperson -- what things from the wire interop list might you be able to take this week?

if we had a thing by the end of the week that spoke the wire protocol, it would be a great start.

[jbenet]

(and anyone else watching, feel free to help! \o/)

[daviddias]

After thesis delivery on Friday morning, my intent is to to focus entirely on:

• write node-multistream and node-multistream-select (may be just one pkg)
• work with someone on "write go-multistream and go-multistream-select (may be just one pkg)" and write tests to test both impl interop (I'm not very go lang savy yet, but guaranteeing that the interop worked from the start can save us a lot of time)
• depending of how this goes and how much time it consumes, also start looking at the DHT stuff

I'll be online, so anyone that wants to work at the same time in this stuff, just let me know

[mafintosh]

@jbenet @diasdavid in multistream, is the newline included in the length? (if so i more or less already have that module done)

[whyrusleeping]

@jbenet i was going to work on the records ipns stuff, but i can do this instead.

[jbenet]

@mafintosh yes, the newline is included in the length. what's your module?

[whyrusleeping]

ill take on "write go-multistream and go-multistream-select (may be just one pkg)"