Implement Gossipsub for sharding p2p

[jamesray1]

Go PR: libp2p/go-libp2p-pubsub#67

I figure I'll implement gossipsub for Rust rather than wait until a go-libp2p daemon is implemented, which wouldn't be optimal anyway.

Note that gossipsub would not only be useful for sharding p2p, but general p2p networks.

Ignore the rest of this post, due to tomaka's comment below.

Only the differences to Floodsub are included below.

To implement with [Rust-libp2p]:

• comm: rpcWithControl: uses ControlIHave, ControlIWant, ControlGraft, and ControlPrune messages,
• which are added to pb,
• which can be converted from rpc.proto (already implemented) to rpc.rs (therefore do these in reverse order)
• floodsub: Join and Leave methods for FloodSubRouter
• floodsub_test: refactor sparseConnect into connectSome with an input for number of hosts, and call it with 3 hosts, as wel as call connectSome from new denseConnect with 10 hosts
• gossipsub
  • imports context, math/rand, time, pb, host and protoocol. Will need packages that do what these packages provide. Asked here.
  • GossipSubID constant (new peer ID) from host
  • vars
  • NewGossipSub
  • GossipSubRouter: core part, has lots of methods. Maintaining a list of peers for both publishing and subscribing may have centralization risks by becoming dependent on certain peers, especially if many others are also dependent on them. Ideally we want to shuffle the list of peers; or have some other way to prevent a tendency towards centralization, or depending on a handful of peers. In the struct it includes mcache, so implement that first.
    • Protocols
    • Attach: uses heartbeatTimer
    • AddPeer
    • RemovePeer
    • HandleRPC: requires GetControl
    • All of the above, plus Publish, Join and Leave (the latter two are new addtions) require PubSubRouter in pubsub below
    • handleIHave
    • handleIWant
    • handleGraft
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
    • [ ]
  • mcache + tests
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
• pb/rpc.pb.go
  • Only this line is modified in the RPC struct: Control *ControlMessage protobuf:"bytes,3,opt,name=control" json:"control,omitempty"``
    - [ ] GetControl (needs ControlMessage)
  • ControlMessage
    • ControlIHave
    • ControlIWant
    • ControlGraft
    • ControlPrune
    • Methods:
      • Reset
      • String
      • ProtoMessage
      • GetIhave
      • GetIwant
      • GetGraft
      • GetPrune
  • ControlIHave
    - [ ] Reset
    - [ ] String
    - [ ] ProtoMessage
    - [ ] GetTopicID
    - [ ] GetMessageIDs
  • ControlIWant struct + methods:
    - [ ] Reset
    - [ ] String
    - [ ] ProtoMessage
    - [ ] GetMessageIDs
  • ControlGraft
    - [ ] Reset
    - [ ] String
    - [ ] ProtoMessage
    - [ ] GetTopicID
  • ControlPrune
    - [ ] Reset
    - [ ] String
    - [ ] ProtoMessage
    - [ ] GetTopicID
  • Additions to TopicDescriptor struct:
    - [ ] proto.RegisterType((*ControlMessage)(nil), "floodsub.pb.ControlMessage")
    - [ ] proto.RegisterType((*ControlIHave)(nil), "floodsub.pb.ControlIHave")
    - [ ] proto.RegisterType((*ControlIWant)(nil), "floodsub.pb.ControlIWant")
    - [ ] proto.RegisterType((*ControlGraft)(nil), "floodsub.pb.ControlGraft")
    - [ ] proto.RegisterType((*ControlPrune)(nil), "floodsub.pb.ControlPrune")
  • rpc.proto:
    - [ ] at the end of message RPC: optional ControlMessage control = 3;
    - [ ] message ControlMessage {
    - [ ] message ControlIHave {
    - [ ] message ControlIWant
    - [ ] message ControlGraft
    - [ ] message ControlPrune
  • pubsub
    - [ ] rand import (use the rand crate)
    - [ ] eval channel
    - [ ] PubSubRouter
    - [ ] Adds Join and Leave plus comments
    - [ ] NewPubSub (renamed from NewFloodSub, probably correlates to FloodSubController.new with Rust. Rust appears to uses inner instead of context,
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
    - [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
  • [ ]
• [ ]
• more to be added from the above PR.

Is your feature request related to a problem? Please describe.
Floodsub is already implemented for rust-libp2p, however it isn't really suitable for a sharding p2p network, as it involves broadcasting a message to all available peers, which is very bandwidth intensive.

Describe the solution you'd like
Gossipsub is the next development by Protocol labs and the lib-p2p team in developing PubSub. More details about it are in the above PR, as well as here

Describe alternatives you've considered

Additional context

Cute Animal Picture

[tomaka]

Note that a direct translation from Go code to Rust is a very very bad idea due to the profound differences between the languages.

Compatibility between the implementations should exist at the wire level.
There is almost nothing in common between the source codes of rust-libp2p and go-libp2p.

[jamesray1]

Right, thanks for correcting me, should've realized that I was on the wrong track!