Skip to content

Commit 37935bc

Browse files
salmad3salmad3
authored
feat: peer discovery and routing section (#294)
1 parent 663a37d commit 37935bc

File tree

6 files changed

+246
-30
lines changed

6 files changed

+246
-30
lines changed

‎content/concepts/discovery-routing/_index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
title : "Peer Discovery and Routing"
3+
description: "Peer discovery and routing protocols are used to discover and announce services to other peers."
4+
weight: 7
5+
---

‎content/concepts/discovery-routing/kaddht.md

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
title: "Kademlia DHT"
3+
description: "The libp2p Kad-DHT subsystem is an implementation of the Kademlia
4+
DHT, a distributed hash table."
5+
weight: 226
6+
---
7+
8+
## Overview
9+
10+
The Kademlia Distributed Hash Table (DHT), or Kad-DHT, is a distributed hash table
11+
that is designed for P2P networks.
12+
13+
Kad-DHT in libp2p is a subsystem based on the
14+
[Kademlia whitepaper](https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf).
15+
16+
Kad-DHT offers a way to find nodes and data on the network by using a
17+
[routing table](https://en.wikipedia.org/wiki/Routing_table) that organizes peers based
18+
on how similar their keys are.
19+
20+
<details>
21+
<summary>A deeper look</summary>
22+
23+
The routing table is organized based on a prefix length and a distance metric.
24+
The prefix length helps to group similar keys, and the distance metric helps to
25+
find the closest peers to a specific key in the routing table. The table maintains
26+
a list of `k` closest peers for each possible prefix length between `0` and `L-1`,
27+
where `L` is the length of the keyspace, determined by the length of the hash
28+
function used. **Kad-DHT uses SHA-256**, with a keyspace of 256 bits, trying to maintain
29+
`k` peers with a shared key prefix for every prefix length between `0` and `255` in
30+
its routing table.
31+
32+
The prefix length measures the proximity of two keys in the routing table and
33+
divides the keyspace into smaller subspaces, called "buckets", each containing nodes
34+
that share a common prefix of bits in their SHA-256 hash. The prefix length is the
35+
number of bits that are the same in the two keys' SHA-256 hash. The more leading bits
36+
that are the same, the longer the prefix length and the closer the proximity of the
37+
two keys are considered to be.
38+
39+
The distance metric is a way to calculate the distance between two keys by
40+
taking the bitwise exclusive-or (XOR) of the SHA-256 hash of the two keys. The
41+
result is a measure of the distance between the two keys, where a distance of
42+
`0` means the keys are identical, and a distance of `1` means that only one
43+
bit is different, meaning the two keys are close to each other (i.e. their
44+
SHA-256 hashes are similar).
45+
46+
This design allows for efficient and effective lookups in the routing table when
47+
trying to find nodes or data that share similar prefixes.
48+
49+
</details>
50+
51+
## Peer routing
52+
53+
The Kad-DHT uses a process called "peer routing" to discover nodes in the
54+
network. When looking for a peer, the local node contacts the `k` closest nodes to
55+
the remote peer's ID asking them for closer nodes. The local node repeats the
56+
process until it finds the peer or determines that it is not in the network.
57+
58+
## Content provider routing
59+
60+
Kad-DHT also includes a feature for content provider discovery, where nodes can
61+
look up providers for a given key. The local node again contacts the `k` closest
62+
nodes to the key asking them for either providers of the key and/or closer nodes
63+
to the key. The local node repeats the process until it finds providers for the
64+
key or determines that it is not in the network.
65+
66+
## Bootstrap process
67+
68+
To maintain a healthy routing table and discover new nodes, the Kad-DHT includes
69+
a bootstrap process that runs periodically. The process starts by generating a random peer
70+
ID and looking it up via the peer routing process. The node then adds the closest peers it
71+
discovers to its routing table and repeats the process multiple times. This process also
72+
includes looking up its own peer ID to improve awareness of nodes close to itself.
73+
74+
{{< alert icon="💡" context="note" text="See the Kademlia DHT <a class=\"text-muted\" href=\"https://github.com/libp2p/specs/tree/master/kad-dht\">technical specification</a> for more details." />}}

‎content/concepts/discovery-routing/mDNS.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
title: "mDNS"
3+
description: "mDNS uses a multicast system of DNS records over a local network to enable peer discovery."
4+
weight: 224
5+
---
6+
7+
## What is mDNS?
8+
9+
mDNS, or multicast Domain Name System, is a way for nodes to use IP multicast to
10+
publish and receive DNS records [RFC
11+
6762](https://www.rfc-editor.org/rfc/rfc6762) within a local network. Nodes
12+
broadcast topics they're interested in. mDNS is commonly used on home networks
13+
to allow devices such as computers, printers, and smart TVs to discover each
14+
other and connect.
15+
16+
## mDNS in libp2p
17+
18+
In libp2p, mDNS is used for peer discovery, allowing peers to find each other on
19+
the same local network without any configuration. In the basic mDNS node
20+
discovery flow a node broadcasts a request which is consecutively replied to by
21+
other nodes within the network with their multiaddresses.
22+
23+
To learn more about
24+
definitions, specific fields, and peer discovery, [visit the mDNS libp2p
25+
specification](https://github.com/libp2p/specs/blob/master/discovery/mdns.md).
26+
<!-- ADD DIAGRAM -->

‎content/concepts/discovery-routing/overview.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
title: "What is Discovery & Routing"
3+
description: "Peer discovery and routing protocols are used to discover and announce services to other peers and find a peer's location, respectively."
4+
weight: 221
5+
---
6+
7+
## Overview
8+
9+
Peer discovery and routing are two essential aspects of P2P networking. In a P2P network,
10+
each node must be able to discover and communicate with other nodes without the need for
11+
a central server.
12+
13+
### Peer discovery
14+
15+
Peer discovery is the process of finding and announcing services to other available
16+
peers in a P2P network. Peer discovery can be done using various protocols, such
17+
as broadcasting a message to all peers in the network or using a bootstrap node to
18+
provide a list of known peers.
19+
20+
### Peer routing
21+
22+
Peer routing, on the other hand, refers to finding a specific peer's location
23+
in the network. This is typically done by maintaining a routing table or a similar
24+
data structure that keeps track of the network topology.
25+
26+
Different algorithms can be used to find the "closest" neighboring peers to a given peer ID.
27+
A peer may use a routing algorithm to find the location of a specific peer and then
28+
use that information to discover new peers in the vicinity. Additionally, a peer may
29+
use both peer routing and peer discovery mechanisms in parallel to find new peers and
30+
route data to them.
31+
32+
{{< alert icon="" context="note">}}
33+
In practice, the distinction between peer routing and peer
34+
discovery is not always clear-cut, and it's worth noting that in a real-world
35+
implementation, discovery and routing usually happen concurrently.
36+
{{< /alert >}}
37+
38+
## Discovery and routing in libp2p
39+
40+
libp2p provides a set of modules for different network-level functionality,
41+
including peer discovery and routing. Peers in libp2p can discover other
42+
peers using various mechanisms, such as exchanging peer
43+
[multiaddresses](./../../fundamentals/addressing) over the
44+
network, querying a directory service, or using a distributed hash table (DHT)
45+
to store and retrieve information about available peers.
46+
47+
These methods include, but are not limited to:
48+
49+
- [Rendezvous](./../rendezvous): a protocol that allows peers to exchange peer multiaddresses
50+
in a secure and private manner.
51+
- [mDNS](./../mdns): a multicast Domain Name System (DNS) protocol that allows peers to
52+
discover other peers on the local network.
53+
- [DHT](./../kaddht): Distributed Hash Table, libp2p uses a DHT called Kademlia, it assigns
54+
each piece of content a unique identifier and stores the content on the peer whose
55+
identifier is closest to the content's identifier.

‎content/concepts/discovery-routing/rendezvous.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
title : "Rendezvous"
3+
description: "The rendezvous protocol can facilitate the routing and discovery of nodes in a P2P network using a common location."
4+
weight: 223
5+
---
6+
7+
## What is Rendezvous?
8+
9+
A rendezvous protocol is a routing protocol that enables nodes and resources
10+
in a P2P network to discover each other. Rendezvous is used
11+
as a common location (point) to route between two routes.
12+
13+
Rendezvous points are typically nodes that are well-connected and stable in
14+
a network and can handle large amounts of traffic and data. They
15+
serve as a hub for nodes to discover.
16+
17+
<details>
18+
<summary>Rendezvous is not decentralized</summary>
19+
20+
It is important to note that Rendezvous is not decentralized but rather
21+
federated. While this has its use cases, it also introduces a single
22+
point of failure into the network. This can be contrasted with fully decentralized
23+
solutions like DHT and Gossipsub. DHT (Distributed Hash Table) and Gossipsub are
24+
decentralized alternatives to Rendezvous.
25+
26+
[DHT](kaddht.md) is a distributed network protocol used to store and
27+
retrieve data in a P2P network efficiently. It is like a hash table mapping keys
28+
to values, allowing for fast lookups and efficient data distribution across the network.
29+
30+
[Gossipsub](pubsub.md), on the other hand, is a pub-sub (publish-subscribe) protocol
31+
that is used to distribute messages and data across a network. It uses a gossip-based
32+
mechanism to propagate messages throughout the network, allowing fast and efficient
33+
distribution without relying on a central control point.
34+
35+
</details>
36+
37+
## Rendezvous in libp2p
38+
39+
{{< alert icon="💡" context="info" text="The current rendezvous implementation replaces the initial ws-star-rendezvous implementation with rendezvous daemons and a fleet of p2p-circuit relays." />}}
40+
41+
The libp2p rendezvous protocol can be used for different use cases. E.g. it can
42+
be used during bootstrap to discover circuit relays that provide connectivity
43+
for browser nodes. Generally, a peer can use known rendezvous points to find
44+
peers that provide network services. Rendezvous is also used throughout the
45+
lifetime of an application for peer discovery by registering and polling
46+
rendezvous points. In an application-specific setting, rendezvous points can be
47+
used to progressively discover peers that can answer specific queries or host
48+
shards of content.
49+
50+
The libp2p rendezvous protocol allows peers to connect to a rendezvous point and
51+
register their presence by sending a `REGISTER` message in one or more
52+
namespaces. Any node implementing the rendezvous protocol can act as a
53+
rendezvous point, and any peer can connect to a rendezvous point. However, only
54+
peers initiating a registration can register themselves at a rendezvous point.
55+
56+
By registering with a rendezvous point, peers allow for their discovery by other peers who
57+
query the rendezvous point. The query may:
58+
59+
- provide namespace(s), such as `test-app`;
60+
- optionally provide a maximum number of peers to return;
61+
- can include a cookie that is obtained from the response to a previous query,
62+
thus the current query only contain registrations that weren't part of the
63+
previous response.
64+
> This simplifies discovery as it reduces the overhead of queried peers and allows for
65+
> the pagination of query responses.
66+
67+
There is a default peer registration lifetime of 2 hours. Peers can optionally specify the
68+
lifetime using a TTL parameter in the `REGISTER` message, with an upper bound of 72 hours.
69+
70+
The rendezvous protocol runs over libp2p streams using the protocol ID `/rendezvous/1.0.0`.
71+
72+
<!-- TO ADD: Interaction diagrams and context -->
73+
74+
### Rendezvous and publish-subscribe
75+
76+
For effective discovery, rendezvous can be combined with [libp2p publish/subscribe](../messaging/pubsub/overview).
77+
At a basic level, rendezvous can bootstrap pubsub by discovering peers subscribed to a topic. The rendezvous would
78+
be responsible for publishing packets, subscribing, or unsubscribing from packet shapes.
79+
80+
Pubsub can also be used as a mechanism for building rendezvous services, where a number
81+
of rendezvous points can federate using pubsub for internal distribution while still
82+
providing a simple interface to clients.
83+
84+
<!-- DIAGRAMS COMING SOON -->
85+
86+
{{< alert icon="💡" context="note" text="See the rendezvous <a class=\"text-muted\" href=\"https://github.com/libp2p/specs/blob/master/rendezvous/README.md\">technical specification</a> for more details." />}}

‎content/concepts/discovery/overview.md

Lines changed: 0 additions & 30 deletions
This file was deleted.

0 commit comments

Comments
 (0)