The Fetch protocol is used for performing a direct peer-to-peer key-value lookup
Lifecycle Stage | Maturity | Status | Latest Revision |
---|---|---|---|
1A | Working Draft | Active | r1, 2019-08-13 |
Authors: @aschmahmann
Interest Group: [Insert Your Name Here]
See the lifecycle document for context about the maturity level and spec status.
The fetch protocol is a means for a libp2p node to get data from another based on a given key. It generally fulfills the contract:
Fetch(key) (value, statusCode)
Currently there is no standard protocol for doing simple data retrieval. Sometimes a small data retrieval protocol is all that's needed to get the job done. This is especially true when it comes to augmenting or layering protocols together.
This specification is helpful because the protocol is already part of a proposal to add a persistence layer on top of PubSub.
There have been numerous discussion revolving around how to allow PubSub to be persistent, some of which go back multiple years (more background information is available here). Initial suggestions included modifying or leveraging the existing PubSub spec and implementations. However, it was decided that PubSub should remain agnostic to any sort of persistence, and that persistence should be layered on top of PubSub as a separate protocol.
There is currently an IPNS-over-PubSub spec PR that describes how the Fetch protocol can be used to create a persistence layer on top of PubSub and therefore greatly improve IPNS performance.
This is one small protocol that as of right now is only being used to improve the go-libp2p-pubsub-router. This protocol should not require any one who doesn't want to use it to change anything about their current use.
- A persistent pubsub protocol easily built on top of Fetch
- A reusable fetch protocol that can be used to augment various other protocols (e.g. add the ability to directly ask peers for data to a DHT)
- Might be abused/overused given the simple nature of the protocol
Should support: Fetch(key) (value, statusCode)
However, the level of specificity in the types of the above variables has wiggle room if people are interested.
The go-libp2p-pubsub-router
implementation requires:
key
: At least as generic as a UTF-8 string
value
: At least as generic as a byte array
statusCode
: Supports at least OK
, NOT_FOUND
, and ERROR
The libp2p protocol ID for this protocol is /libp2p/fetch/0.0.1
The messages in the Fetch protocol use on of the following protobufs (proto3 syntax):
syntax = "proto2";
message FetchRequest {
string identifier = 1;
}
message FetchResponse {
StatusCode status = 1;
enum StatusCode {
OK = 0;
NOT_FOUND = 1;
ERROR = 2;
}
bytes data = 2;
}
Setup:
- Peers involved,
A
,B
A
wants to fetch data fromB
corresponding to the keyk
Assumptions:
A
has connection toB
Events:
A
sends aRequestLatest{Identifier : k}
message toB
B
does some internal lookup and responds with aRespondLatest
message- If
B
finds (and elects to send) some datav
toA
it sendsRespondLatest{status: SUCCESS, data: v}
- If
B
has no data to send it responds withRespondLatest{status: NOT_FOUND, data: null}
A
ignores any information in thedata
field if the status isNOT_FOUND
- If
Note: If at any time A
or B
break from the protocol in some way, either by disconnecting/closing streams or by sending
invalid data there is no guarantee on the behavior from the other party.
Would love this protocol to work over a packet instead of a stream oriented transport since it's very short lived and will have cases where the payloads are tiny.
It could also be useful to have an aggregate version of this protocol where instead of fetching key-value pairs we fetch sets of key-value pairs. This will be more efficient, much more when using a stream oriented transport, than running fetch multiple times.