Skip to content

Commit d4f4ef1

Browse files
authored
Merge pull request #287 from ipfs/reframe-caching
reframe: cacheable GET endpoint for HTTP transport
2 parents ab34fc9 + 8833b66 commit d4f4ef1

File tree

4 files changed

+230
-62
lines changed

4 files changed

+230
-62
lines changed

reframe/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Reframe Specifications
2+
3+
- [`REFRAME_PROTOCOL.md`](./REFRAME_PROTOCOL.md) ← start here
4+
- [`REFRAME_KNOWN_METHODS.md`](./REFRAME_KNOWN_METHODS.md)
5+
- [`REFRAME_HTTP_TRANSPORT.md`](./REFRAME_HTTP_TRANSPORT.md)

reframe/REFRAME_HTTP_TRANSPORT.md

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Reframe: HTTP Transport
2+
3+
**Author(s)**:
4+
- Adin Schmahmann
5+
- Petar Maymounkov
6+
- Marcin Rataj
7+
8+
**Maintainer(s)**:
9+
10+
* * *
11+
12+
**Abstract**
13+
14+
The Reframe over HTTP protocol is defining the transport and message
15+
serialization mechanisms for sending Reframe messages over HTTP `POST` and
16+
`GET`, and provides guidance for implementers around HTTP caching.
17+
18+
# Organization of this document
19+
20+
- [HTTP Transport Design](#http-transport-design)
21+
- [HTTP Caching Considerations](#http-caching-considerations)
22+
- [POST vs GET](#post-vs-get)
23+
- [Avoiding sending the same response messages twice](#avoiding-sending-the-same-response-messages-twice)
24+
- [Client controls for time-based caching](#client-controls-for-time-based-caching)
25+
- [Rate-limiting non-cachable POST requests](#rate-limiting-non-cachable-post-requests)
26+
- [Implementations](#implementations)
27+
28+
# HTTP Transport Design
29+
30+
All messages sent in HTTP body MUST be encoded as DAG-JSON and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=1`
31+
32+
Requests MUST be sent as either:
33+
- `GET /reframe/{mbase64url-dag-cbor}`
34+
- Cachable HTTP `GET` requests with message passed as DAG-CBOR in HTTP path segment, encoded as URL-safe [`base64url` multibase](https://docs.ipfs.io/concepts/glossary/#base64url) string
35+
- DAG-CBOR in multibase `base64url` is used instead of DAG-JSON because JSON may include characters that are not safe to be used in URLs, and re-encoding JSON in base would take too much space
36+
- Suitable for sharing links, sending smaller messages, and when a query result MUST benefit from HTTP caching (see _HTTP Caching Considerations_ below).
37+
- `POST /reframe`
38+
- Ephemeral HTTP `POST` request with message passed as DAG-JSON in HTTP request body
39+
- Suitable for bigger messages, and when HTTP caching should be skipped for the most fresh results
40+
41+
Servers MUST support `GET` for methods marked as cachable and MUST support `POST` for all methods (both cachable and not-cachable). This allows servers to rate-limit `POST` when cachable `GET` could be used instead, and enables clients to use `POST` as a fallback in case there is a technical problem with bigger Reframe messages not fitting in a `GET` URL. See "Caching Considerations" section.
42+
43+
44+
If a server supports HTTP/1.1, then it MAY send chunked-encoded messages. Clients supporting HTTP/1.1 MUST accept chunked-encoded responses.
45+
46+
Requests and Responses MUST occur over a single HTTP call instead of the server being allowed to dial back the client with a response at a later time.
47+
48+
If a server chooses to respond to a single request message with a group of messages in the response it should do so as a set of `\n` delimited DAG-JSON messages (i.e. `{Response1}\n{Response2}...`).
49+
50+
Requests and responses MUST come with `version=1` as a _Required Parameter_ in the `Accept` and `Content-Type` HTTP headers.
51+
52+
Note: This version header is what allows the transport to more easily evolve over time (e.g. if it was desired to change the transport to support other encodings than DAG-JSON, utilize headers differently, move the request data from the body, etc.). Not including the version number is may lead to incompatibility with future versions of the transport.
53+
54+
## HTTP Caching Considerations
55+
56+
### POST vs GET
57+
58+
HTTP `POST` requests do not benefit from any preexisting HTTP caching because
59+
every `POST` response will overwrite the cached resource.
60+
61+
While it is possible to write custom middleware to cache `POST` responses based on
62+
request body, this is not a standard behavior and is discouraged.
63+
64+
Use of `GET` endpoint is not mandatory, but suggested if a Reframe deployment
65+
expects to handle the same message query multiple times, and want to leverage
66+
existing HTTP tooling to maximize HTTP cache hits.
67+
68+
### Avoiding sending the same response messages twice
69+
70+
Implementations MUST always return strong
71+
[`Etag`](https://httpwg.org/specs/rfc7232.html#header.etag) HTTP header based
72+
on digest of DAG-JSON response messages. This allows clients to send
73+
inexpensive conditional requests with
74+
[`If-None-Match`](https://httpwg.org/specs/rfc7232.html#header.if-none-match)
75+
header, which will skip when the response message did not change.
76+
77+
### Client controls for time-based caching
78+
79+
Implementations can also return (optional)
80+
[`Last-Modified`](https://httpwg.org/specs/rfc7232.html#header.last-modified)
81+
HTTP header, allowing clients to send conditional requests with
82+
[`If-Modified-Since`](https://httpwg.org/specs/rfc7232.html#header.if-modified-since)
83+
header to specify their acceptance for stale (cached) responses.
84+
85+
### Rate-limiting non-cachable POST requests
86+
87+
HTTP endpoint can return status code
88+
[429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585#section-4)
89+
with `Retry-After` header to throttle the number of `POST` requests a client can send.
90+
91+
The body returned with `429` response should suggest use of HTTP `GET` endpoint
92+
for cachable Reframe methods:
93+
94+
```
95+
HTTP/1.1 429 Too Many Requests
96+
Content-Type: text/plain
97+
Retry-After: 3600
98+
99+
too many POST requests: consider switching to cachable GET or try again later (see Retry-After header)
100+
```
101+
102+
103+
# Implementations
104+
105+
https://github.com/ipfs/go-delegated-routing

REFRAME.md renamed to reframe/REFRAME_KNOWN_METHODS.md

Lines changed: 43 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Reframe
1+
# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Reframe: Known Methods
22

33
**Author(s)**:
44
- Adin Schmahmann
@@ -10,63 +10,26 @@
1010

1111
**Abstract**
1212

13-
The Reframe protocol is designed for request-response messages that is sufficiently generic and extensible to evolve over time as new needs for it arise. This includes separately defining the transport and message serialization mechanisms from the actual method types and semantics.
14-
15-
The initial use case motivating the protocol's design is to help peers discover various routing hints that enable them to find the content or service they are actually looking for. The extensibility required in this case is that various routing systems could share the protocol so that applications can work on them generically and that recursive routing systems could choose to use the protocol themselves to maximize interoperability.
13+
This document is defining known methods (request-response message types) and semantics.
1614

1715
# Organization of this document
1816

19-
- [Introduction](#introduction)
20-
- [Spec](#spec)
21-
- [Intended Upgrade Paths](#intended-upgrade-paths)
17+
- [Known Methods](#known-methods)
18+
- [Cachable/Non-Cachable Methods](#cachablenon-cachable-methods)
19+
- [Message Specs](#message-specs)
20+
- [Error](#error)
21+
- [Identify](#identify)
22+
- [DAG-JSON Examples](#dag-json-examples)
23+
- [FindProviders](#findproviders)
24+
- [DAG-JSON Examples](#dag-json-examples-1)
25+
- [GetIPNS](#getipns)
26+
- [DAG-JSON Examples](#dag-json-examples-2)
27+
- [PutIPNS](#putipns)
28+
- [DAG-JSON Examples](#dag-json-examples-3)
29+
- [Method Upgrade Paths](#method-upgrade-paths)
2230
- [Implementations](#implementations)
2331

24-
# Introduction
25-
26-
The Reframe protocol is a request-response based protocol. Upon receiving a request a Reframe server should respond to clients with information they have pertaining to the request. It's possible that a Reframe server may not have all of the information necessary for a client to actually get the data they need, but even partial resolution is acceptable. For example, if a peer is looking to download some block of data a Reframe server may only know about some peers that have the data but not their network addresses. That's ok and the client can always choose alternative mechanisms to discover the missing information.
27-
28-
# Spec
29-
30-
To build an implementation of the protocol it is required to both define a transport for the messages and the method types supported by the given implementation. Over time both the method and transport types may grow and also be included in this specification.
31-
32-
## Interaction Pattern
33-
34-
Given that a client C wants to request information from some server S:
35-
36-
1. C opens sends a request to S that is a single message
37-
2. S will respond to C with either a single message or a group of messages
38-
39-
## Transports
40-
41-
### HTTP + DAG-JSON
42-
43-
All messages MUST be encoded as DAG-JSON and use explicit content type `application/vnd.ipfs.rpc+dag-json; version=1`
44-
45-
46-
Requests MUST be sent as HTTP POST requests. If a server supports HTTP/1.1, then it MAY send chunked-encoded messages. Clients supporting HTTP/1.1 MUST accept chunked-encoded responses.
47-
48-
Requests and Responses MUST occur over a single HTTP call instead of the server being allowed to dial back the client with a response at a later time.
49-
50-
If a server chooses to respond to a single request message with a group of messages in the response it should do so as a set of `\n` delimited DAG-JSON messages (i.e. `{Response1}\n{Response2}...`).
51-
52-
Requests and responses MUST come with `version=1` as a _Required Parameter_ in the `Accept` and `Content-Type` HTTP headers.
53-
54-
55-
Note: This version header is what allows the transport to more easily evolve over time (e.g. if it was desired to change the transport to support other encodings than DAG-JSON, utilize headers differently, move the request data from the body, etc.). Not including the version number is may lead to incompatibility with future versions of the transport.
56-
57-
## Protocol Message Overview
58-
59-
We can represent each message as an IPLD Schema to denote its abstract representation independent of the serialization scheme.
60-
61-
To help visualize example messages and illustrate implementation of the first concrete transport (HTTP + DAG-JSON) example messages will be given in DAG-JSON.
62-
63-
The payload sent as a request is a single message. The response is a repeated set of messages until the transport signals that the response is completed (e.g. closing a stream, connection, etc.)
64-
65-
Reception, on the server or client side, of:
66-
1. New or unknown fields in any request or response message are ignored (rather than causing an error)
67-
2. Missing fields or unknown union cases cause a terminal error
68-
69-
### Known Methods
32+
# Known Methods
7033

7134
The known Request types are the following and are described below:
7235

@@ -94,7 +57,25 @@ type Response union {
9457
Note: Each Request type has a corresponding Response type.
9558
Every message except the Error type should end in Request/Response.
9659

97-
#### Error
60+
## Cachable/Non-Cachable Methods
61+
62+
The following methods (request-response pairs) are _cachable_:
63+
64+
```ipldsch
65+
type CachableRequest union {
66+
| "IdentifyRequest" IdentifyRequest
67+
| "FindProvidersRequest" FindProvidersRequest
68+
| "GetIPNSRequest" GetIPNSRequest
69+
}
70+
```
71+
72+
Methods that are not listed above are considered _non-cachable_.
73+
74+
Implementations are encouraged to improve performance of `CachableRequest` methods by applying transport and method-specific caching strategies.
75+
76+
# Message Specs
77+
78+
## Error
9879

9980
The Error message type should be used in Responses to indicate that an error has occurred.
10081

@@ -104,7 +85,7 @@ The Error message type should be used in Responses to indicate that an error has
10485
}
10586
```
10687

107-
#### Identify
88+
## Identify
10889

10990
A message for discovering which messages a server supports. May be used by applications to optimize which servers get sent which types of requests.
11091

@@ -118,7 +99,7 @@ A message for discovering which messages a server supports. May be used by appli
11899

119100
`IdentifyResponse` should return the set of supported methods aside from the "Identify" method.
120101

121-
##### DAG-JSON Examples
102+
### DAG-JSON Examples
122103

123104
Request:
124105
```
@@ -132,7 +113,7 @@ Response:
132113
}}
133114
```
134115

135-
#### FindProviders
116+
## FindProviders
136117

137118
A message for finding nodes that have an interest in a given key. Some common examples include finding which peers have advertised that they have a given CID, or which peers are interested in a given pubsub topic.
138119

@@ -185,7 +166,7 @@ Note: While the Key is a CID it is highly recommended that server implementation
185166
}
186167
```
187168

188-
##### DAG-JSON Examples
169+
### DAG-JSON Examples
189170

190171
Request:
191172
```
@@ -218,7 +199,7 @@ Response:
218199
}}
219200
```
220201

221-
#### GetIPNS
202+
## GetIPNS
222203

223204
A message for finding the latest IPNS records for a given identifier.
224205

@@ -234,7 +215,7 @@ A message for finding the latest IPNS records for a given identifier.
234215
}
235216
```
236217

237-
##### DAG-JSON Examples
218+
### DAG-JSON Examples
238219

239220
Request:
240221
```
@@ -252,7 +233,7 @@ Response:
252233
}}...
253234
```
254235

255-
#### PutIPNS
236+
## PutIPNS
256237

257238
A message for putting the latest IPNS records for a given identifier.
258239

@@ -265,7 +246,7 @@ A message for putting the latest IPNS records for a given identifier.
265246
type PutIPNSResponse struct {}
266247
```
267248

268-
##### DAG-JSON Examples
249+
### DAG-JSON Examples
269250

270251
Request:
271252
```

reframe/REFRAME_PROTOCOL.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) Reframe
2+
3+
**Author(s)**:
4+
- Adin Schmahmann
5+
- Petar Maymounkov
6+
7+
**Maintainer(s)**:
8+
9+
* * *
10+
11+
**Abstract**
12+
13+
The Reframe protocol is designed for request-response messages that is sufficiently generic and extensible to evolve over time as new needs for it arise. This includes separately defining the transport and message serialization mechanisms from the actual method types and semantics.
14+
15+
The initial use case motivating the protocol's design is to help peers discover various routing hints that enable them to find the content or service they are actually looking for. The extensibility required in this case is that various routing systems could share the protocol so that applications can work on them generically and that recursive routing systems could choose to use the protocol themselves to maximize interoperability.
16+
17+
# Organization of this document
18+
19+
- [Introduction](#introduction)
20+
- [Spec](#spec)
21+
- [Interaction Pattern](#interaction-pattern)
22+
- [Cachability](#cachability)
23+
- [Transports](#transports)
24+
- [Protocol Message Overview](#protocol-message-overview)
25+
- [Known Methods](#known-methods)
26+
- [Method Upgrade Paths](#method-upgrade-paths)
27+
- [Implementations](#implementations)
28+
29+
# Introduction
30+
31+
The Reframe protocol is a request-response based protocol. Upon receiving a request a Reframe server should respond to clients with information they have pertaining to the request. It's possible that a Reframe server may not have all of the information necessary for a client to actually get the data they need, but even partial resolution is acceptable. For example, if a peer is looking to download some block of data a Reframe server may only know about some peers that have the data but not their network addresses. That's ok and the client can always choose alternative mechanisms to discover the missing information.
32+
33+
# Spec
34+
35+
To build an implementation of the protocol it is required to both define a transport for the messages and the method types supported by the given implementation. Over time both the method and transport types may grow and also be included in this specification.
36+
37+
## Interaction Pattern
38+
39+
Given that a client `C` wants to request information from some server `S`:
40+
41+
1. `C` opens sends a request to `S` that is a single message
42+
2. `S` will respond to `C` with either a single message or a group of messages
43+
44+
## Cachability
45+
46+
Some methods are prone to being cachable while others are not. Methods are tagged within the spec as cachable/not cachable. Transports may leverage that information to decide if cachable/not cachable methods should be treated differently.
47+
48+
## Transports
49+
50+
Reframe is designed to be transport-agnostic, multiple transports are expected
51+
to exist in the future.
52+
53+
Available transport specifications:
54+
55+
- [`REFRAME_HTTP_TRANSPORT`](./REFRAME_HTTP_TRANSPORT.md)
56+
57+
## Protocol Message Overview
58+
59+
We can represent each message as an IPLD Schema to denote its abstract representation independent of the serialization scheme.
60+
61+
To help visualize example messages and illustrate implementation of the first concrete transport (HTTP + DAG-JSON) example messages will be given in DAG-JSON.
62+
63+
The payload sent as a request is a single message. The response is a repeated set of messages until the transport signals that the response is completed (e.g. closing a stream, connection, etc.)
64+
65+
Reception, on the server or client side, of:
66+
1. New or unknown fields in any request or response message are ignored (rather than causing an error)
67+
2. Missing fields or unknown union cases cause a terminal error
68+
69+
### Known Methods
70+
71+
The known Request types are described in:
72+
73+
- [`REFRAME_KNOWN_METHODS.md`](./REFRAME_KNOWN_METHODS.md)
74+
75+
# Implementations
76+
77+
https://github.com/ipfs/go-delegated-routing

0 commit comments

Comments
 (0)