Add HTTP Gateway Specs #283
Conversation
This adds gateway specs under ./http-gateways directory. The aim is to document _current_ behavior (implementation in go-ipfs 0.13) and switch the way we do the gateway work to be specs-driven. Long term goal is to provide language and implementation agnostic specification that anyone can use to implement compatible gateways.
http-gateways/DNSLINK_GATEWAY.md
Outdated
| Defines the DNSLink name to resolve into `/ipfs/{cid}/` prefix that should be | ||
| prepended to the `path` before the final IPFS content path resolution is | ||
| performed. |
There was a problem hiding this comment.
Should we define that:
- a dnslink that points to a
/ipnspath must be resolved to an immutable path first - a dnslink with a path component should be resolved to it's target CID
...path before appending the request path
| - `Cache-Control: no-cache, no-transform` should be returned with | ||
| `application/vnd.ipld.car` responses if the block order in CAR stream is not | ||
| guaranteed to be deterministic. |
There was a problem hiding this comment.
Might be worth adding an informative note about this topic.
if the block order in CAR stream is not guaranteed to be deterministic
I don't really know how to parse this. By deterministic block order do we mean:
- "block order can be recreated identically by the current implementation on a subsequent request" OR
- "block order conforms to some canonical order defined by this other spec, see here, it's the best order, so feel free to caches these ones."
perhaps canonical order is the thing this is aiming at?
There was a problem hiding this comment.
I believe this specific spec cares about behavior on subsequent HTTP request.
This is one of the weaker parts of the spec:
- added this paragraph as quick placeholder (second leg is
Accept-Ranges(response header) which has the same caveat about block ordering) - afaik go-ipfs 0.13:
- streams CAR in deterministic order because we walk the DAG the same time each time, but it is not documented anywhere, not it is a canonical thing (afaik)
- streaming CAR means we don't know its total size without creating it in-memory first → can't set
Content-Lengthcorrectly → range requests are not possible- lack of range requests over CAR could be a bug, or could be a feature: by not allowing range requests we could encourage use of IPLD selectors (for partial requests, resuming partial DAG downloads etc).
Anyway, if we had a spec for canonical order in CARs, mentioning it here would make things MUCH easier for implementers.
I think @alanshaw @hugomrdias worked on something like this? (can't recall the details..)
There was a problem hiding this comment.
This is great @lidel . Thanks for putting it together, it is super useful for everyone writing IPFS Gateways or building on top
| resolving content paths that start with `/ipfs/` and `/ipns/`. It allows for | ||
| traversal based on link names, which provides a better user experience than | ||
| low level logical pathing from IPLD: | ||
|
|
There was a problem hiding this comment.
Should we recommend similar behaviour for case where given path is not valid?
There was a problem hiding this comment.
@vasco-santos mind elaborating?
There was a problem hiding this comment.
Sorry for not being clear. For instance, if I provide /ipfs/bafy...foo/this/path/does/not/exist, this path won't exist. My suggestion here would be if we should recommend a response as part of the spec so that each gateway send same response back.
I would love to have a better response then what we get as https://dweb.link/ipfs/bafkreidyeivj7adnnac6ljvzj2e3rd5xdw3revw4da7mx2ckrstapoupoq/test
There was a problem hiding this comment.
@vasco-santos added "Handling traversal errors" section here – lmk what additional info should be added in that section
Co-authored-by: Adrian Lanzafame <adrianlanzafame92@gmail.com> Co-authored-by: Vasco Santos <vasco.santos@moxy.studio> Co-authored-by: Oli Evans <oli.evans@gmail.com>
As suggested in #283 (comment)
Co-authored-by: Steve Loeppky <stvn@loeppky.com>
optional header suggested in: #283 (comment) rationale: having specific name as a suggestion of 'best practice' in the specs will simplify debugging across ecosystem
| - Every `.` is replaced with `-` | ||
| - DNSLink label decoding | ||
| - Every standalone `-` is replaced with `.` | ||
| - Every remaining `--` is replaced with `-` |
There was a problem hiding this comment.
Odd number of --- here does not look like a valid encoding:
/ipns/foo-bar.example.com→foo--bar-example-com.ipns.dweb.link/ipns/foo--bar.example.com→foo----bar-example-com.ipns.dweb.link/ipns/foo---bar.example.com→foo------bar-example-com.ipns.dweb.link
Is someone requests foo---bar-example-com.ipns.dweb.link the first -- will be turned into single - producing foo--bar.example.com. Content will load only if such domain name exists and DNSLink exists.
There was a problem hiding this comment.
Are foo-.bar.example.com or foo.-bar.example.com valid DNS domains? Those would both translate to foo---bar.example.com, right?
There was a problem hiding this comment.
no, - is not allowed at the beginning, nor end of a DNS label:
The labels must follow the rules for ARPANET host names. They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen. There are also some restrictions on the length. Labels must be 63 characters or less.
– https://datatracker.ietf.org/doc/html/rfc1035#section-2.3.1
Synthesis of ideas from: ipfs/kubo#8455 and ipfs/kubo#9058
|
PSA: I will merge this PR by the end of the week. Feedback from key stakeholders was positive, and things added in the past few days were mostly clarifications, |
|
Thank you all for feedback, it made it way better than the initial version. ❤️ If you were late to review this, please submit fixes or improvement proposals in new PRs. PRs with new features such as new response types, new headers should include an Improvement Proposal document (template in ipfs/kubo#289) that explains motivation behind proposed change. It will make it easier to discuss / review. 🙏 |
This PR adds initial HTTP gateway specs under
./http-gatewaysdirectory.👀 PREVIEW
💡 Why we need this
The goal of this work is to document the current behavior (based on implementation in go-ipfs 0.13-rc1).
This is a prerequisite that will allow us to switch future gateway improvements to be specs-driven:
🔎 Needs more eyes!
I did my best to document the most important aspects of the current state, but there is a lot to cover. If you see something is missing, or needs rephrasing (i am not a native speaker, and bad writer in general), please comment on this PR.
All feedback will be apprecieted
Next
Content-Rangein range responses?uri=forregisterProtocolHandler?format=dag-jsonand?format=rawTODOblocks into separate PRs, prototype and document the light RFC process