Verifiable HTTP Gateway Responses

[lidel]

Summary

Various organizations are exposing IPFS content via Public HTTP Gateways (eg. Cloudflare).

There are users who can't run local IPFS node, but still could get some guarantees provided by content-addressing if they have software capable of re-calculating CIDs from arbitratry payloads.

This issue tracks ways we could do that.

Problem statement

• It is not enough to "fetch data from gateway and calculate CID", as we don't know how DAG looked like, so if CID does not match, we don't know if it is false-negative.
  • 2020Q4: CAR import/export seems to be the best solution to that problem (Make ipfs.dag.export built-in feature of HTTP gateways #170).
  • 2021Q1: @aschmahmann proposed that we add support for hashes of "unchunked" files in RFC|BB|L2-09/10 Handling arbitrarily large blocks and Treating files as large blocks
    • This is better than CAR, as does not require user agent to support CAR format and can verify regular responses based on URI alone

Use Cases

• (UC1) Package Manager fetching archive from HTTP Gateway
• (UC2) User running ipfs-companion or other browser extension
• (UC3) IoT / low power devices that are unable to use true p2p, but can reach HTTP gateways
• (UC4) Web browser that wants to natively delegate IPFS handling to public gateways, but provide users with integrity guarantees of IPFS

Work

• Research limitations of existing vendor-specific prior art
• Research how to resolve CID of an arbitrary path and get insight into its DAG structure
  • A: 💢 additional lookup via local js-ipfs used only for that or fallback to http://ipfs.io/api/v0/
  • B: 💛 Gateway exposes all metadata required for Reproducible File Import via HTTP header, so no additional lookup is required
    • Gateway support for CAR export (Make ipfs.dag.export built-in feature of HTTP gateways #170) is a variant of this
  • C: 💛 UnixFS files identified using hash of the full content (RFC|BB|L2-09/10 Handling arbitrarily large blocks and Treating files as large blocks protocol/beyond-bitswap#29)
    • Improves on top of B because it does not require client to know/implement Unixfs/IPLD specifics, just read hash function type from CID and then compare it against the hash of entire file.
    • Requires separate provider records, so won't work for legacy data/links
• Gateway can return all parameters needed for re-building DAG during validation (or CID can be validated directly)
  • BLOCKED by Reproducible File Imports ipld/legacy-unixfs-v2#15
  • Note: this may become not a problem if we switch approach to CAR import/export: Make ipfs.dag.export built-in feature of HTTP gateways #170
    • if we are able to control the depth of CAR export, we could have parallel streaming from multiple gateways
  • Note: alternative to CAR is RFC|BB|L2-08: Handle Arbitrary Block Sizes
    • This is better than CAR, as does not require user agent to support CAR format and can verify regular responses based on URI alone
• (UC1)(UC4) CLI tool is able to fetch HTTP payload and validate it against a CID in offline mode without need for a full IPFS node
  • Highly useful for use in Package Managers
• (UC2) Browser Extension is able to validate HTTP payload against its CID
  • Limited by APIs provided by browser vendors, details tracked under IPFS Companion: HTTP Gateway Validator ipfs-companion#593
• (UC3)(UC4) full DAG can be fetched from public gateway
  • CAR export built-in feature of HTTP gateways: Make ipfs.dag.export built-in feature of HTTP gateways #170

Resources

Prior Art

• (UC2) Cloudflare

  • End-to-End Integrity with IPFS and DNSSEC
  • cloudflare/ipfs-ext PoC IPFS Gateway Validator browser extension
    • supports only DAGs created with default settings (hash, chunker etc)

• https://tools.ietf.org/html/rfc6249: Metalink/HTTP: Mirrors and Hashes
  Very relevant RFC, which provides HTTP-native semantics for returning Digest with the payload:

   Link: <http://example.com/example.ext.asc>; rel=describedby;
   type="application/pgp-signature"
   Digest: SHA-256=MWVkMWQxYTRiMzk5MDQ0MzI3NGU5NDEyZTk5OWY1ZGFmNzgyZTJlO
   DYzYjRjYzFhOTlmNTQwYzI2M2QwM2U2MQ==
  

IPFS

• go-ipfs: Gateway response is unverifiable by clients Gateway response is unverifialbe by clients  kubo#1594
• Universal HTTP Gateway Validator in browser extension | HTTP Gateway Validator ipfs-companion#593
• Reproducible File Imports
  • unixfsv1 has MetaData struct: go-unixfs/pb/unixfs.proto#L24
  • unixfsv2: Reproducible File Imports ipld/legacy-unixfs-v2#15
• RFC|BB|L2-09/10 Handling arbitrarily large blocks and Treating files as large blocks

[lidel]

• done: block and CAR response formats (feat(gateway): Block and CAR response formats kubo#8758) are supported by HTTP Gateway provided by go-ipfs 0.13+
• wip: specs for "Trustless Gateway" are part of HTTP Gateway specs Add HTTP Gateway Specs specs#283
• next: spec for "Trustless HTTP Clients" needs to be created, documenting full request cycle and minimal subset of IPFS required

[lidel]

This is solved since Kubo 0.13 (see release notes).

Please upgrade your gateways and clients.
Every Gateway should support Block and CAR responses.

tl;dr summary

When you don't trust gateway, use Block and CAR responses. CAR verification does not require running IPFS node. Clients can leverage standalone tools and libraries such as @ipld/car or ipfs-car:

$ npm i -g ipfs-car
$ curl "https://ipfs.io/ipfs/bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q?format=car" | ipfs-car
$ ls ./bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q/
1007 - Sustainable - alt.txt
1007 - Sustainable - transcript.txt
1007 - Sustainable.png

Further reading

• Gateway User referece docs: https://docs.ipfs.tech/reference/http/gateway/#trusted-vs-trustless
• Trustless Gateway spec documents minimum API subset when file hosting is not needed, and gateway operator wants to only support Block and CAR transports.