HTTP fetch of fs-migrations should use CAR

[lidel]

Version: 0.14.x

Fetching migration data from IPFS was added in #8064, but we did not have #8758 at the time and HTTP fetch is still delegating trust to the gateway.

I consider that a bug: migrations should be fetched in trustless mode as a CAR by requesting them as ?format=car and verifying every block before applying the migration.

This allows us to use third-party gateways af fallback and/or in scenarios where ipfs.io is blocked by ISP etc.

[BigLep]

@Jorropo will tackle this after some related light-client work he is doing.

[BigLep]

@lidel : is this a duplicate of #8851 ?

I see there is also this PR: #9250

[lidel]

I've closed #8851 as a duplicate (this one has more context).

[hacdias]

I'm marking this as blocked by the IPIP-402. Once we have that, our life will be easier. Also, I want to add that we need IPNS resolution to solve this too. Copy of my comment:

I just took a second look at this and indeed it'll be much easier once we have widespread IPIP-0402. There are two main problems here:

1. We need to be able to resolve distPath. If it is a /ipfs path, we're good to go. If it's a /ipns path we have to:
   a. Check if it's regular IPNS. If so, we can fetch record from gateway via ?format=ipns-record, validate and get the /ipfs path.
   b. If it's DNSLink (most likely), setup a NameSys resolver and resolve it via DNS.
2. With the distPath being an /ipfs path, we can now request a CAR for {distPath}{filePath}?dag-scope=entity and verify if the CAR and path resolves correctly. Use that data for the migration file.

I was hoping we could do something now, but it'd just be too complicated and we're already working on IPIP-402 so I would mark this as blocked for now.

[lidel]

IPIP-402 shipped. Once ipfs.io upgrades to Kubo 0.21 or later, we will be able to fetch /ipfs/cid/path/file as entity+path and verify it from any gateway.

We should make it work with all gateways, so if anything is missing in CAR response, fetch it block-by-block (we have prior art in bifrost-gateway).

I don't think we should block on IPNS record resolution.
We do not use it for migrations, and for that we have hardcoded CID, we only care about /ipfs/ paths:

• https://ipfs.io/ipfs/QmYerugGRCZWA8yQMKDsd9daEVXUR3C5nuw3VXuX1mggHa/fs-repo-13-to-14/versions?format=car
• https://dweb.link/ipfs/QmYerugGRCZWA8yQMKDsd9daEVXUR3C5nuw3VXuX1mggHa/fs-repo-13-to-14/versions?format=car
  etc