Make proof param fetching resilient to single points of failure

[jennijuju]

Checklist

• This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• I have a specific, actionable, and well motivated feature request to propose.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

What is the motivation behind this feature request? Is your feature request related to a problem? Please describe.

user reported proof param fetch failure from proof.filecoin.io due to a connection error (the host was down).

more details https://filecoinproject.slack.com/archives/CPFTWMY7N/p1721496569980859

Describe the solution you'd like

implement a fallback option to avoid central point of failure , i.e: fetch from the cloudflare forest team host

https://github.com/ChainSafe/forest/blob/main/src/utils/proofs_api/paramfetch.rs#L140-L145

Describe alternatives you've considered

No response

Additional context

No response

[jennijuju]

lotus gets proof from https://github.com/filecoin-project/go-paramfetch/blob/master/paramfetch.go

[f8-ptrk]

doesn't the IPFS_GATEWAY env var already do that?

[f8-ptrk]

https://github.com/ChainSafe/forest/blob/main/src/utils/proofs_api/paramfetch.rs#L37-L39C1

hardcodes the paths into DEFAULT_IPFS_GATEWAY and CLOUDFLARE_PROOF_PARAMETER_DOMAIN

if env(FOREST_PROOFS_ONLY_IPFS_GATEWAY) == true then it fetches from env(IPFS_GATEWAY) if set or from DEFAULT_IPFS_GATEWAY

else env(FOREST_PROOFS_ONLY_IPFS_GATEWAY) != true then it downloads fromCLOUDFLARE_PROOF_PARAMETER_DOMAIN

if that fails it falls back to fetch from env(IPFS_GATEWAY) if set or from DEFAULT_IPFS_GATEWAY

flip that around to fetch default and download from cf as a fallback or something like env(FOREST_PROOFS_ONLY_IPFS_GATEWAY)

[jennijuju]

Yah I think the question is whether users would like lotus to provide at least 2 trusted options - which I thought could be nice when I opened issue

[f8-ptrk]

yes, users would like that.

i am not sure if the forest download is simple http or something r2 specific.

we host them on http internally and move them pre lotus setup. lotus/go-paramfetch could integrate that and default to forest cloudflare

like IPFS_GATEWAY now, SOME_VAR_TO_HHTP_PARAM_FETCH_FROM

[LesnyRumcajs]

i am not sure if the forest download is simple http or something r2 specific.

It's over HTTP, see here. So, in short, https://filecoin-proof-parameters.chainsafe.dev/ + param file name, e.g., v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk -> https://filecoin-proof-parameters.chainsafe.dev/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk

[BigLep]

A couple of thoughts here:

1. Yes, we shouldn't fail because of one DNS entry mistake/failure or one particular service being down.
2. We should fetch bytes in a trustless fashion. I'm not understanding https://github.com/filecoin-project/go-paramfetch/blob/master/paramfetch.go well enough to know whether it's doing it, but like Forest, Lotus should fetch using trustless verifiable retrieval: https://docs.ipfs.tech/reference/http/gateway/#trusted-vs-trustless
3. Let's not use proofs.filecoin.io but instead use https://trustless-gateway.link per https://docs.ipfs.tech/concepts/public-utilities/#public-ipfs-gateways. I'm suggesting this because:
   • That domain only serves trustless responses and so is far less likely to get blocked. (Maybe that is the case for https://proofs.filecoin.io/ipfs/ - I don't know).
   • Using http://proofs.filecoin.io looks like a config mistake waiting to happen for the IP Shipyard team that is responsible for hosting this currently as I believe its legacy that they even are hosting an HTTP gateway under that domain. Basically if something happens to the proofs.filecoin.io domain, they may not notice for awhile. They'll definitely notice if there's a problem with trustless-gateway.link
   • Hardware-wise I believe IP Shipyard (operators of proofs.filecoin.io, ipfs.io, trustless-gateway.link, etc.) are partitioning the fleet so that the trustless fleet gives better responses than folks relying on hosts that can serve trusted responses (e.g., ipfs.io).
4. With trustless responses, we now don't care which gateway we hit, and could also add other gateways to the list from https://ipfs.github.io/public-gateway-checker/
5. That said, assuming there is only one paid provider to advertise and serve the content to "IPFS nodes", I see the value in having a fallback to another provider, so Lotus falling back to trusted HTTP retrieval from https://filecoin-proof-parameters.chainsafe.dev Forest and then validating the response makes sense. (Ideally Filecoin Foundation would pin with a separate provider than IP Shipyard, but if that is too much work or beuracracy, using something self-service and already established like Chainsafe's R2 bucket makes sense).

I changed the title to "Make proof param fetching resilient to single points of failure" to account for the broader scope. I agree this is a "good first issue".

[LesnyRumcajs]

Also, please note that we mirrored only the keys (based on our needs) - .vk files. We can upload the proof files as well if you’d like to use our bucket as a backup. See ChainSafe/forest#4579

[jennijuju]

there is another source we are managing that is export IPFS_GATEWAY=https://proof-parameters.s3.cn-south-1.jdcloud-oss.com/ipfs/