Practical Fast Retrieval on Filecoin

[zixuanzh]

Abstract

Enable Fast Retrieval on Filecoin by creating a second type of proof, dubbed "Fast PoRep", potentially with fewer layers of SDR (or other mechanisms) and requiring higher per-unit-storage initial pledge requirements to offset the loss in hardware security.

Motivation

• One of the biggest challenges for the Filecoin Network to deliver value is that it is expensive to retrieve the data stored on Filecoin.

  • There are L2s building on Filecoin as Hot Storage Layers but 1) GTM and fundraising can be quite hard 2) if Filecoin Retrieval itself is still very slow (fastest unsealing time requires around 10 minutes), then the argument to use Filecoin as a backend/value accrual can be weak.

• While the ecosystem can spend millions of dollars and months of research time to develop a new PoRep, it is also a design choice of the Filecoin protocol to prioritize hardware security.

• As such, as a naive proposal, it is worth considering to reduce the hardware security from the number of layers of SDR required in the current PoRep (or other forms of proof) to enable fast sealing and unsealing quickly.

• This then creates a marketplace of PoRep with a tradeoff over their security and usability.

• To counterbalance the loss in security over usability with fewer number of SDR layers or other forms of proofs, the per unit-storage initial pledge requirement of such new PoRep should also increase.

Design Rationale/Consideration

• Time to market is key.

  • which includes research and implementation time
  • only if we also believe this FIP moves the needle for Filecoin

• The goal is to find a solution that can be live by the next upgrade, hence "practical", making Filecoin more usable, lowering hardware requirements, and locking more tokens to compensate for the loss in hardware security.

• As such, this FIP tries to avoid shaking existing business models on the network and create a positive sum outcome in the ecosystem.

• SPs can choose different PoReps that suit their needs (similar FoFR regardless of which PoRep that SPs choose from a mining perspective). One can argue for a lower FoFR for Fast PoRep since there is less hardware commitment and require higher security compensation from initial pledge, locking up more Filecoin as usage increases.

[nicola]

This is a good direction. Let me give you some context on why specifically lowering the number of layers is not enough:

• Reducing the number of layers of SDR will make PoRep not secure. PoRep needs some number of layers - what we have today - in order to mantain the security of space-hardness needed for consensus security. In other words, if you remove the layers you lose security both in the cost model and in the time model, since there is no way to force an SP to reconstruct the sector if they were to delete more than 20% (the spacegap). Another way to view this is that you are increasing the spacegap - which would make PoRep not suitable for storage-backed consensus security (and if you are ok losing that, we have a new upcoming proposal - happy to collaborate - where we give up storage-backed security and at that point you can use a simpler Proof of Data Possession, which much simpler and requires no sealing).
• Reducing the number of layers will not really give "fast retrieval", the fact that the PoRep layers are sequential will play a big penalty on your latency - e.g. if you were to go to half the layers will still take 30-45 mins on a CPU and way more on GPUs

[momack2]

Current status is that FilOz is working on a draft FIP for the “proof of retrievable copy” Luca described above, aiming for hopeful community acceptance before EOQ 👍

[zixuanzh]

hey @lucaniz , do you need to store an extra copy of the data on top of the copy that you use for mining for either Proof of Data Possession (separate proof of clear data) and Proof of Access (WindowPoSt and WinningPoSt on sectorkey+replica)? Trying to understand do we double the “storage requirement” for either of these

[zixuanzh]

In addition, what's the hardware requirement like (if we can give a percentage estimate) for running those two proofs vs SDR today?

[lucaniz]

You'd have to have access of a clear copy of the data (meaning that this copy can be shared if the file is stored by multiple entities). Or you can use an extra copy of the data if for some reason you are already storing one for backup reasons

[lucaniz]

In addition, what's the hardware requirement like (if we can give a percentage estimate) for running those two proofs vs SDR today?

you can think of the additional proving step as something comparable with WindowPoSt

[jaychoucc]

Fast retrieval is the most important issue for filecoin

[f8-ptrk]

does this proposal indicate that we can exchange [hardware security] and [PoS security] at a certain rate?

To counterbalance the loss in security over usability with fewer number of SDR layers or other forms of proofs, the per unit-storage initial pledge requirement of such new PoRep should also increase.

because then we can just PoS the L1 completely, by far the easiest, fastest, solution for the problems this proposal wants to work on and move all the annoying proof stuff to L2 market implementations.

[f8-ptrk]

SPs can choose different PoReps that suit their needs (similar FoFR regardless of which PoRep that SPs choose from a mining perspective). One can argue for a lower FoFR for Fast PoRep since there is less hardware commitment and require higher security compensation from initial pledge, locking up more Filecoin as usage increases.

as soon as we offer 2+ ways to PoST on the L1 for blockrewards we have a competition between those 2+ solutions. the market will be fast to eliminate all but one of them from being used.

[f8-ptrk]

good example of the effects of offering 2+ solutions with vastly differing cost profiles to achieve the same outcome regarding FoFR

[zixuanzh]

I am trying to reduce scope of changes to reduce both r&d and consensus building time. I think just doing PoS on L1 is a huge change to Filecoin. Changing L1 consensus to PoS also does not solve the problem of enabling fast retrieval.

[f8-ptrk]

it allows to scale the cost of porep hw security to the storage clients needs on the spot without compromising chain security

[f8-ptrk]

To counterbalance the loss in security over usability with fewer number of SDR layers or other forms of proofs, the per unit-storage initial pledge requirement of such new PoRep should also increase.

more important it solves the problem of years of lame duck governance. we can quit the dance around where to put the needle on the PoW - PoS meter. no one whatsoever benefited from SDM, CDM, fip36/56...78....80...stuff. no one! it is a colossal waste of resources and it leaves the project weaker than before.

we managed to implement exactly 0 crypto economical changes since we decided on fip26, in 2021! i wouldn't even call this a crypto economic fip tbh. last real change was october 14th 2020 with fip04

you are smart in putting this out in an early stage to get it shut down or approved on a theoretical level. before r&d is done. because if this gets close/r to getting a community vote be sure i will drag you into another SDM/CDM situation to try to move the needle on the meter more to where i would like to see it. and as long as you put out proposals that devalue miners investments i can do this very efficiently and effective. not moving means stability...

does this proposal indicate that we can exchange [hardware security] and [PoS security] at a certain rate?

if yes - there is no reason to not choose PoW or PoS. i honestly rather have PoS than more PoS - the direction all of your proposals want to move the network in. i am happy to listen to the argument for keeping both, always. keeping both has a proven track record - an ugly one - try your luck. there is no credible version of its future to sell anymore, sry - oversold.

it is how it is, isn't it? i do not enjoy this - at all. - believe me. nothing personal thou, just money!

[jennijuju]

fastest unsealing time requires around 10 minutes)

is there any more info on how is this achieved, Whcih also means sealing can be done in 10min?

[ribasushi]

☝️ seconded. @zixuanzh if what you are saying: unsealing time of 10 minute does indeed exist - this is really bad. That would mean complete invalidation of the current Filecoin security model. Please share your information/findings either here or alternatively with security@filecoin.org as soon as possible.

[irenegia]

Yes, "Sealing in 10 mins" could pose a security problem. To understand if that is the case, we need to know how actually this particular sealing procedure is done (we need details on HW used, storage used, etc)! So it would be great to know more and riba's suggestion to share info by email is good, but we can also do a call.

[ribasushi]

@jennijuju @cryptonemo @nicola can one of the protocol-design groups start some sort of incident doc on this or whatnot? This remark on the existence of a "10 minute SDR" should be thoroughly investigated. Initiatives like e.g. PoA hinge on the security of SDR being sound.

[momack2]

@zixuanzh do you mean that folks can unsnap in 10 mins? that'd be expected. But SDR sealing/unsealing is expected ~2.5hr minimum

[willscott]

I think this fip / proposal is somewhat mis-named. This is not a getting us to 'fast retrieval' but rather is a proof of unsealed copy.

The reasons we are aware of why why content is not made available in filecoin is both a risk and economic problem, and this will chip away at the problem, but it does not address the concept that people expect when they hear 'retrieval'. that needs to both include something like retriev that has a bounty to offset the potential liabilities of serving user generated content, and a protocol which makes the serving of data profitable, while also structuring expected SLAs such that the provider can meet them without expensive/un-needed over-provisioning.

[lucaniz]

FYI

#988 (reply in thread)

[jennijuju]

Will - for proof of retrievable copy I believe @lucaniz is planning to propose a separate FIP discussion and see #988 (reply in thread), we agree on clearly setting the expectation that we believe such new proof can help making fast retrieval POSSIBLE (no need to wait for unsealing).

[jennijuju]

And I agree with what you are raising. Should this fip discussion be retitled to “theoretical fast retrieval on filecoin” instead? 😆

[steven004]

I appreciate the term "a proof of unsealed copy."

[zixuanzh]

This is just a FIP discussion and there is no formal proposal yet. I wanted to align us on the requirements first. Ultimately, I think we want 1) something that we can go to market quickly, hence "practical" 2) make Filecoin L1 a strong dependency for L2s or other projects (without some ability for fast retrieval on L1, the dependency can be quite weak) 3) increase demand for using the FIL token