Boost HTTP Retrieval

[dirkmc]

We should consider implementing retrieval for

1. Raw CAR files

• Allow clients to request the CAR file by
  • deal uuid (lookup piece cid in DB)
  • payload cid (lookup piece cid in DAG store)
  • piece cid
• Copy implementation in retrieval provider's getPieceInfoFromCid()
  • Query piece store for sector info (sector ID, offset, length)
  • Query lotus miner to check if piece is unsealed
  • Query lotus miner for Reader over piece data in sector at offset

2. Original file (before it was converted to a CAR file)

• Get reader over CAR file as above
• Convert to raw file. See lotus client's outputUnixFS()
  • Note that this function assumes DAG-PB UnixFSv1 format
  • In future we could allow the client to pass other formats as a parameter
• Need to work out if it's possible to support resumption

[dirkmc]

PoC of http retrieval of raw CAR files: #495

[jacobheun]

Experimental feature support released in https://github.com/filecoin-project/boost/releases/tag/v1.2.0, we'll create new issues for additional features.