experiment: http retrieval

[dirkmc]

HTTP Retrieval

To serve a file over HTTP we need to

1. Check if the piece is unsealed
2. Get a reader over the piece data
3. Get the offset of the CAR file inside the piece data
4. Get the size of the CAR file with padding removed

HTTP Server as an embedded process

Dependencies required for an HTTP retrieval endpoint:

PieceStore

The Piece Store is needed to get the sector id, offset and length of a deal

SectorAccessor (defined in lotus/markets/dagstore)

The Sector Accessor is needed to get a reader over the piece.

The dependencies of the sector accessor are:

• dtypes.MinerAddress
• sectorblocks.SectorBuilder
  Requires a connection to the sealing RPC API
• api.FullNode (full node API)
  Requires a connection to the full node RPC API
• sectorstorage.PieceProvider
  Requires a connection to the sealing RPC API and Remote storage RPC API

DAGStore

The DAG store is needed to calculate the size of the unpadded CAR file:

• Get the index from the DAG store
• Get the largest offset in the index
• Seek to that position in the file and read the size of the section
• The size of the unpadded CAR file = offset + section size

HTTP Server as a separate process

To run the HTTP server as a separate process, it would need configuration for

• Sealing RPC API
• Full node RPC API
• Storage config (storage.json)
• Boost RPC API

The Boost RPC API would need the following new methods:

// Gets the sector, offset, length of a piece
GetPieceInfo(pieceCid) PieceInfo
  (calls PieceStore)

// Gets the largest offset in the index for a piece
PieceLargestBlockOffset(pieceCid) offset
  (calls DAGstore)

[dirkmc]

Closing in favour of #574