Thinking through Motion deal preparation

[hannahhoward]

Problem Overview

For the motion MVP, we will received byte data sent to us via POST request.

• In order to store these bytes on Filecoin, we'll need to take this data and put it in one or more CAR files (only one for MVP given <32GB upload limit).
• The data will need to be managed by Motion, either in its original format or in CARs, until a deal is made. We may want to store these staging files on an external provider such as s3 (question: is this needed for MVP?)
• When the user makes a POST request, we will give them a data ID. A user may later use this data ID to query the status of Filecoin storage for their data, or to retrieve their data. Our data preparation solution will have to account for tying prepared CAR files (and later to Filecoin storage deals) back to a data ID for status and retrieval

Data preparation options:

Data preparation through Singularity

Singularity has a well built out pipeline for data preparation. An overview is found here (please read this first as the rest of this solution relies on understanding how this works).

To prepare data for singularity, Motion would first create a single Dataset and Source for the entire Motion instance. Then for each POST request, we could push a new Item into the created Source, and potentially return as an dataID the items primary key. Assuming we ran a dataset worker(s), singularity would automatically create CAR(s) for each item and it's associated Item Parts. We could then use the deal making system in Singularity (which needs to be built out further) to store the CAR files on filecoin.

In terms of storage, Singularity has two options for CAR files

• it can store the actual CAR files themselves. For this option, Singularity currently writes only to a local storage, though Xinan says he's built POCs that put the data on external storage.
• it can create a proxy server which assembles CAR files on the fly to serve to SPs from the original data source. This might be a neat option cause it would allow using any supported singularity data source (which are numerous), and we could keep management of source data within motion. Note in this case we'll need to maintain a database row for each block in each CAR file, at least until it's stored, which could lead to a rapidly growing database.

For status and retrieval, we'd need to build a relational data query that could map Item -> ItemParts -> Chunks -> CARs -> StorageDeals

A retrieval would probably look like retrieving each ItemPart for an Item and stitching them back together. Since each ItemPart is ultimately always stored in a single piece, we could execute Lassie retrievals for each item part individually, and make use of all supported protocols to get entire ItemParts in a single request (1GB). We can also support range requests because the range of each ItemPart in the original Item is stored in the database. (I'm going to explore retrieval paths more in depth in another issue)

A few things to consider about this system:

• To faciliate storage, status, and retrieval, the Singularity database is critical. Moreover, especially if we use the approach of having Singularity assemble CAR files on the fly, the database could grow large. Long term we'll need to make sure ISV's backup their database
• There is no retrieval or status software written yet that can tie this all the way through

Data Preparation Through RIBS

RIBS offers a blockstore interface that offloads portions of the store to Filecoin deals. An overview of their approach is found here.

Since RIBS assumes a blockstore model, for data preparation for the RIBs case, we could need to convert each POST request of data into an IPLD UnixFS DAG. We could do this by running Kubo or a more lightweight DAG generator. We'd then put every block into the RIBS blockstore, and RIBS would generate CAR files for us as we filled up its staging area.

Most likely, rather than use RIBS for deal making, we'd take the assembled CAR files and put them directly into Singularity's deal making engine. We can hook this up by providing RIBS with a custom ExternalStorageProvider (it's interface offloading to Filecoin) and Singularity's ability to add assembled CARs directly to its deal making engine.

For storage, RIBS can stage data locally, or it already has prototype S3 support. RIBS would simply manage staging storage for us.

RIBS already supports retrieval through Lassie. It retrieves individual blocks, transparently switching between those it has locally and retrieval through Filecoin. Currently, it retrieves blocks individually from Filecoin, which might not scale super well for large amounts of data. Retrieval through RIBS would simply look like executing a DAG traversal for a piece of data backed by the RIBS blockstore.

Status through RIBS might a bit harder since RIBS queries are almost entirely block based. RIBS provides a facility to map a set block hashes of a set of per-CAR groups. We could run this query with all hashes immediately after adding an item, and then store the associated CAR groups for a given item. We could query the status of Group in RIBS, then the the status of actual deal making through Singularity.

A few things to consider about this system:

• Just like the Singularity data prep, the indexes held by RIBS of where blocks are stored is critical to status or retrieval, so we'd similarly need to back it up and account for database storage needs
• The RIBS solution IMHO looks like less code to write to get retrieval working.

Proposal: Aim for using Singularity only, use RIBS as needed to get to MVP faster

My gut feeling is that ultimately, Singularity's data prep is more robust, if more complex. There are some additional features to write for it, but writing those features is an investment in Singularity development, which seems like a win for the ecosystem.

That said, for an MVP, RIBS seems like a solution we can use as a prototype so we can focus on building Singularity deal making features, which make up the majority of technical complexity in MVP.

[willscott]

• I agree there isn't an obvious or reasonable-looking way to avoid a local metadata DB for mapping between data and the deals it has ended up in.
• The proposal here seems reasonable to me

It sounds like we're hearing pretty strongly that the reality is that most SPs will want to pull data from the client. That will mean either exposing an HTTP server at the ISP for them to pull from directly, or provide that interface by:

• staging on s3
• using something like cloudflare argo
  to allow fetching without direct inbound exposed port.

[TorfinnOlsen]

FYI this link is now broken: https://github.com/data-preservation-programs/singularity/blob/feat/data-preparation-overview/docs/en/technical-architecture/data-preparation.md