Replacing build context tarballs

[simonwo]

As part of the WASM refactoring in #818, I am starting to think about how we can formalise the "file upload on job submission" feature into a more general feature. At the moment, the feature is:

• for Language jobs, will targzip up a directory or file of choice and pin it to IPFS
• will place a pinned storage spec into the Contexts part of the job spec
• Docker jobs don't/can't access the functionality currently afaics
• WASM jobs use this functionality for uploading a WASM binary

This interaction is handled at the API level, so the compressed data never ends up in the job spec.

The motivation for refactoring this is:

• We want a lower barrier to entry for Bacalhau, so it should be easier to put local data somewhere that Bacalhau can access it (e.g. if I have a config file or maybe secret data on my machine, I don't want to think too hard about how to get it into IPFS/anywhere to use it on Bacalhau, I want it to just work)
• We want to move to a data structure where all context that is needed to run a job is the executor-specific part of the spec (so stop using Contexts for that – discussed in Native execution of very simple WASM jobs #816) – so I want to be able to upload a blob into an arbitrary part of the Wasm spec.

My design idea so far is to extend model.StorageSpec to allow "inline" data. This would be a new StorageSourceInline type, and either a new property on the StorageSpec for inline data or encoding it as a data:base64,... url in the URL field. E.g.

inlineSpec := model.StorageSpec {
    StorageSource: model.StorageSourceInline,
    Name: "my-content",
    URL: "data;base64,aGVsbG8gY29kIQ",
}

When submitting the job, the API endpoint would look at the storage specs that are being asked for and identify any that are of StorageSourceInline type. It can then decide what it wants to do with them: one strategy would be to pin them to IPFS, but another might be to allow small bits of data (like short strings) through to continue to be included inline. For any it wants to pin to IPFS, it would then replace those fields in the JobSpec with IPFS storage specs.

This makes it easy to write more command line tools that use local data – the CLI command then just needs to include the data inline in the job spec, and the API server will handle putting it on IPFS (or not) – as opposed to having the CLI needing to join IPFS or upload to some other server via some other API. The user can continue to just run 1 command as expected. It also means this functionality can be used anywhere a normal StorageSpec is expected, rather than it being hardcoded to a single place as it is now.

We would make sure this mechanism ins't abused (any more than it is currently!) by keeping some size limit restriction as we do currently.

What do we think about this?

[lukemarsden]

Sounds good to me. I'm specifically interested in avoiding leaking large strings into libp2p/gossipsub, as it's not designed to handle that AIUI. So as long as the replacing of StorageSourceInline -> StorageSource happens in the publicapi layer, I'm fine with this. Thanks!

[simonwo]

As of #1628, this feature is now merged!