Storing results in a private IPFS network

[simonwo]

Hi all, I've been working with @JvD007 to get bacalhau get working with downloads from a private IPFS cluster.

Initial problem report was that bacalhau get fails with a timeout when trying to do this. Investigation reveals that the connection with the IPFS node is never established due to some opaque errors and turns out this is because the Bacalhau client needs the same swarm key as the private IPFS cluster.

The workaround is:

1. On your client machine, create an IPFS repo thus: export IPFS_PATH=/tmp/something; ipfs init -e
2. Get the swarm key for the private swarm and copy it into the repo: cp swarm.key $IPFS_PATH
3. Tell the Bacalhau client to use your IPFS repo: export BACALHAU_SERVE_IPFS_PATH=/tmp/something
4. Tell the Bacalhau client to use your IPFS node: export BACALHAU_IPFS_SWARM_ADDRESSES=...
5. Tell the Bacalhau client to use your private API server: export BACALHAU_API_HOST=...

Now bacalhau get should work without issues. This isn't a problem for bacalhau serve because it uses the unauthenticated HTTP API to save data (I presume?)

Slightly different steps where we allow Bacalhau to create the IPFS repo (e.g. skip step 1 and just mkdir) also work, but fail after the first get because the IPFS config written into the repo by Bacalhau doesn't quite work.

Jaco explains the issue with Bac-generated config

Problem is coming from the config file which is create during the first run of Bacalhau. You need to add the correct bootstrap in the config file, remove the defaults. Defaults are:

"Bootstrap": [
   "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
   "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
  ]

These entries are for a Public IPFS setup, remove them and add your correct bootstrap; So change this to your private IPFS bootstrap node

"Bootstrap": [
    "/ip4/<private-ip>/tcp/4001/p2p/1<private-p2p-id>"
  ]

Run again and everything works fine.

So the situation is that getting data from a private Bacalhau cluster currently requires non-trivial configuration on the client side. I would like to discuss if/what we want to do to make this smoother. AFAICS the options are:

1. Do nothing, document the workaround

We can include these details in our documentation for running private clusters. Clearly better than nothing, but this either involves having the ipfs binaries on the client machine or getting clients to manually mess with the config after the first bacalhau get – nethier are super great.

2. Make bacalhau write better config

We should be able to just get Bacalhau to use the value of BACALHAU_IPFS_SWARM_ADDRESSES in the config it outputs. This would remove the need to mess around with the config but would still require clients to manually establish an IPFS repo and copy the swarm key into it.

3. Make bacalhau accept a swarm key config parameter

We could give Bacalhau the power to recognise an environment variable for the swarm key. This would mean that user would only need to have the swarm key on their local machine and all of the IPFS repo management can remain with Bacalhau. E.g. setting BACALHAU_IPFS_SWARM_KEYFILE=... would allow Bacalhau to use that key as part of it's temporary repo during a bacalhau get.

4. Make bacalhau get automatically detect the private swarm details

We could make the published result returned by the requester node automatically include the swarm addresses and swarm key. When using bacalhau get, the in-process IPFS node could use them to establish a private connection to the swarm. This means that we don't need to do any special configuration to establish a private connection - essentially, the requester node provides the secret details automatically to authenticated clients.

5. Introduce our own data download mechanism

We could introduce our own protocol between requester and client for downloading results. Essentially the requester would become a proxy for downloading the data. We have talked about this previously as a way of having a private data storage location whilst still allowing users to download data for just their jobs. Such a thing would also mean we could deliver data to lower-trust users without having them make a direct conneciton to the data store. But this would be a non-trivial engineering problem and would be reinventing a wheel that has already been invented a number of times before.

I'd welcome feedback on these options and what we think is reasonable now and next.

[rossjones]

tldr; 3 then 5.

As it currently stands, the Bacalhau CLI contains the code necessary to interact directly with any of the providers implementing the available publisher specs (as long as they are IPFS or HTTP based). This works as long as the CLI has access to the storage (e.g. it is accessible to the machine performing the download).

The CLI already has to create a local IPFS node in order to communicate with other nodes and retrieve the data. Apart from this situations, this works fine in unconstrained envrionments, but is unlikely to work well should we introduce storage plugins.

Option 5, even if the first version is simply replicating what the CLI does now has many benefits. First it removes the need entirely for both the temporary repository and the optional manually managed IPFS node on the client machine. Then it gives us a policy decision point around access to the data, and then access to all the storage providers - even if the CLI knows nothing about them as they'll just need to read from a HTTP endpoint. As we look forward to implementing pluggable storage providers (perhaps via the container storage interface) there will be no need at all for the client to understand, access or interop with the underlying storage.

For the shorter term, option 3 seems like the best compromise in that it will solve the problem and won't require any deep changes that might influence future design decisions. Any option that requires installing IPFS on a user's desktop/laptop is suboptimal as I think it's unfair to expect users to have to run infra.

[simonwo]

Agreed. So I think the minimum we need to do is both 3 and 2 – so that when Bacalhau generates the private config it still works (I think that's an independent issue with private swarms).

We can continue to discuss 5 at length here.