npm on IPFS

[jbenet]

• ipfs-blob-store
• ipfs-daemon-ctrl control whatever daemon is installed
• plug in ipfs-blob-store to reginabox
• figure if we can 302 publish
• mirror the registry
• bundle it all into one module
• make physical nodes

cc @bengl

[whyrusleeping]

just idle thought? Or discussion?

[jbenet]

@whyrusleeping notes from discussion in person :)

[bengl]

As mentioned in the first checkbox, one blocker for this is that someone needs to make a ipfs-blob-store adhering to abstract-blob-store. I'm working on this on and off in my spare time. Slowly. If someone beats me to it, yay!

[seidtgeist]

@jbenet Hmm... After you explained the ipfs docker mount hack yesterday, what if we mount ipfs on node_modules and require('hash')? Why am I even asking and not just trying this? brb

[whyrusleeping]

@ehd how did that work out?

[seidtgeist]

@whyrusleeping It works :)

$ cat add.js
'use strict';

module.exports = function(a, b) {
  return a + b;
};

$ ipfs add add.js
added QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x add.js

$ cat prog.js
'use strict';

var add = require('QmXctmnj6tEMSJsb433jKwcdGwuBy4uGMnqva5Nkvp5E7x');

console.log(add(1, 2));

$ node prog.js
3

Two things:

• I'd rather not rely on the filesystem or fuse, that's doable
• I'd rather require names instead of SHAs and move the mapping to metadata

[whyrusleeping]

@ehd thats great! what would the ideal interface be for you to want to use ipfs for js modules?

[seidtgeist]

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require. Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.
Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

[jbenet]

Update on the list above: @krl made https://github.com/ipfs/node-ipfsd-ctl

---

@ehd all of that sounds awesome!

Ideally we'd have a simple bootstrapping js/node program in ipfs that can
load other things from ipfs using require.

Yeah! like a special require that uses ipfs directly. Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

Btw, what @bengl and I want to do above with reginabox will work with stock npm and require.

Reminds me of the StackStream
opcodes actually ☺️ Anyway, without a file system the ideal
delivery mechanism would be the true ipfs network, so baking a simple ipfs
client into a runnable JavaScript bundle sounds like an idea worth trying.

that sounds awesome

Initially this could be done by just using the global or local ipfs http
gateway for loading code. I'm not sure whether a dynamic module loader
(require on demand) or a dynamic bundler (compile program, then start
program) would be more viable. Whatever runs in <script src="browser-fetchable-address-on-ipfs"/> or vanilla node I guess.

@krl's https://github.com/ipfs/node-ipfsd-ctl can make ephemeral ipfs nodes using the go-ipfs package, so you dont even have to install ipfs beforehand. it's huge (cause go-ipfs is huge) but once we have node-ipfs this will be a drop-in replacement.

Another related idea we've bounced: storing computational results in ipfs.
Say you have blobs for js-compile and js-source-with-links; you can load
both and run js-compile(js-source-with-links), then store the result and
use that as the above entry point. It's basically how all browser side JS
already works, just that it's very sad how we store build results in dumb
files without linking to how they've been conceived (creation metadata! cc
@jbenet).

Absolutely! And-- i want to make a way to do browserify with partial results separated out. THOUGH, rabin fingerprinting will get us 80% of the way there without the worries anyway.

Side note: Ironically I'm in a place without wifi and just looked up
stackstream on my mac's browser history, but of course I can't access the
already cached page. Idea: any ipfs browser/browser extension should pin
history items. Immediate benefit over any other real world browser.

Yeahhh i really want this. wonder what the overhead + pains of doing this might be... hard to get right because people dont hash things or use cache headers correctly :/

[bengl]

Could patch npm to make requires that start with "/ipfs/..." require straight out of ipfs.

You'd have to patch node. All the require logic is self-contained in node, not npm.

There are also many lifecycle scripts and compilation steps and all kinds of other things that are done to modules during the install process. In some modules, the results of these scripts are different depending on where they are in the node_modules tree.

I think it's much more practical to keep a strict separation between the source of packages (ipfs/ipns, in an ideal world) and the current install directory. Requiring them directly might work for JS-only modules with no lifecycle scripts. But then you're not supporting a huge chunk of available modules.

I'm only really talking about node here, but for people using frontend build pipelines that depend on npm, the same problems can arise.

So yeah, I'm not sure grabbing node dependencies directly from ipfs at run-time is realistic.

[jbenet]

@bengl ah thanks! makes a lot of sense.

i guess we could do it to specific packages people decide to do this with (i.e. people explicitly publish them + require them with hashes, so they take on the responsibility of not making all the lifecycle stuff happen?) not sure.

[bengl]

Yeah for sure. ipfs-pm/ipfspm/ippm. You'd pretty much have to make a clear separation from npm at that point, I'd think.

I still would like to see the ipfs leveraged as the package source, as we discussed before. Should be easy once ipfs-blob-store exists :)

[jbenet]

@bengl maybe we can hack that (ipfs-blob-store) out this week. I'm in bay area atm. (cc @krl @travisperson @mappum @diasdavid in case they're inspired)

[travisperson]

Definitely, I took a look at the spec today. @bengl do you have any of the work you've made towards it published anywhere?

[bengl]

So really we're looking at potentially two different use-cases for ipfs-blob-store. Might be better off as two different modules. I'm not sure.

Use Case 1: Raw Blob Store

In this case, we're really just plopping blobs onto IPFS and getting their IPFS paths back in the metadata for later retrieval. So in createWriteStream, you wouldn't provide a key in the opts as that's what you'd be getting back from IPFS (i.e. the IPFS path). In createReadStream, the key you'd pass in would be the IPFS path, and it would stream back the blob at that path.

Use Case 2: IPNS Mutable File System Thing

In this case (which could use the previous case under the hood), in the constructor for the blob store, we'd provide all the necessary data to read and write to an IPNS namespace (keypair? just the hash? Sorry, my IPNS knowledge is rather limited). Here, createWriteStream would use the key option from the args to determine where to put the blob, and the metadata returned would be kind of irrelevant, but would have the same key. Behind the scenes it would update IPNS properly. So the key would be something like /path/in/namepsace. Similarly for createReadStream.

---

Unfortunately I don't spend as much time on this as I'd like, so I'm unfamiliar with how things actually work in IPNS-land right now. Sorry!

Here's the abstract-blob-store API reference.