Research the feasibility to providing a VFS extension mechanism to `fs`, `require()`, `child_process`, etc

[jviotti]

Refer to #31 (comment) for a better explanation of the problem.

Monkey-patching all of these functions is non-trivial and requires a lot of code. The user-facing surface of this is pretty large, and I wonder if providing an extension point at the lower levels is actually easier.

Can we do some research to evaluate the feasibility of this?

@RaisinTen You have dug a bit on this on Postman's internal bootstrapper patch, but we do welcome any help or insights from the community.

[RaisinTen]

Btw, after nodejs/node#44537 landed, we are having some related conversation in nodejs/node#44713.

[RaisinTen]

The user-facing surface of this is pretty large, and I wonder if providing an extension point at the lower levels is actually easier.

Sounds like nodejs/node#39711 (comment) is a potential approach we can try experimenting with:

// node.h
class UvDelegate {
 public:
  explicit UvDelegate(node::Environment* env);

  virtual int uv_fs_open(...);
  virtual int uv_fs_close(...);
};

// node.cc
void InternalModuleReadJSON() {
  ...
  int fd = env->delegate()->uv_fs_open();
  ...
}

// user.cc
class EmbedderUvDelegate : node::UvDelegate {
  int uv_fs_open(...) override;
  int uv_fs_close(...) override;
};

node::CreateEnvironment(..., delegate, ...);

or something simpler:

// node.h
struct UvAPIs {
  UvFsOpenFunc uv_fs_open = uv_fs_open;
  UvFsCloseFunc uv_fs_close = uv_fs_close;
  ...
};

// user.cc
UvAPIs uv_apis;
uv_apis.uv_fs_open = ...;
uv_apis.uv_fs_close = ...;

node::CreateEnvironment(..., uv_apis, ...);

but I'm not sure if this is actually "easier".

[jviotti]

@RaisinTen Yeah, this is exactly what I'm talking about!

but I'm not sure if this is actually "easier".

I do think it's easier from an "amount of code" point of view and I'd be interested in exploring this further. Maybe do a demo patch if you are up for it @RaisinTen ?

The only limitation I can see here is that while it does make it potentially easier to implement a VFS on C++, it would be hard to use the same APIs from JavaScript, right? I wonder if we could also expose the same "delegate" concept or something like that to JS too?

[RaisinTen]

The only limitation I can see here is that while it does make it potentially easier to implement a VFS on C++, it would be hard to use the same APIs from JavaScript, right?

My understanding was that the patching would take place completely at the C++ level and there should be no extra APIs exposed to JS?

[arcanis]

It's still unclear to me why you want/need C++ hooks? Why isn't patching fs enough?

[RaisinTen]

Patching fs is problematic because it's hard to keep those patches in sync with the upstream fs API changes from node. This is a problem Electron's ASAR has faced and we have sent a PR to address at least one of the API changes in electron/electron#34764.

[arcanis]

Yes, I'm familiar with this problem; still:

• if the goal of this project is to have one "blessed" VFS implementation specifically made for reading files from archives, then it doesn't have to have a hook API, it can be integrated with Node. By this point, having it in JS is fine, since tests will be part of the Node codebase itself.

• if the goal is instead to allow for multiple VFS implementation, then native is out of the question since it's not portable (Node scripts and packages are portable by default, and I think it's something very important to preserve).

And for experimentations, patching fs ourselves isn't perfect for the reasons you said, but we've done that for some time and we can keep doing it until we find a format we all agree on. We probably don't need a C++ API just for this?

[RaisinTen]

if the goal is instead to allow for multiple VFS implementation

I think that was the general agreement.

We probably don't need a C++ API just for this?

I don't think doing C++ hooks is a strict requirement. We're trying to find the most optimal level where allowing the configuration of these APIs would reduce maintenance burden for both node and userland. It probably should be somewhere between monkey-patching of node's fs apis and allowing overloads for libuv c++ functions.

Today @jviotti and I were discussing about doing this in the fd level by just monkey-patching fs.open but it seems that there are a lot of path based APIs that would remain unaffected, so we should think of a different approach.

We also realized that supporting the plain monkey-patching approach would require patching:

• fs
• child_process
• cjs module loader - Module._stat and Module._readPackage (not sure if esm needs patching of some other components)

I wonder if we can come up with a single API that allows users to configure all of these at once.

[arcanis]

if the goal is instead to allow for multiple VFS implementation

I think that was the general agreement.

This wasn't my understanding, and under this prompt I believe the work drastically changes. If, instead of finding a blessed format, and integrating it transparently inside Node itself (similar to phar, eggs, or jar archives), the intent is to simply make it easier to support multiple VFS implementations, then:

• It's much less useful imo, performance and transparent behaviors were the two main user-facing advantages I saw to this project, over existing solutions.

• Node.js already has the proper layers (patching for CJS, a resolver overriding node:fs for ESM). All that remains then is to build a proper way for userland authors to test that their VFS implementations have the expected behavior compared to the standard Node APIs (essentially, a testsuite-as-a-library). And potentially to add a couple of VFS in CITGM, although it's almost there already, only waiting for lookup: add yarn to lookup.json citgm#905 to land.

[GeoffreyBooth]

cc @nodejs/loaders

[jviotti]

if the goal of this project is to have one "blessed" VFS implementation specifically made for reading files from archives, then it doesn't have to have a hook API, it can be integrated with Node. By this point, having it in JS is fine, since tests will be part of the Node codebase itself.

The observation here is that even if we provide a "blessed" VFS implementation in the context of SEAs, there are other projects that implement VFSs for reasons other than SEAs, and they still monkey-patch a lot of functions. If there is a way we can provide hooking capabilities in Node.js, then we would both solve the SEA problem AND solve the overall monkey-patching situation that everybody seems to agree sucks and should not happen (and it's incompatible with ESM).

I think that if we widen our scope a bit more, we might be able to come up with a solution that fixes the entire problem for once and for all.

That said, I don't think blocks the rest of the SEA work. For now, we can monkey patch as everybody else is doing (or patch Node.js directly for the sake of our experiments) while we try to find a proper solution to the problem, and hopefully use that later on.

This wasn't my understanding, and under this prompt I believe the work drastically changes. If, instead of finding a blessed format, and integrating it transparently inside Node itself (similar to phar, eggs, or jar archives), the intent is to simply make it easier to support multiple VFS implementations, then:

We want to find a "blessed" one, but based on all the previous discussions we had over the past weeks, it seems that identifying a VFS that checks all the boxes will be tricky. The decision that came out of the last meeting was:

• Accepting that we don't know of a VFS implementation that checks all the boxes
• Embracing that knowledge gap as the ground for further experimentation on the VFS area
• Run SEA experiments with multiple VFS and see what we learn from that
• Then properly revisit the problem of selecting a blessed one

For point 3, we will need to integrate with many VFS (for the time being) as we experiment with them, so the problem of hooking in into Node.js becomes more problematic for us. This is all internal experimentation and we would hope to ship a blessed VFS at some point, but we don't know which yet.