[Feature] Fuse linker

[goloveychuk]

• I'd be willing to implement this feature (contributing guide)
• This feature is important to have in this repository; a contrib plugin wouldn't do

Describe the user story

We all want fast yarn install, and node_modules which don't eat our disk.
Pnp is pretty good solution, but it has compatibility issues: bundlers, resolvers, etc. Migration effort if pretty high.

Describe the solution you'd like

Fuse linker.
Idea consist from three elements:

1. pnpm layout.
   I forked pnpm linker, removed hardlinks logic from it.
   I've split packages into two dirs, .store-fuse and .store-unplugged.
   Unplugged has packages which need postinstalls, they are unpacked to fs.
   Fuse has other packages, and mounted as readonly.
   If fuse linker is not supported in OS, all packages fallback to unplugged.

2. fuse mount of .store-fuse.
   Supported os: linux (libfuse), macos 15.4 (fskit). Probably windows with wsl 2.
   For unprivileged mounts linux needs fusermount util installed.
   Macos need app installed.
   I've implemented custom fuse fs, which ended to be a bit complicated. After finished I've found out that I can generate erofs binary image instead, and reuse erofs-fuse or even erofs native mount on linux (it's supported by kernel, but needs root).

3. fetching archives on demand from remote cache (optional)
   Fuse fs supports async fs actions, which means that we can add fetching data when its requested.

This all will make yarn install time = resolution + cache fulfil + postinstalls.

Sources: https://github.com/goloveychuk/fuse

Describe the drawbacks of your solution

Needs binaries downloaded, need support by OS, could have bugs.

Describe alternatives you've considered

pnp

@arcanis wdyt?

[arcanis]

I deemed it not viable many years ago due to:

• licensing drama
• portability headaches
• native dependencies

Not sure how much of that changed since last time. That's why I went with the Node.js-based FS layer: harder to maintain, but at least we were in control of how it executed on users' machines (usually).

[goloveychuk]

licensing drama

licensing is now fixed, in macos we can now implement fskit extension without relying on macfuse. Thats what I've done in my repo. Libfuse (linked) has LGPL.

portability headaches + native dependencies

this remains. It could be fixed a bit by building go binaries (for linux + wsl2), which are completely statically linked.
But still, linux should have root or fusermount installed, and macos should have extensions installed.
Still, it could fallback to unpack if platform is not supported.
I'm not saying it should replace pnp. But it could be an alternative which has some pros and const.
Also, it could be external berry plugin, but if it would be part of berry builtin plugins, it could simplify an adoption.

Btw, it would help if berry has a plugin interface for Cache. Currently it's hardcoded to zip all over the place.
I can implement this if it fits into design.

[goloveychuk]

Btw, about whole pnp + nodejs runtime. I saw your pr to nodejs which adds native zip support, and it's not merged (and probably won't ever).
I was thinking to target this by doing two steps.

1. RFC for packagemaps
   I think we can write a RFC for nodejs (mb with @guybedford help) which will define a manifest file which will tell nodejs where to find a package from other package. Very similar to importmaps, but from what I understand importmap won't work with nodejs resolution logic we have now (e.g. package exports).
   Idea is to have smth like

        "file:///abspath/package1": {
                   "package2": "file:///abspath/package2"
        },
      ...
       "*": { ...hoistedPackages }

This will fix many problems with node_modules, no need for pnpm, no hoisting needed, can reuse global cache.
Implementation in nodejs resolution algo should be pretty easy, just change the "node_modules/package" lookups with json lookup. Other resolution logic will remain the same (exports field).
2) second step is to introduce other protocol, maybe zip, maybe some other, optimized for performance (erofs, squashfs, custom format)
This will require implementing vfs subsystem in nodejs itself, they probably won't accept it, but we can try.

[ItsHarper]

licensing is now fixed

I know the person I quoted knows this, but for anyone else seeing this who might have gotten confused like I did at first, osxfuse/macfuse is still closed source, but now fskit exists which is a new macOS API that can be used instead of macfuse, though it's not a drop-in replacement.

[goloveychuk]

Btw, I've faced with some limitations which could make runtime performance slower in macos. Fskit does not currently support kernel caching for readdir, lookup, getattr. Libfuse in comparison has great performance.
I've created an issue in apple forum, they could working on this from what I understood. At least they are aware of it. Will check new macos.