Yarn PnP compatibility?

[bencmbrook]

I believe the issue described here (remarkjs/remark-language-server#6 (comment)) is a result of an assumption that packages are stored in a node_modules directory.

I tried running resolve against my project and got the following error

% yarn node test.mjs
file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/errors.js:322
    Error.captureStackTrace(error)
          ^

Error: Cannot find package 'remark' imported from /<mycwd>/test.mjs
    at new NodeError (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/errors.js:276:5)
    at packageResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1060:9)
    at moduleResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1117:20)
    at defaultResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1266:15)
    at resolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/index.js:29:12)
    at file:///<mycwd>/test.mjs:20:19
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:533:24) {
  code: 'ERR_MODULE_NOT_FOUND'
}

where test.mjs is

import {resolve} from 'import-meta-resolve'

console.log(await resolve('remark', import.meta.url))

• remark is installed
• I'm using Yarn PnP, where packages are in .yarn/cache/... rather than node_modules
• Node 16.17.0, Yarn 3.2.3

A more thorough example:

test2.mjs

import {resolve} from 'import-meta-resolve';
import * as remark from 'remark';
import { createRequire } from 'module';

const require = createRequire(import.meta.url);

console.log('remark:', typeof remark, '\n');
console.log('require.resolve:', require.resolve('remark'), '\n');
console.log('import-meta-resolve:');
console.log(await resolve('remark', import.meta.url));

Yields:

remark: object 

require.resolve: /<mycwd>/.yarn/cache/remark-npm-14.0.2-9e7a75b6a9-a534b6c6bb.zip/node_modules/remark/index.js 

import-meta-resolve:
file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/errors.js:322
    Error.captureStackTrace(error)
          ^

Error: Cannot find package 'remark' imported from /<mycwd>/test.mjs
    at new NodeError (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/errors.js:276:5)
    at packageResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1060:9)
    at moduleResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1117:20)
    at defaultResolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/lib/resolve.js:1266:15)
    at resolve (file:///<mycwd>/.yarn/cache/import-meta-resolve-npm-2.1.0-fcf1208127-4554ea5e2d.zip/node_modules/import-meta-resolve/index.js:29:12)
    at file:///<mycwd>/test.mjs:10:19
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:533:24) {
  code: 'ERR_MODULE_NOT_FOUND'
}

[wooorm]

Can you make a tiny example repo that showcases this, using a modern yarn? I am not a yarn (pnp) user myself. That would likely help me.

[bencmbrook]

@wooorm For sure - thank you for looking into this.

Here's a minimal reproduction (and the README.md describes reproduction steps) https://github.com/bencmbrook/import-meta-resolve-pnp-repro

[wooorm]

Thanks. Running yarn in that repo does generate node_modules. How do you expect me to install dependencies without node_modules?

[bencmbrook]

Hm, must be some environment differences - let me dig in on that.

Can you confirm yarn -v logs 3.3.0?

[wooorm]

Yes, 3.3.0. Node 19 or 16.15, then running yarn, both generate a node_modules.

[wooorm]

Managed to reproduce. It does say when running yarn:

➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental

That’s likely it. yarn doesn’t support ESM well.

[bencmbrook]

Hmm yes that could be it, but I don't think it's because the target package, remark, is ESM. I tested it with a package import that's CJS (swap remark for request) and I get the same error.

[wooorm]

I don’t particularly mean that remark itself is esm. More that this project loads files the way Node.js loads files, when in ESM. It can load CJS fine. But yarn PNP is doing magic things to hook into require calls, to not need ‘node_modules’, those hooks are different on the ESM side

[bencmbrook]

Ah, yeah I'm tracking.

I just pushed up another test, using the native import.meta.resolve() and it resolves successfully.

If you pull the changes, yarn start runs the script with --experimental-import-meta-resolve

[bencmbrook]

Digging deeper here... The source of the ERR_MODULE_NOT_FOUND is here

import-meta-resolve/lib/resolve.js

Lines 1019 to 1022 in ef50015

	let packageJsonUrl = new URL(
	'./node_modules/' + packageName + '/package.json',
	base
	)

Oddly, the reference implementation also does this? https://github.com/nodejs/node/blob/3c040348fe9c49687a796fbeea3fe7a9bc1251ed/lib/internal/modules/esm/resolve.js#L874-L875

This might be Yarn's magic? https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-pnp/sources/esm-loader/hooks/resolve.ts

[wooorm]

There may have landed new code to support ESM loaders in the upstream import.meta.resolve algorithm.

I am wondering if Yarn PnP can be solved in your actual use case though.
You will need to call Node.js (or even somehow this package deep down through load-plugin in unified-engine through remark-language-server) with an experimental Node.js flag. And pass Yarns custom loader in. Where would you even do that?

Here’s what Yarn says about Yarn + ESM. My conclusion is different though: Don‘t use Yarn if it doesn’t support ESM/Node’s resolving algorithm.

[wooorm]

As I understand it (I backported recent changes just now), loaders are implemented above resolve: https://github.com/nodejs/node/blob/3ebe7535b456771078368ba72345f8d2e16ce695/lib/internal/modules/esm/loader.js.

I don’t think there’s anything to do in this project.

[bencmbrook]

Yeah, I think I'm just out of luck with this combo. Thanks again for looking into this.

I am wondering if Yarn PnP can be solved in your actual use case though.

My use-case is specifically getting the remark VSCode extension working in a PnP repo, so I have little control over the Node flags. But I might be able to yarn patch this library (or something else on the load-plugin, unified-engine, remark-language-server chain). Maybe a patch to switch load-plugin to use require.resolve or something.