Proposal: extension map in package.json

[ljharb]

Internally in the fork, we seem to have an extensionFormatMap, which is something exactly like:

{
  '__proto__': null,
  ".js": "cjs"
  '.cjs': 'cjs',
  '.js': 'esm', // obviously this shouldn't be the default here, but that's a separate discussion
  '.mjs': 'esm'
}

My suggestion is a field in package.json, "extensionsMap". The implementation would roughly be like this:
change https://github.com/nodejs/ecmascript-modules/blob/da0667d4b0c4fcd26b595a5af3fafd6d743cd2d1/lib/internal/modules/esm/default_resolve.js#L75 to:

const packageFormatMap = doMagicToGetThis();
// ^ this would throw if any keys in the object did not match `/^\.[^\.\s]+$/`, say
// ^ this would also either throw, or ignore, if any of the values
//   in the object were not a recognized parse goal
//  (ie, an existing value in `extensionFormatMap`)

const actualFormatMap = { ...extensionFormatMap, ... packageFormatMap };
let format = (legacy ? legacyExtensionFormatMap : actualFormatMap)[ext];

Now, you've got a mechanism to add new extensions, override existing ones, and even set them to null, perhaps, to block them from being loaded - all per package boundary. If this were to land, I'd have no objection to a "type" or "mode" field, as either mutually exclusive sugar for this field, or, as sugar for also merging in { '.js': 'esm' }, or whichever.

Thoughts?

[ljharb]

cc @GeoffreyBooth

[GeoffreyBooth]

Thank you for proposing this.

It looks like you proposed this in #160 (comment) and there were objections to using file extensions on both sides of the mapping, because it’s confusing (what does .js represent? Not all users immediately consider it to be CommonJS, and even less so in the new implementation) and because of the potential for circular references. The end result of the discussion in that thread was that the group favored a map of extensions to MIME types.

I am also opposed to requiring users to use any secondary extension, whether .mjs or .cjs, to represent some of the JavaScript files in their codebase. It’s a very common pattern to have ESM .js files in src/ and CommonJS .js files in dist/. Users should be able to continue this practice.

[ljharb]

It wasn't that "the group favored", it was that some members of the group favored.

What do you mean "secondary extension"? .mjs is the primary ESM extension, full stop.

Either way, you can continue that practice with this proposal by adding a package.json inside src, or dist, or both, that alters the extensions mapping to whatever you like.

[devsnek]

@MylesBorins @bmeck there was another thread where myles posted a suggested interface which brad then built upon. I've not been able to find it but if either of you remember what it was i think it would be a good thing to track here as it was similar.

[MylesBorins]

if I recall my proposal was for loaders. the idea would be that we could give canonical names to our built in loaders esm, cjs, native, wasm, json etc. and tie an extnesion to them. although I didn't have the dot in it

An MVP could just be exposing the translators we already have e.g.

https://github.com/nodejs/ecmascript-modules/blob/modules-lkgr/lib/internal/modules/esm/translators.js

which is what the object in https://github.com/nodejs/ecmascript-modules/blob/da0667d4b0c4fcd26b595a5af3fafd6d743cd2d1/lib/internal/modules/esm/default_resolve.js#L18-L23 gets mapped to

so if we were to expose the built in loaders in a @nodejs/ namespace (that's for you @ljharb 😇) I could represent some of our current defaults

--mode=modules

"loaders" {
  "js": "@nodejs/loaders/esm",
  "mjs": "@nodejs/loaders/esm",
  "cjs": "@nodejs/loaders/cjs"
}

--mode=cjs or perhaps --mode=legacy 😇

"loaders" {
  "js": "@nodejs/loaders/cjs",
  "mjs": "@nodejs/loaders/esm",
  "cjs": "@nodejs/loaders/cjs",
  "json": "@nodejs/loaders/cjs",
  "node": "@nodejs/loaders/cjs"
}

but you could also extend this in a bunch of different ways

"loaders" {
  "js": "@standard-things/esm/ng",
  "mjs": "./my-custom-loader.mjs",
  "ts": "typescript-loader-from-npm",
}

We also talked about making it composabe. I just added * as an example below as perhaps a pre processing step for all loader? @nodejs/loaders/reload could be a step that could pipe a module record to another loader based on meta data in the module record.

--mode=modules

"loaders" {
  "*": ["@nodejs/loaders/legacy-extension-resolution", "@nodejs/loaders/reload"],
  "jsx": ["./jsx-2-esm", "@nodejs/loaders/esm"]
}

The above api is all somewhat dependent on brads custom loader implementation, which I have not looked at recently. We would also likely have to refactor our internal loaders quite a bit to allow composability, but we could likely get an MVP together that only support a single loader per extension.

[guybedford]

Just to take a step back here - it would really help to understand what the exact use cases / problems / user stories are that this feature solves.

Is this purely to support TypeScript in Node.js like Deno?

[ljharb]

@guybedford no, it’s to enable anyone to choose any extensions they want for anything - without that full ability, i don’t think there’s any benefit (in fact i think it’d be harmful) to allow anyone to use anything but the default extensions.

Put another way, this mechanism is the only way (modulo sugar/shortcuts) i think you should be able to have ESM in a .js file.

[guybedford]

no, it’s to enable anyone to choose any extensions they want for anything

Why is this a goal? Who is "anyone" here? I would really appreciate it being phrased in terms of a genuine use case.

[ljharb]

"Consistency" is a genuine use case - making "ESM in .js" a special, magical, first-class citizen is confusing, and incredibly short-sighted. People have come up with all kinds of use cases for require.extensions that nobody anticipated at the time, and the concept of "loaders" has grown out of that.

Basically, with CJS, users have a default extension for each parse goal, but there's a mechanism to override or augment it - if we're taking that ability away from them for ESM, we're saying "we know better than you do how extensions should work" - and in that case, why not force .mjs for ESM for everyone?

If, instead, we want to say "you, the user, have the ability to use extensions as you see fit, but we're providing defaults", then every extension and parse goal should be equally customizable.

[guybedford]

if we're taking that ability away from them for ESM, we're saying "we know better than you do how extensions should work" - and in that case, why not force .mjs for ESM for everyone?

We're not taking this ability away. The resolver can be intercepted with custom behaviours just like require.extensions supports. And actually a custom resolve is a far more general way than require.extensions supports.

If, instead, we want to say "you, the user, have the ability to use extensions as you see fit, but we're providing defaults", then every extension and parse goal should be equally customizable

This is exactly what we're doing. Users can override the resolver if they want to and inject this customization.

I guess my primary concern is that when something is in the package.json that means it is now being published to npm with those customizations, and I can install an npm package that I don't necessarily know is making these customizations. Now say I want my http server to serve to the browser, but it won't know what MIME type to use for the module that is now stored at file.module that my server doesn't understand but Node.js has understood without telling me because we added a deep customization to the pacakge.json that was invisible to me the user.

Similarly, as mentioned many times before, I'm very concerned about users shipping custom languages to npm uncompiled. This creates a performance and tooling problem that is "invisible" if everything just worked after npm install pkg && node -e "import('pkg')". Only after many packages installed do I then realise I'm compiling 10000 lines of TypeScript on every Node execution.

I guess I just want to understand the user use cases that drive this need, so we can do this in a way that doesn't become a viral customization that will start to impinge on the health of the ecosystem.

[devsnek]

Now say I want my http server to serve to the browser

you need to use a bundler for this case because of cjs and node-specific apis and etc. the idea of node_modules somehow being magically browser safe is absolutely untenable.

I guess I just want to understand the user use cases that drive this need, so we can do this in a way that doesn't become a viral customization that will start to impinge on the health of the ecosystem.

maybe we can use gzemnid to find packages that modify require.extensions. i think the goal here is to provide better scoping so one package doesn't break another, which can happen with the current design.

another part is that the current --type api is pretty terrible. this is a pretty good alternative imo.

[ljharb]

@guybedford that same thing applies to a "type" or "mode" or "exports" field in package.json - you always have to know the possibilities, and build your tooling to accomodate them - otherwise, we can't allow any deviation from the defaults (which includes, writing ESM in anything that doesn't end in .mjs).

[guybedford]

@guybedford that same thing applies to a "type" or "mode" or "exports" field in package.json - you always have to know the possibilities, and build your tooling to accomodate them - otherwise, we can't allow any deviation from the defaults (which includes, writing ESM in anything that doesn't end in .mjs).

Certainly, but "type": "module" in package.json does not create a pattern for publishing packages on npm that will lead to me unwittingly installing code that compiles at runtime.

[ljharb]

I'm not sure what you mean by "compiles at runtime"; there's nothing about this proposal that forces that, and as for "generating the import map", that's also something npm could optimize by doing at publish time.