.mjs extension trade-offs revision

[YurySolovyov]

I obviously can't speak for the whole community, but it seems like a lot of people are not happy with .mjs.

One of the main arguments to keep .js is that if we can detect 99% of cases where we CAN tell if is it CJS or ESM (or where we just know what to do), we may just call rest 1% edge cases and deal with it.

We can even come up with some linter rules and/or workarounds to simply teach people to do the right thing.

[bmeck]

I want to start off with the 99% fallacy here. We are talking about module graphs. If 1% of your graph is wrong it can affect 100% of your graph. Let us keep that in mind.

Lets also try and enumerate where things differ between CJS, Script, and ESM whenever we talk about the problematic situations.

Lets also try and enumerate use cases where those situation might occur whenever we accept or dismiss them as being valid or invalid.

[YurySolovyov]

Let's start with simple stuff.
I think one of the most common patterns in node is to require() a bunch of core modules and export either one object with some API, or just export a plain function:

// Full CJS
const fs = require('fs');
const path = require('path');

const api = function(...params) {
  // ... 
};

module.exports = {
  api: api
};

Which in ESM I guess translates to something like:

// Full ESM
import fs from 'fs';
import path from 'path';

const api = function(...params) {
  // ... 
};

export {
  api: api
};

Are there any problems so far in that particular example?

[bmeck]

Parsing (focus on when ESM is not a parse error but others are):

Source	Script	CJS	ESM
var arguments	global	local	parse error
var eval	global	local	parse error
import("x")	ok	ok	ok
import "x";	parse error	parse error	ok
export {};	parse error	parse error	ok
with({}){}	ok	ok	parse error
<!--\n	ok	ok	parse error
-->\n	ok	ok	parse error
0777	ok	ok	parse error
delete x	ok	ok	parse error
try {} catch (eval) {}	ok	ok	parse error
try {} catch (arguments) {}	ok	ok	parse error
(function (_, _) {})	ok	ok	parse error
eval = eval	ok	ok	parse error
arguments = []	ok	ok	parse error
implements	ok	ok	parse error
interface	ok	ok	parse error
let	ok	ok	parse error
package	ok	ok	parse error
private	ok	ok	parse error
protected	ok	ok	parse error
public	ok	ok	parse error
static	ok	ok	parse error
yield	ok	ok	parse error
return	parse error	ok	parse error
await	ok	ok	parse error

Eval differences (overlap parses and runs, but diff results)

Source	Script	CJS	ESM
this	global	module	undefined
var x	global	local	local
(function (){return this}())	global	global	undefined
(function (x) {x = 1; return arguments[0];})()	1	1	undefined
(function () {return typeof this;}).call(1)	"object"	"object"	"number"
var x = 0; eval('var x = 1'); x	1	1	0
__filename	global	local	global
__dirname	global	local	global
require	global	local	global
exports	global	local	global
module	global	local	global
arguments	global	local	global

Timing differences (not a problem, listed for posterity)

Source	Timing	Hoisted	Blocking
require('foo');	sync	no	yes
import "foo";	untimed (async generally)	yes	yes
import('foo');	async	no	no

Potential relevant use cases w/o ESM specific declarations:

In all cases where only import declarations are used import() may be used instead. This list really is about things that do not export values.

• Mutation of Globals
  • Polyfills
• Usage of Globals
  • Configuration of globals (source-map-support, some config scripts)
  • JSONP-like calling of globals
• Prefetching (wants to be unordered parallel so uses import())
• Init scripts (CLI etc.)
• REPL like text (either for local testing or things like vm.runInContext)
  • Anything really that uses the completion value of the text

Potential relevant use cases that may combine source texts

Relevant to when import or export declarations may be added/removed.

• Optimizers / Build Scripts
  • Inlining of dependencies
  • Concatenation of files
• Editing a file
  • Copy/Paste from internet

Potential relevant use cases w/o package.json

• One off Scripts
  • Bash-like utils
  • Education situations
• Configuration scripts
  • Shared .rc style files
• Movable scripts (npm bin scripts have similar but not same)
• Serverless-like single file situations
• Eval from CLI (STDIN, "-e", "-p", REPL)
  • including node myapp.js

Potential tooling that lacks package.json capabilities

• Scripts without JSON libraries
  • nvm in particular
• OS-like file associations
• Code editors

Potential relevant use cases w/o a file extension

• Eval from CLI (STDIN, "-e", "-p", REPL)
  • excluding node myapp.mjs

Potential tooling that lacks file extension capabilities

None known yet.

Mandates

Resolution

Resolution in ESM is URL based and should be 100% compatible in non-error cases with the web specification. This is known to have minor differences with CJS.

Forward Path

A path for CJS to being using ESM modules must exist.

1. It must be possible for CJS files to import() ESM.
   • Since this is available in the Script goal, this is automatically fulfilled.

Backwards Path

A path for ESM to be created that uses legacy or CJS files must exist.

1. It must be possible to import CJS files.
   • Some things such as Node core are unable to be upgraded to ESM due to design or usage differences. Other Modules such as meow use things that are not present/do not translate into ESM like module.parent (due to parallel loading, idempotency requirement, etc.).
   • Some things like "deep" linking is in use and necessary to support like "lodash/chunk"
2. There must be an upgrade path for ESM to be safe against CJS becoming ESM.
   • Generally achieved by ensuring CJS files are a facade with only a single default export
3. Mixed mode situations (both ESM and CJS in same app/package) must be supported. Note: again, there may be cases where code can never be updated to ESM.

ESM only future

It should be possible for the ecosystem to move to be ESM only for newly written code.

1. Whatever path is taken, it should be considered debt if files are still easily or accidentally able to be CJS.
2. Whatever path is taken, it should be ready for the so called "3rd goal" problem.
   • Relevant to WASM that is looking to use same stuff as JS
   • Could be relevant with other things that have been proposed like a "use pure" situation if syntax or semantic changes are required.

[bmeck]

@YurySolovyov can you clarify what you mean by "Are there any problems so far in that particular example?" I mean, they do act similar but aren't the same.

[YurySolovyov]

I mainly meant that they don't introduce ambiguity, right?
Even if we'll have to parse them >1 times, we can with 100% certainty tell which is which, right?

[bmeck]

I am not asking in this issue if ESM can fulfill a use case, I'm asking about use cases where ambiguity is possible / exists. Take for example a simple prefetching script:

import('./something-for-later-1')
import('./something-for-later-2')
import('./something-for-later-3')
import('./something-for-later-4')

Such text is either a Script, CJS, or a Module, but could rely on something pretty easily that causes it to cease functioning:

import(`${__dirname}/something-for-later-1`)

[bmeck]

@YurySolovyov for

const fs = require('fs');
const path = require('path');

const api = function(...params) {
  // ... 
};

module.exports = {
  api: api
};

We can make a good guess that it is CJS if we parse for require() but not all CJS uses require(). We need to define what mechanism are you using to perform these guesses.

[bmeck]

It also would need to ensure no local variable named require exists.

[bmeck]

Also in theory that CJS can parse and eval just fine in Module.

[YurySolovyov]

Will the absence of import/export be a better sign of CJS than trying to parse requires ?

[bmeck]

If import and export declarations do not exist, it could be either ESM, [Script], or CJS. It is better to say import and export declarations (not import()) signifies something that currently is only able to parse in ESM.

[bmeck]

Detecting CJS really won't work due to amount of overlap in that direction, detecting [something that is only] ESM is possible though.

[YurySolovyov]

So we can't just always start with ESM and then fallback to CJS if that failed?

Looking at all these parse errors in the table above, I don't think many of them are useful or just make a lot of sense in general.

I have a question though about let, yield, return and await: is this about top-level ones or what?
I can't imagine a module system where you can't use these in function's body.

[bmeck]

No, default needs to be CJS since that is what existing backwards compat needs.

I have a question though about let, yield, return and await: is this about top-level ones or what?
I can't imagine a module system where you can't use these in function's body.

return works in CJS at top level.

let, yield, await, etc. are reserved words in ESM but not in the other goals.

Looking at all these parse errors in the table above, I don't think many of them are useful or just make a lot of sense in general.

Indeed! thats why https://github.com/bmeck/UnambiguousJavaScriptGrammar went to TC39!

Eval errors are much more insidious. Too many chats going on right now for me to complete.

[bmeck]

To note as well: the guess cannot change over time. Once you ship the guessing mechanism it will stay backwards compat (so if something guesses CJS, it will always guess CJS even 10 years from now)