Discussion: New “ESM by default” mode

[GeoffreyBooth]

Building off of #49295 (comment), we’re considering a new mode where all of the current places where Node defaults to CommonJS would instead default to ESM. I think this should be enabled by flag, and once it ships we can potentially ship a separate binary where it’s enabled by default; and/or we can someday make the new mode Node’s default in a semver-major change.

The use cases solved by such a mode are (and I’ll edit this comment to update this list as people think of more):

• I want to write ESM syntax by default without needing to opt into it, such as via package.json file or file extension or --input-type=module.

• I want to write what are typically called shell scripts in ESM JavaScript, where I can have a single extensionless file anywhere on my disk that uses ESM syntax and it doesn’t need a nearby package.json file or a symlink or wrapper file in order to run as ESM.

• I want consistency in how the CLI handles references to files; currently --import accepts URL strings but the main entry point must be a path string (and therefore can’t be a data: URL or https: URL, such as to work with --experimental-network-imports).

• If this mode becomes enabled by default in a future version of Node, when using that version of Node I want to be able to start a new project and write ESM syntax without needing to take any steps to opt into enabling ESM support.

To handle these use cases, the changes that this mode should make are (and likewise I’ll edit this to reflect consensus):

• When a .js file is outside of any defined package.json scope, because there are no package.json files in its folder or any folders above it, it will be treated as ESM JavaScript.

• When a .js file is in a package scope where its nearest parent package.json lacks a "type" field, it will be interpreted per the conditions we establish in Discussion: In an ESM-first mode, how should a package.json file with no type field be handled? #49494: when the package scope is under a folder named node_modules, the file will be treated as CommonJS; otherwise it will be treated as ESM.

• When an extensionless file is outside of any defined package.json scope, it can be run as an entry point or imported per the conditions we establish in Discussion: In an ESM-first mode, how should extensionless entry points be handled? #49431: if the file begins with the Wasm magic bytes, it will be evaluated as Wasm, and likewise for any other future file types with defined headers that don’t conflict with JavaScript; otherwise it will be evaluated as ESM JavaScript. (Still subject to --experimental-wasm-modules until that stabilizes.)

• Input from STDIN or --eval would be treated as ESM unless --input-type=commonjs is passed. Or in other words, the default value of --input-type would flip from commonjs to module.

• The CLI will parse the main entry point as an URL string, similar to how the value of --import is parsed. process.argv[1] will not be parsed by the CommonJS loader.

This new flag will be named per the discussion in #49541.

Prior art: #32394, #49295, #49407

@LiviaMedeiros @nodejs/loaders @nodejs/wasi @nodejs/tsc

[mcollina]

I don't think this is feasible to change the default if package.json is present due to the massive amount of modules available on npm.

My 2 cents on how to solve this in a much easier way: let's make npm init add the necessary field in package.json.

[GeoffreyBooth]

I don’t think this is feasible to change the default if package.json is present due to the massive amount of modules available on npm.

I’ve considered this and I don’t think it’s a blocker. As discussed in #49295 (comment), there are many potential solutions, the simplest of which is to just crawl through subfolders adding "type": "commonjs" to package.json files that lack it. I don’t think this needs to be a worry or something we need to deal with at this point.

Also, the proposal isn’t to go straight to changing Node’s default. That will be months if not years away, once this flag has shipped for a long time and issues have been resolved. The time for considering the impact of changing Node’s default is once this flag has been available for a while and we can see how it works. We may never change Node’s default, and that’s fine; this flag and mode has merit regardless. The point of mentioning that this might lead to changing Node’s default is so that we design with that potential goal in mind, so that whatever we add has a clear way to opt out if/when it becomes the new default.

[bmeck]

@GeoffreyBooth how does adding that field work for people using other languages/processes that spawn child processes for existing codebases? I'm mostly curious since it is a much larger breaking change than most we see and affects all sorts of runners of node applications in non-simplistic ways.

[GeoffreyBooth]

how does adding that field work for people using other languages/processes that spawn child processes for existing codebases? I’m mostly curious since it is a much larger breaking change than most we see and affects all sorts of runners of node applications in non-simplistic ways.

First off, adding a flag isn’t a breaking change. The breaking change would only be if we make the flag the new default behavior, which I don’t think we need to worry about just yet. It’s premature to consider larger ecosystem effects of making this new mode the default when we haven’t designed, much less implemented, the flag to enable this new mode.

The flag is no different than achieving these effects via module customization hooks (the changes that the hooks can achieve, anyway). Just as other languages/processes wouldn’t know how you’re customizing the files in your project when using hooks, they wouldn’t know whether your project is meant to run under this new flag or not. This is a problem we already have, and it’s on those other tools to provide a way for the user to signal to them that they should expect Node to be in this new mode, like how tsconfig.json is used. Shipping the flag long before we change Node’s default allows the ecosystem time to adapt.

One thing we can do to help ease the transition is to start making the "type" field mandatory. As in, you don’t need to have a package.json file, but if you have one, it needs a "type" field. Then at least for applications, which are our dominant use case, there’s still a deterministic way for tools that work with apps to know how all the files within that application should be interpreted. We could make this one of the effects of the flag, so that in the new mode we need explicit package.json types.

[bmeck]

@GeoffreyBooth I'm not opposing the flag in any way, I'm just concerned about the default swap which has been attempted in the past in #32394

[bmeck]

Then at least for applications, which are our dominant use case, there’s still a deterministic way for tools that work with apps to know how all the files within that application should be interpreted. We could make this one of the effects of the flag, so that in the new mode we need explicit package.json types.

I'm more skeptical on this due to files being loaded for config/init like ~/.*rc files outside of the application directory.

[joyeecheung]

I think the idea of having multiple releases in general is not great - if we want to do it, builds with pointer compression enabled would’ve been a thing already and then we would have another headache about what combinations of releases we want to maintain….

A runtime flag that makes ESM the default sounds good to me though, why can’t we just have that and allow it in NODE_OPTIONS? That should be enough for the use cases mentioned in the OP already

[GeoffreyBooth]

I’m more skeptical on this due to files being loaded for config/init like ~/.*rc files outside of the application directory.

There’s an infinitely long tail of ecosystem tools and libraries that don’t support whatever latest and greatest features we ship. Several years ago there were presumably tools that relied on the fact that .js files were CommonJS, and those tools needed to update to check for the type field before making that assumption; this is no different. If we limit ourselves by what today’s tools expect out of Node, then we’d never be able to change anything, even additive/non-breaking changes. Users can make the choice of deciding to use this flag or deciding to use a tool that's incompatible with it.

I think the idea of having multiple releases in general is not great

What do you mean by “multiple releases”?

[joyeecheung]

What do you mean by “multiple releases”?

multiple binaries, e.g. if you can have node-esm, then you can also also have node-pc (pointer compression), then you can also have a combination of node-esm-pc, then you can have…other combinations

[GeoffreyBooth]

What do you mean by “multiple releases”?

multiple binaries, e.g. if you can have node-esm, then you can also also have node-pc (pointer compression), then you can also have a combination of node-esm-pc, then you can have…other combinations

Yes, that’s perhaps an argument against shipping an alternate binary (see #49407). I think we should worry about the flag first before contemplating binaries; I think the flag stands on its own, and maybe there’s a solution for a binary with that flag enabled that doesn’t involve us needing to ship or maintain it, but we can worry about it later.

A runtime flag that makes ESM the default sounds good to me though, why can’t we just have that and allow it in NODE_OPTIONS? That should be enough for the use cases mentioned in the OP already

Shebangs can’t include flags on many platforms. See #49295 (comment) and linked and following comments. I don’t know if shell scripts have much need for pointer compression, but assuming they don’t, then the shebang case is an argument for a separate binary, as shell scripts being unable to specify flags is a particular constraint that only a separate binary can solve, and hopefully this is the only Node option that such scripts need control over.

Anyway to your larger point, yes, let’s ship the flag first and go from there. Many platforms do support flags in shebangs, and there are other use cases solved by the flag, so we’re still making a lot of progress just with the flag by itself at first.