Create doc on how to support ESM and CommonJS when publishing modules

[mhdawson]

From discussion at the last collaborator summit.

This would help as the suggestion was that there were some easy ways to do this.

[ljharb]

Given that by far the easiest way is “only publish CJS”, and if you want named exports, “publish CJS transpiled from ESM” - what more would go in this document?

[lemanschik]

@ljharb for example that you should prefer only namedExports from esm and transpile to CJS there are a lot of clashes when it comes to typescript because it ads a extra resolve matrix that is not compat

a great example are .d.ts files which use export = Module all that can get avoided if we do only namedExports by default and avoid assigning anything else then string propertys to a Object on module.exports

so never do module.exports = function. that also saves a lot of tooling quircs i can write whole books about this cross issues that come up as soon as you bundle more then 1 single code base.

at present for example i am struggling on puppeteer and devtools-protocol also vscode makes a lot of trouble in general all the big code bases that have typescript in it are relativ blocked from ESM migration.

[ljharb]

@lemanschik i strongly disagree with that; a default export is what a module is; named exports are what module has, and most modules should be something. That’s a stylistic choice and not something node should be weighing in on. I, and the Airbnb style guide ftr, say that you should almost always do module.exports = function.

[lemanschik]

@ljharb they say so because of compat i am working since ages with ESM only and can report back that no one did produce correct ESM Builds that load in a correct ESM Environment. (Other then NodeJS)

[ljharb]

I have a number of projects that work perfectly compatibly with a build process, so I’m not sure about those reports.

[lemanschik]

@ljharb your again correct with build tools which correct the imports everything works at the end but the Nativ code that should work untranspiled and should even get patched at runtime does not work.

thats why i got rollup and typescript contributor only because of such issues.

[lemanschik]

Maybe it helps to show a true isomorphic cjs ESM module that uses a global

Load Isomorphic CJS that uses only dynamic import and assigns to global as ESM and CJS share the same global this works.

globalThis.module = { exports: {}}
import('cjs_esm').then(_m=> { // _m is empty as it is CJS without export keyword to keep it isomorphic
 const v = globalThis.module.exports
 delete globalThis.module.exports
 return v
});

cjs_esm.js

const moduleName = {
  getSomething: import(''); 
}
module.exports = moduleName;

note

NodeJS has some logic in it that blocks loading correct ECMAScript you can workaround that via going Buffer.toString("base64") data:text/javascript urls this way node does not know the filetype so it works like in the browser.

node 19.1+ got a --imports[] flag that makes above note obsolete.

[lemanschik]

This pattern is also nice when you work with CJS ESM NodeJS

my_esm.js

const fnName = () => {};
const moduleName = { fnName };

export { fnName }

my_cjs_wrapper

const moduleName = {
   async fnName() {
    return (await import('./my_esm.js')).fnName()
  }
}

note while the above shows only a single combination the general pattern is able to partial translate from sync to async for example if the bespoken module function only is sideEffect based anyway we can expose in the my_cjs still a sync static function call and simply call import fn()

also if the module was async so returning promise anyway we can directly return import + function call.

the import gets handled engine internal as if would do top level import in the same document i benchmark performance of v8 a lot.

[lemanschik]

Maybe i should also open some eyes for how big this problem is:

• https://github.com/microsoft/TypeScript/issues?q=is%3Aissue+is%3Aopen+moduleResolution

this is in general a 1:1 Representation of the real nightmare that is under the hood of bundlers which abstract this reality.

to be fair most of that is caused by NPM CJS TYPESCRIPT. Pure ESM always works and thats why it is so importent to go ESM only in the source code and then generate obscured builds but not code obscured and generate eg: use package.json to resolve to something diffrent depending on environment is a killer Matrix even with the new static string replace methods. even more obscured code.

creating a entrypoint per environment is 1000% more compatible and efficent.

Companys like Blomberg did invent tooling stacks to unify depdendencys and maintaing them is hell

I did by the way audit over 8k NPM Modules by Hand in the last years and that are only the once that where so interristing that i did even forked them to play around.

Priority 0 Issue that solves a lot

If this would land we could drop large parts of typescript and .ts in the ecosystem to reduce the resolution matrixes by N*N as there are even resolve matrixes that accept plugins. It is maybe the most importent proposal for the next 20 years or the general existence of ECMAScript as a lang. Else all will stay Typescript which is Incompatible to ESM by Default.

• https://github.com/tc39/proposal-type-annotations
  • History background: the problem got identifyed early this proposal got inited as strawman by the Typescript Dev Lead.
  • But we have no community scale and power to enforce the land fast.
  • I my Self did even push that into Oracle ECMAScript code bases to get earlier adoption there.
• Alternativ: when i would have the time i would create a .ts => js + JSDOC type transpiler and we could resolve a lot of problems.

Importent note

Even typescript generates better .d.ts files out of ESM modules when u use allowJs and checkJs you get typescript typechecking but working with real ESM. and if you inline types with jsdoc no additional typeresolve step is needed.

Even creating a new language because of existing design failures is discussed

with well known people like the creator of Main Engineers of v8, Creators of Dart Lang, Python, Typescript, JSON5, Fleece (Binary Append Only log JSON eq queryable)

This would not be on the list if there would be no clear need that can be solved by the community. But spread all the needed knowleg is also not as easy. And in General NodeJS has some Fundamentals that Block other Ecosystems.

other ecosystems need to replicate NodeJS to be Compatible to the resolve algos in Fact NodeJS created its own ECMAScript version. Not directly based on JSON as ECMAScript is they are based on the package.json which even made Rayn the creator of NodeJS leave the project.

[lemanschik]

Complex educational problems

loop, async, globalThis aka Context, Scopes, incremental js upgrading via "use strict" directive

all this maybe sounds short and clear but all that is a complex matrix and it has effects on the behavior of objects and everything is a object in ECMAScript/JS

• Support Node's ES Modules electron/electron#21457 (comment)

shows the education problem size as electron uses nativ sideEffects to detect the end of the first engine run and then throw events which is a common embedder way to do in ECMAScript engine the coders do not understand what that means and run into 1000+ problems that spread like a big tree from that fundamentals

[lemanschik]

This should get a list of problems maybe

This could be better adddressed when the Plugin Hook Loader API Exists and would get used without Package.json resolve or at last a none magical package.json resolve that is explicitly written in a reuseable loader.. As that exists in all other Engines also

• https://github.com/microsoft/playwright-java <= ships whole NodeJS to get Playwright running which has less then 5mb raw code in ESM none minified all comments included. so shipping graal-node (A NodeJS Fork to run nodejs ECMAscript on GraalVM) GraalVM Comiplier including all needed tools to run and build that takes ~2GB only Runtime not debug dev.
• - The whole graal-node wrapper exists only because of package.json and the nodejs cjs ecosystem. not because GraalVM has leaking functonality or that they need to support the well defined Node ABI in fact they have directly LLVM bindings no need for wrapper code and they can even execute ECMAscript more performant on most long time workloads in GraalJS without graal-node which is nodejs with a v8 replacement that uses graaljs
• https://www.bloomberg.com/company/stories/10-insights-adopting-typescript-at-scale/ shows a lot of problems with typescript that now get multiplyed into issues like
  • A Proposal For Module Resolution microsoft/TypeScript#50152
    • Which opens up node12 node14 nodeNext node16 module like resolutions. not because ESM is not working.
• The whole deno nodejs compat stack exists not because of the great nodejs abi also rust in fact again has LLVM directly it exists because of package.json and CJS nodejs compat.
  • npm specifiers roadmap denoland/deno#15960
• https://github.com/nodejs-mobile/nodejs-mobile a mobile fork of nodejs using the ios or android ECMAScript engine only to offer resolve package json support again its not the problem that Android or IOS got no api's to link too
• @ljharb one of the most active spec people i meet him all over the ecosystem pointed out
  • transpil everything to CJS to get it at last running
    • leaves us to solutions like we use them with the Monaco Editor we create a HTML Component Module as Iframe to isolate the CJS Context for security implementations and isolations. which would be again provided out of the box by ECMAScript
  • he also sayed creating CJS from ESM and that is also a valid conclusion that i also got everything else is doomed.
  • he pointed out that AirBnb forces anti patterns to please the bundler they do module.exports = function to re combine into a module later with a bundler see the ECMAScript spec for modules they consist only out of string propertys thats what they are a Module as he also pointed out everything should be something and in case of ECMAScript Modules it should always be a Module if it gets imported and not used via Require directly Require is more like the nativ import of a ECMAScript engine in case of NodeJS it applys around 5 or even more diffrent module types. Ranging from package.json modules to json modules to nativ code dlopen to CJS import JS.

Also interristing observation

• all none nodejs ECMAScript engines that depend on ECMAScript do not have any module reuse exchange problems.
• all none nodejs ECMAScript engine ship NodeJS or a NodeJS fork to load ECMAScript Modules written for NodeJS as a language.
  • It is importent to keep in mind ECMAScript is a UTF8 Char based language it should run without nativ code anywhere.
  • console.log('') should not require a 4gb stack of binary instructions. to build in a away that it is useable in many places out of the box.

Identified Fundamentals when incremental upgrading CJS to ESM

• noSideEffects everything with a sideEffect is not a Module its a Script.
• try to run the code isolated with "use strict" on top if it works it is a indicator that you can do a ESM Wrapper where you use import and correct the namedExports and then bundle that to get relativ clean ESM if the CJS code has no additional dependencies left over so is already like a bundle.
• make sure that there is no usag of this. outside Classes. functions that use this are highly danger later for code hardening and security analyzes Many people like the author of "clean code" a book that maybe all know here pointed that out. This security reasons are by the way a blocker for commercial adoption of ECMAScript in general.
  • see: above mentioned educational fundamental problems. As rule of thumb you need to be execution Context aware.
• You should treat JS/ECMAScript like lua with dynamic JIT Build abilitys. in Fact a lua and ECMAScript have many common and the amount of ECMAScript engines is exploding so it gets more and more a lua replacement.
• You should be aware of the fact that there will be no original Python 4 and the reasons for that.

Short Conclusion

package.json when used for anything else then referencing a main + dependencies is a bad thing. For Example the new import maps features overlap with ECMAScript import as soon as used you need to recode them the reuse the code. This could be directly done as additional js file that only injects the dependencie specifiers.

also the whole scripts section can be replaced by simply putting the same scripts into the folder and run node script.js and not npm run-script build which calles node build.js the list is endless. Then on Top the Require CJS breaks everything as it simply supports to much diffrent resolutions and module types.

Solution ways

the most easy compat fix for at last the require CJS problem would be to let require return module and not module.exports this way it would algin with the ECMAScript spec so it exports a object with string specifiers "exports":