Skip to content
This repository was archived by the owner on Sep 2, 2023. It is now read-only.
This repository was archived by the owner on Sep 2, 2023. It is now read-only.

Patterns for interoperability #139

Closed
@demurgos

Description

@demurgos

Edit: This post ended up pretty long, I hope it depicts a relatively complete picture of the current state of native CJS/ESM interop.

Introduction

Hi,
Following the current discussions around defining a better term instead of "transparent interop" (#137, #138), it seems that most of it revolves around allowing libraries to migrate to ESM without impacting the consumer ("agnostic consumer"). I'd like to do a summary of the migration path I see given the current state of the discussion. I'll base it around "patterns" enabling libraries to safely migrate to ESM.
This relates to the feature Transparent migration (#105) and the use-cases 18, 20, 39, 40.

I'll leave aside the discussion around detecting the module type: commonJs (CJS) or ESM. To indicate the type of the module, I will either include cjs or esm in its name or use file extensions (.js for CJS, .mjs for ESM). Other module types (.json, .node, .wasm, etc.) are not discussed (they can be reduced to CJS or ESM).

I'll use a Typescript-like notation in some places to explain the types of the values.

I expect the modules to move forward from CJS to either CJS+ESM or ESM-only and won't focus on moving backward from ESM to CJS.

I'll focus on solutions, without loaders or @jdalton's esm package. In the rest of this post, esm always refer to "some module with the ES goal", not the esm package.


Here are my assumptions regarding the various ways to import the modules:

  • require("cjs") continues to work unchanged.
    If module.exports has the type Exports, it returns Exports.
  • import ... from "esm" and import("esm") works as defined by the spec: statically import bindings or get a promise for a namespace object.
    If the ESM namespace has the type Ns, import("esm") returns Promise<Ns>.
  • import ... from "cjs" exposes a namespace with a single export named default. It has the value of module.exports in cjs.js.
  • Similarly, import("cjs") returns a promise for a namespace object with a single key default set to the value of module.exports. It has the same behavior whether it is used from CJS or ESM.
    If module.exports has the type Exports, import("cjs") returns Promise<{default: Exports}>
  • require("esm") returns a promise for the namespace of esm. Synchronous resolution is discussed later (but from what I understood it is not likely to happen due to timing issues).
    If the ESM namespace has the type Ns, it returns Promise<Ns>.
// By abusing Typescript's notation, we have:
require<Exports>("cjs"): Exports;
require<Ns>("esm"): Promise<Ns>;
import<Exports>("cjs"): Promise<{default: Exports}>;
import<Ns>("esm"): Promise<Ns>;
import * as mod from <Exports>"cjs"; mod: {default: Exports};
import * as mod from <Ns>"esm"; mod: Ns;

Today, Node only supports CJS. It means that we have a CJS lib and a CJS consumer. We want to move to ESM lib & ESM consumer. In the general case, converting both modules at the same time is not possible (for example they are in different packages). It means that the transition will go through a phase where we have either ESM lib & CJS consumer, or CJS lib & ESM consumer.

It is important to support both cases to allow libraries and consumers to transition independently. Creating a dependency between the lib and consumers transition was the main failure of the Python 2-3 transition and we want to avoid it. Python finally managed to transition by finding a shared subset between both version. Similarly, the Node transition from CJS to ESM may need to pass through an intermediate phase where an API uses the intersection between CJS and ESM to facilitate the migration.

Specifically, I am interested in the following two goals:

  • A library can switch internally from CJS to ESM without breaking its consumers.
  • A consumer can switch from CJS to ESM and get expected values when importing its dependencies.

The first point is about enabling the migration of libs, the second about the migration of consumers.
Both are important even if this post is more intended for libs.

It helps to further break down the requirements for lib migrations:

  • CJS-safe lib migration: A lib module can switch between CJS and ESM without breaking CJS consumers.
  • ESM-safe lib migration: A lib module can switch between CJS and ESM without breaking ESM consumers.
  • safe lib migration: A lib module that can switch between CJS and ESM without breaking any of its consumers (CJS or ESM).

When a consumer moves from CJS to ESM, I consider that he gets expected results if one of the following is true:

  • module.exports in CJS becomes the default export in ESM. It keeps the same value:
    // main.js
    const foo = require("./lib");
    // main.mjs
    import foo from "./lib";
    import("./lib")
      .then(({default: foo}) => { /* ... */ });
  • module.exports in CJS is a plain object, its properties become ESM exports. They keep the same values:
    // main.js
    const {foo, bar} = require("./lib");
    // main.mjs
    import {foo, bar} from "./lib";
    import("./lib")
      .then(({foo, bar}) => { /* ... */ });

The consumer expects that its migration will happen as one of the two cases above. Any other result is surprising. This is what consumers expect today: --experimental-modules introduced the first case, Babel and Typescript started with the second case. The actual way to migrate is left to lib documentation or consumer research. The consumer is active at this moment: we want to reduce its overhead but a small amount of work can be tolerated. Any other result when the consumer moves from CJS to ESM is surprising: we need to avoid it. Especially, we need to avoid returning values of different types when there are strong expectations that they'll be the same.

From a library point of view, here is a migration path that allows a progressive process:

  1. Current CJS lib. Its API may not allow it to do a safe migration (without breaking its consumers). This is the case of most libraries today. For example, a CJS lib exposing a function as its module.exports cannot move to ESM-only without breaking its CJS consumers.
  2. Breaking change to an API allowing a safe migration. This change should future-proof the lib API, implementing this API should not require the use ESM ideally.
  3. Patch update to internally migrate from CJS to ESM.

This path emerged during the discussions as something that we would like to support. It enables to dissociate the API changes from the migration. It allows libraries to prepare for ESM before native ESM support.

Once a lib reached ESM and most of its consumers migrated, it may decide to do a breaking change and drop the compat API if maintaining compat it is no longer worth the cost (depends on the lib...).
The library may be initially authored using ESM and transpiled to CJS. In this case the step 3 corresponds to stopping the transpilation and directly publishing the ESM version.


Pattern CJS-safe ESM-safe CJS API ESM API Consumer migration Lib migration Lib tooling Uses mjs + js
Default export No Yes N/A {default: Api} N/A OK No
PWPO Yes No Promise<Api> N/A Unsafe OK No
ESM facade Yes Yes Api Api OK No Optional Yes
Promise wrapper + facade Yes Yes Promise<Api> Api OK OK Optional Yes
Dual build Yes Yes Api Api OK OK Required Yes
Default Promise Yes Yes Promise<{default: Promise<Api>}> {default: Promise<Api>} OK OK No

default export

The lib replaces its CJS module.exports by export default in ESM.
This pattern enables the migration of lib only if its consumers already use ESM.

  • CJS-safe migration: NO
  • ESM-safe migration: Yes
  • Safe migration: No (breaks CJS consumers)
  • The consumer gets expected results when moving to ESM: N/A, this pattern is available to the lib only if its consumer already uses ESM

This pattern relies on Node's ESM facade generation for CJS when importing them from an ESM consumer.

// Given:
import<Exports>("cjs"): Promise<{default: Exports}>;
import<Ns>("esm"): Promise<Ns>;
// Your ESM consumer can agnostically access a value of type Api if:
type Exports = Api;
type Ns = {default: Api}
// This is also true for static `import` statements

Examples:

CJS lib:

// lib.js
module.exports = function() { return 42; };

Equivalent ESM lib:

// lib.mjs
export default function() { return 42; };

Example agnostic ESM consumer (=does not know the module type used by lib):

// main.mjs
import lib from "./lib";
console.log(lib()); // prints `42`, regardless of the module type of `lib`

Since exports is an alias for module.exports in CJS, the following are also equivalent:

// lib.js
module.exports.foo = "fooValue";
module.exports.bar = "barValue";
// lib.mjs
const foo = "fooValue";
const bar = "barValue";
export default {foo, bar};

Agnostic ESM consumer:

// main.mjs
import lib from "./lib";
console.log(lib.foo);
console.log(lib.bar);

The default export pattern allows library authors to have a common subset between their CJS and ESM implementation. Once they moved to ESM, they can do a minor update to extend their API using named exports. This may be useful to provide a more convenient access to the properties of the default export. The previous example can be extended as:

// lib.mjs
export const foo = "fooValue";
export const bar = "barValue";
export default {foo, bar};

ESM consumer

// main.mjs
import lib, {foo, bar} from "./lib";
console.log(lib.foo);
console.log(lib.bar);
console.log(foo);
console.log(bar);

As mentioned at the beginning, this pattern allows libraries to migrate without breaking its consumers ONLY IF ITS CONSUMERS ALREADY USE ESM. This is the primary pattern available today with --experimental-modules.
This means that the ecosystem migration using this pattern would have to start at the consumers and move up the dependency tree. (If you want to avoid breaking changes). It's good to have this option but it is not enough for the ecosystem to move quickly: it requires the lib to control its consumers.
This case is still relevant. Internal projects can use this to update: migrate the consumer, update the lib API, migrate the lib, repeat.
After thinking more about it, this pattern is also relevant in situation where you can rely on transpilation at the lib and consumer side. I mostly see it for front-end related frameworks, for example for Angular strongly encouraging Typescript. The lib can be authored using this pattern and transpiled to CJS, a consumer can then configure its build tool to import this lib and automatically get the default export (in Typescript, use esModuleInterop with allowSyntheticDefaultImports, I think that Babel has something similar). This is not a true ESM migration because it still uses CJS under the hood, but the source code should be compatible.

Promise-wrapped plain object

module.exports is a Promise for a plain object equivalent to an ESM namespace. I'll abbreviate it as PWPO.

  • CJS-safe migration: Yes
  • ESM-safe migration: NO
  • Safe migration: No (breaks ESM consumers)
  • The consumer gets expected results when moving to ESM: NO 🚨

I mention this pattern here because it was discussed, but I currently consider it as an anti-pattern. Using it enables a CJS-safe migration but if the consumer uses ESM, it breaks in suprising ways.
This ultimately creates a "race-condition" between the lib and consumer when they both try to move to ESM. To move transparently to ESM, the lib must assume that all of its consumer use CJS. If a consumer migrates before the lib, there will be breakage.
This pattern is useful if it already applies to your CJS lib. Do not use it as an intermediate state because it will require you to go through 2 breaking changes (initial to PWPO, then PWPO to something esm-safe).

This pattern relies on the fact that require("esm") returns a Promise for the ESM namespace. By setting module.exports to a promise for a namespace-like object you can return the same value for require("./lib") regardless of the module-type of lib.

// Given:
require<Exports>("cjs"): Exports;
require<Ns>("esm"): Promise<Ns>;
// Your CJS consumer can agnostically access a value of type Api if:
type Exports = Promise<Api>;
type Ns = Api;

Given the assumptions above, require("esm") returns a Promise so it forces us to use promises in the compat API. ESM exposes a namespace object, you cannot export a function directly.

It means that your compat API for CJS consumers is a Promise-wrapped plain object.

CJS lib:

// lib.js
const foo = "fooVal";
const foo = "barVal";
module.exports = Promise.resolve({
  foo,
  bar,
});

Is equivalent to the ESM lib:

// lib.mjs
export const foo = "fooVal";
export const bar = "barVal";

Example agnostic CJS consumer:

// main.js
require("./lib")
  .then((lib) => {
    console.log(lib.foo);
    console.log(lib.foo);
  })

Here is another example exporting a single function, it uses an IIAFE for the promise:

CJS lib:

// lib.js
module.exports = (async () => {
  // Even if you can use `await` here, you should avoid it
  // The ESM equivalent is top-level await (it's still unclear how it would work)
  return {
    default () {
      return 42;
    },
  };
})();

ESM lib:

// lib.mjs
export default function () {
  return 42;
}

Agnostic CJS consumer:

// main.js
require("./lib")
  .then((lib) => {
    console.log(lib.default()); // Prints 42, regardless of consumer module type.
  })

This pattern is more complex but allows you to migrate your lib to ESM, if your consumer use CJS.
As discussed at the beginning, this pattern causes unexpected breaking changes if the consumer moves to ESM.

Here is what would happen if the consumer from the last example moves to ESM:

// main.mjs
import("./lib")
  .then((lib) => {
    console.log(lib);
    // If the lib uses CJS:
    // { default: Promise { { default: [Function: default] } } }
    // If the lib uses ESM:
    // { default: [Function: default] }
  });

If the consumer moves to ESM before the lib, two bad things happen:

  • The value of the default property on the result changes from Function to Promise<{default: Function}>. This is highly unexpected and very confusing.
  • Later on, when the library migrates to ESM thinking that it is safe, it will break the consumer: the returned value changes back.

The consumer would need to be very defensive when importing a lib using this pattern, this defeats the goal of allowing a simple migration: the consumer needs to intimately know the lib.

See "Promise-wrapped + facade" for an extension solving ESM-safety.

ESM facade

Use two files for the lib module: .js for the implementation and CJS API, .mjs for the ESM API.

  • CJS-safe migration: Yes
  • ESM-safe migration: Yes
  • Safe migration: Yes
  • The consumer gets expected results when moving to ESM: Yes, assuming both files are synced.

This relies on the resolution algorithm to pick the right file depending on the module-type of the caller. This means that it relies on .mjs or any other mechanism that lets you have two versions for the same module. You can start using this pattern without native ESM support (the .mjs will be ignored) but if you decide to add it afterwards, it may be a breaking change if default in .mjs does not have the same value as module.exports in .js.

You need both files in the lib:

// lib.js
module.exports = function() {
  return 42;
}

// lib.mjs
import lib from "./lib.js";  // The extension forces to resolve the CJS module
export default lib.default;

CJS consumer:

const lib = require("./lib");
console.log(lib()); // 42

The consumer can move to ESM and get expected results:

import lib from "./lib";
console.log(lib()); // 42

Your .mjs facade file can also expose named exports. I recommend to keep the default export set to the default value of the lib.

Lib:

// lib.js
const foo = "fooValue";
const bar = "barValue";
module.exports = {foo, bar};

// lib.mjs
import lib from "./lib.js";  // The extension forces to resolve the CJS module
export const foo = lib.default.foo
export const bar = lib.default.bar;
export default lib.default;

CJS consumer:

// main.js
const {foo, bar} = require("./lib");
console.log(foo);
console.log(bar);

The consumer can move to ESM and get expected results:

// main.mjs
import {foo, bar} from "./lib";
console.log(foo);
console.log(bar);
// main2.mjs
import lib from "./lib";
console.log(lib.foo);
console.log(lib.bar);

This relies on the fact the ESM lib module is picked first if the consumer uses ESM. lib.mjs acts as a static facade defining the named exports of the lib and allowing it participate in the ESM resolution.
I use this kind of pattern to also expose named exports in my own projects. It probably deserves more documentation (easier to author/consume than the promise-wrapped plain object).
This scenario is an important reason for both .js and .mjs.

This pattern is nice, but unless you use the .mjs file for named exports, it boils down to manually doing what Node is doing automatically with --experimental-modules. You still need to write your implementation in CJS and cannot migrate. The goal of this pattern is actually to update your API to expose named exports so your consumers can migrate to use named exports. To get rid of CJS, you'll need to go a step further and use the dual build pattern or promise-wrapped+facade.

Promise wrapped + facade, by @bmeck

This pattern is a combination of promise-wrapped and ESM facade. It fixes the ESM-safety issue of PWPO by handling the ESM imports in a facade.

  • CJS-safe migration: Yes
  • ESM-safe migration: Yes
  • Safe migration: Yes
  • The consumer gets expected results when moving to ESM: Yes, a bit different but still reasonable: CJS consumers must be async but ESM consumers can resolve either statically or dynamically (async)

Your lib needs an implementation file and two entrypoints (one for CJS and one for ESM):

// impl.js
module.exports = {foo: 42};

// lib.js
module.exports = Promise.resole(require("./impl"));

// lib.mjs
import impl from "./impl";
export const foo = impl.foo;

Example CJS consumer:

// main.js
require("./lib")
  .then((lib) => {
    console.log(lib.foo);
  });

Example ESM consumer

// main.mjs
import {foo} from "./lib": 
console.log(foo);

// main2.mjs
import("./lib")
  .then((lib) => {
    console.log(lib.foo);
  });

This pattern forces your CJS consumers to use an async API, this is less convenient.
The upside is that it allows you to do a safe migration of your impl file from CJS to ESM (you are not stuck with a CJS impl as with a simple ESM facade pattern).

Here is how to update your lib once you moved the impl to ESM (you can merge impl with lib.mjs):

// lib.js
module.exports = require("./lib.mjs");

// lib.mjs
export const foo = 42;

Dual build

This pattern is an extension of the ESM facade pattern. Instead of re-exporting the values defined in CJS, define your values both in CJS and ESM so both files are independent.

  • CJS-safe migration: Yes
  • ESM-safe migration: Yes
  • Safe migration: Yes
  • The consumer gets expected results when moving to ESM: Yes, assuming both files are synced, REALLY SYNCED.

Example lib:

// lib.js
const foo = "fooValue";
const bar = "barValue";
module.exports = {foo, bar};

// lib.mjs
export const foo = "fooValue";
export const bar = "barValue";
export default {foo, bar};

The consumers are the same as in the ESM facade:

// main.js
const {foo, bar} = require("./lib");
console.log(foo);
console.log(bar);
// main.mjs
import {foo, bar} from "./lib";
console.log(foo);
console.log(bar);
// main2.mjs
import lib from "./lib";
console.log(lib.foo);
console.log(lib.bar);

Given the current constraints, I feel that this is the pattern providing the best consumer experience: the lib can adopt ESM and drop CJS without breaking the consumers (assuming it leaves time for the consumers to move 😛 ) and does not depend on the module type of the consumer. The consumer gets expected results when migrating.
The obvious drawback is that both files MUST provide the same values to actually ensure the consumer can migrate without surprises. It means that the lib needs to use tooling for this use case. This should be easy to achieve if the source-code is transpiled using Typescript or Babel. If you are writing the files manually, it's best to avoid. This also means that this pattern requires you to keep using tooling for the duration of the transition, even if one of the goals of ESM was to allow more use-cases without tools. Native support by ESM will not affect the benefits of using Typescript but some teams may consider removing the Babel overhead.

This is the pattern I settled on for my personal projects, but I spent a lot of time tinkering with my build tools.

Default Promise

This pattern gives you full compat without relying on "js + mjs", at the cost of having a user-hostile API.

  • CJS-safe migration: Yes
  • ESM-safe migration: Yes
  • Safe migration: Yes
  • The consumer gets expected results when moving to ESM: Yes

CJS lib

module.exports = Promise.resolve({
  default: module.exports,
  foo: 42,
});

ESM lib

export const foo = 42;
export const default = Promise.resolve({
  default,
  foo
});

CJS consumer:

require("./lib").then({default} => default)
  .then(lib => {
    console.log(lib.foo);
  });

ESM consumer

import("./lib").then({default} => default)
  .then(lib => {
    console.log(lib.foo);
  });

I found it by combining the constraints of both "default export" and PWPO.
This pattern allows you to support any combination of lib and consumer module type, using a single file.

It's good to know that this exists, but the API is so bad (you need to await a promise twice) that I hope that nobody will have to use this. Still, it offers an escape hatch if resolution based on the consumer (mjs+js) is not available.

Transparent interop

Hehe, you'd like it. Unfortunately I don't know how to achieve it today, but at least I can give you a definition of a transparent interop pattern.
Transparent interop would be a library pattern (meaning the libraries may need to change their code) such that:

  • The lib can do a CJS-safe migration: the lib moves from CJS to ESM without breaking its CJS consumers
  • The lib can do an ESM-safe migration: the lib moves from CJS to ESM without breaking its ESM consumers
  • The consumers can migrate from CJS to ESM and get expected results:
    "Importing default is the same as the CJS module.exports" and/or "Named imports are the same as the CJS properties of module.exports"

If any of those is not achieved then we can't call it transparent interop and the migration will be measured in decades.
Ideally it would be simpler to maintain than "Dual Build" and less user-hostile than "Default Promise".

Forewords

Ecosystem migration bias

The current path to migrate the ecosystem from CJS to ESM has a dependency between the lib and consumer. The migration is biased in favor of the consumer: he can migrate more easily than its libraries.

Sync require("esm")

If we had sync require("esm"), the situation would be:

// By abusing Typescript's notation, we have:
require<Exports>("cjs"): Exports;
require<Ns>("esm"): Ns;
import<Exports>("cjs"): Promise<{default: Exports}>;
import<Ns>("esm"): Promise<Ns>;
import * as mod from <Exports>"cjs"; mod: {default: Exports};
import * as mod from <Ns>"esm"; mod: Ns;

A migration-safe solution exists using something like:

Ns = Api & {default: Api};
Exports = Api;

CJS lib:

// lib.js
const foo = 42;
module.export = {foo};

And the equivalent ESM implementation

// lib.mjs
export const foo = 42;
export default {foo};

You can use it this way:

// main.js
const lib = require("./lib");
console.log(lib.foo);
// If lib is CJS, `lib.foo` corresponds to `module.exports.foo = 42;`
// If lib is ESM, `lib.foo` corresponds to `export const foo = 42;` (enable by sync require(esm))
// main.mjs
import lib from "./lib";
console.log(lib.foo);
// If lib is CJS, `lib.foo` corresponds to `module.exports.foo = 42;`
// If lib is ESM, `lib.foo` corresponds to `export default {foo};`

But sync require("esm") is impossible due to timing issues.
Again, I'm hoping that someone can find a pattern for "transparent interop" without sync require or a way around the timing issues.

require("esm")

Edit: I wrote this before knowing about "promise-wrapper and facade", I am less worried about the use cases now.

I am not sure about the use case for an async require("esm").
It enables the Promise-Wrapped Plain Object pattern for CJS-safe lib migration, but using it this way is a footgun because of the surprising behaviors and breaking changes if the consumer uses ESM.

If you remove the "transparent interop" use case, I see:

  • I am a first party consumer and actually want to import an ESM module dynamically. What's the point of using require("esm") instead of import here? You already know specifically that the lib uses ESM, and if your runtime supports ESM then it also supports import(...).
  • You are a third-party consumer that needs to load modules dynamically, the module specifiers are provided to you and you don't know anything about the result. You need to work across various versions of Node (you have no control over it). Because of this, you cannot use import(...) reliably so you can backport it using Promise.resolve(require("esm")). But then you need to deal with a whole can of worms anyway to actually understand what's going on in the module. My example for this use case is a lib like mocha: it wants to import test specs that may be written in CJS or ESM, and has to work on versions where using import(...) throws an error. Actually, mocha had a PR to handle ESM: it simply used eval(import(specifier)). Still, we are talking about very specific use cases where I expect implementors to be familiar with Node's module system and already have to deal with edge-cases. require("esm") may be nice for them, but there are already workarounds.

I am not convinced that require("esm") has use cases that aren't better served by import(...), even considering interop. I'd be happy to hear more about it.

@jdalton's esm package and other tools

@jdalton did some great work with his esm package. I deliberately chose to not talk about loaders or other advanced features here, but until we get true native ESM everywhere using tools like this will definitely help. The situation is not that bad. It may require a bit more work by the consumer but at some point it's unavoidable.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions