Synthesize namespace records on CommonJS imports if necessary

[DanielRosenwasser]

TL;DR

Flags like --allowSyntheticDefaultImports will just work without extra tools like Webpack, Babel, or SystemJS.

TypeScript will just work with the --ESModuleInterop flag without extra tools like Webpack, Babel, or SystemJS.

See the PR at #19675 for more details.

Background

TypeScript has successfully delivered ES modules for quite some time now. Unfortunately, the implementation of ES/CommonJS interop that Babel and TypeScript delivered differs in ways that make migration difficult between the two.

TypeScript treats a namespace import (i.e. import * as foo from "foo") as equivalent to const foo = require("foo"). Things are simple here, but they don't work out if the primary object being imported is a primitive or a value with call/construct signatures.
ECMAScript basically says a namespace record is a plain object.

Babel first requires in the module, and checks for a property named __esModule. If __esModule is set to true, then the behavior is the same as that of TypeScript, but otherwise, it synthesizes a namespace record where:

1. All properties are plucked off of the require'd module and made available as named imports.
2. The originally require'd module is made available as a default import.

This looks something like the following transform:

// Input:
import * as foo from "foo";

// Output:
var foo = __importStar(require("foo"));

function __importStar(mod) {
    if (mod && mod.__esModule) return mod;
    var result = {};
    if (mod != null) {
        for (var k in mod) {
            if (Object.prototype.hasOwnProperty.call(mod, key)) {
                result[k] = mod[k]
            }
        }
    }
    result.default = mod;
    return result;
}

As an optimization, the following are performed:

1. If only named properties are ever used on an import, *the require'd object is used in place of creating a namespace record.

   Note that this is already TypeScript's behavior!

2. If

   1. a default import is used at all in an import statement and
   2. if the __esModule flag is not set on the require'd value, then

   A fresh namespace record whose default export refers to the require'd value will be created in place of the require'd value.
   In other words:

   // This TypeScript code...
   import { default as d } from "foo";
   d.member;
   
   // Would become this CommonJS JavaScript code...
   var foo = require("foo");
   var foo_2 = foo && foo.__esModule ? foo : {default: foo};
   foo_2.default.member;

Note that there is no optimization for never using the default. This is probably because you'd need alias analysis and could never be sure that someone else would use the default.

Proposal

I believe that we'd serve both communities well if we decided to adopt the aforementioned emit.

Drawbacks

Performance & Size Impact

This certainly impacts the size and readability of TypeScript's output.
Furthermore, for large CommonJS modules, there could be a speed impact to keep in mind - notice that these synthesized namespace records are not cached at all.

Tools already do this for us

Those who've been using tools like Webpack 2 and SystemJS have been getting this functionality for a few months now.
For example, if you target ES modules, Webpack 2 will already perform this behavior.
Taking this strategy makes it an opt-in for users who care about the interop behavior.

I mainly bring this up because I want to immediately bring up the counterpoint.
While these tools are definitely central to many modern web stacks, they won't cover

1. Vanilla Node.JS users
2. React Native Packager users
3. Browserify users (duh)
4. Users whose test code doesn't run through these tools

[DanielRosenwasser]

cc @TheLarkInn @hzoo @guybedford

[aluanhaddad]

This is fantastic.
I really want to see this happen.
Thank you very much.

I want to add that one benefit of this is that it simplifies documentation, tutorials, and other learning materials that currently either have long explanations regarding these differences or, as is more often the case, only show an import usage example that works for either the TypeScript community or for the Babel community but not for both.

[guybedford]

This is great to hear and I'm really glad TypeScript is looking to support this interop.

@DanielRosenwasser do you perhaps mean to correct if the __esModule flag is set on the require'd value, then to read if the __esModule flag is not set on the require'd value, then above?

Just to share the current shift SystemJS has had to make wrt interop - I know this has probably been discussed ad nauseam, but just to voice my opinion again...

Namespace lifting for CommonJS

NodeJS doesn't seem to have any plans of treating CommonJS modules as namespaces, and instead simply providing them as default exports, due to all the edge case concerns here of calling getters etc. While I know this is gambling on a possibility, I think it is worse to gamble that NodeJS will get over this restriction any time soon in its adoption of ES modules. SystemJS took this hit already (while using __esModule as a fallback method), and it has definitely affected users.

Just thought it is worth mentioning as I do think it would benefit TypeScript to double down on the syntax import x = for CJS as this allows a great way to syntactically separate CJS requires and their named exports, without assuming that CommonJS modules can be magically converted into ES module namespaces. import x from 'x' for CJS can still be allowed, while gradually deprecating import {x} from 'cjs'.

Edit: and perhaps Node core modules can be treated as exceptions here due to the expectation that in the conversion to ES modules, NodeJS core would permit named exports for these.

[aluanhaddad]

Just thought it is worth mentioning as I do think it would benefit TypeScript to double down on the syntax import x = for CJS as this allows a great way to syntactically separate CJS requires and their named exports, without assuming that CommonJS modules can be magically converted into ES module namespaces. import x from 'x' for CJS can still be allowed, while gradually deprecating import {x} from 'cjs'.

Just a thought, but I wonder if it might be worth adding another --module target to TypeScript, say CommonJSStrict, that only supports the set of operations that Node currently plans to support for interop.
I realize this would add complexity to TypeScript, but it might be easier than deprecating named imports. If Node adopts named imports, then that feature could be added to the new module target.

I've been recommending the import = syntax to people targeting CommonJS for a while now.

Also, CC @unional, @Kovensky

[unional]

Thanks @aluanhaddad, for notifying me about this wonderful issue. Why can't we add more than one 👍 to the OP? 🌷

I have also been recommending the import = syntax for a while. It works nicely in my use cases.

However, I want to bring up that unfortunately, it cannot be a long-term solution. The issue is browsers' eventual support of ESM.

When user target es6, the import = needs to be interop to import default as the require() function is not available in the browser without the help of any loader.

This means CJS -> ESM interop and a unify view of CJS <-> ESM is still needed. And it also means magically converting CJS -> ESM (for the dependencies of the compiling package) is still needed to be performed during compilation.

(edit: we should point this out to the nodejs folks as their decision on this prohibits code sharing on server and browser unless all dependencies are ESM)

Another major issue needs to be dealt with is typings and direct ts source code distribution.
As mentioned in #7398, the implicit capturing of the tsconfig in the typings is very problematic and could greatly complicate the implementation of this work and the adaptation of the new mode CommonJSStrict if introduced.
We should also look at that one to see if there is a solution to that or a migration path on updating all typings available.
Speaking of which, making typings to respect scope would be desirable, but that's another issue. 🌷

[unional]

cc @blakeembrey @basarat @felixfbecker

[DanielRosenwasser]

@unional I think that the .d.ts issue is something we'll have to deal with separately down the line. I've already been pushing new PRs in DefinitelyTyped that describe value-exported entities to use the export = syntax, and use import x = require('...') syntax.

@DanielRosenwasser do you perhaps mean to correct if the __esModule flag is set on the require'd value, then to read if the __esModule flag is not set on the require'd value, then above?

@guybedford Yes, thanks, I corrected my post.

Just thought it is worth mentioning as I do think it would benefit TypeScript to double down on the syntax import x = for CJS as this allows a great way to syntactically separate CJS requires and their named exports

@guybedford I agree, but unfortunately, a lot of people are extremely turned off by that idea for philosophical reasons. The existence of import foo = require('...') makes complete sense and is actually useful, but my experience is people don't want to hear it.

Just a thought, but I wonder if it might be worth adding another --module target to TypeScript, say CommonJSStrict

@aluanhaddad that's something @mhegazy and I have discussed at some point, and I think it'll be on the table, but since nobody really knows what will happen in 6 months time, I think we should wait. At the very least, adopting this behavior will mean that there is a smooth migration path for users (as opposed to now where users can't get ready for a change like that at all without an external tool).

[DanielRosenwasser]

After our design meeting today, I don't think this is 2.4-bound, but what we're looking for is the best migration path we can conceive, along with a story that synchronizes with type declarations on DefinitelyTyped.

Current concerns are that the Babel emit does not allow namespace imports to be callable, which actually does break a lot of (technically invalid) code, such as the following:

import * as padLeft from "pad-left";
padLeft(10, "zup doug");

So a potential modification could be the following

function __importStar(mod) {
    if (mod && mod.__esModule) return mod;

    // For callable/constructable things, tack properties onto a function record
    // that passes arguments through to the actual import
    var result = !(mod instanceof Function) ?
        {} :
        function() { return mod.apply(this, arguments); };

    if (mod != null) {
        for (var k in mod) {
            if (Object.prototype.hasOwnProperty.call(mod, key)) {
                result[k] = mod[k]
            }
        }
    }
    result.default = mod;
    return result;
}

But we might also just be willing to have a new mode that goes with Babel's emit and breaks away from the old behavior in that respect.

[Jessidhia]

The behavior of import { x, y, z } from 'foo' is only correct if foo is an ES module. If it is a commonjs-exported object, it should look like a namespace with only a default export (i.e. x, y and z should be undefined regardless of whether foo's module.exports has those keys, and ideally be build-time errors).

All non-star-import emits should behave as if a hidden import * as _ref from 'cjs' was done; with the this value erased if the direct imports are called as a function; e.g. importedDefault() -> (0, _ref.default)(). The star import is basically imported.__esModule ? imported : Object.freeze({ __proto__: null, [Symbol.toStringTag]: 'Module', default: imported }).

Any of the changes to make TypeScript any closer to ESM/CJS interoperation spec compliance will be breaking because of how off the TypeScript implementation is from the actual proposed interoperation semantics. This would also need stricter semantics than --allowSyntheticDefaults -- a star import from CJS is a namespace with a default export, not the CJS exports with a self-referencing default.

[felixfbecker]

A lot of libraries have started to do stuff like this:

module.exports = BigNumber
module.exports.BigNumber = BigNumber
module.exports.default = BigNumber

To support nicer ES6 imports à la

import { BigNumber } from 'bignumber.js'
import BigNumber from 'bignumber.js'

the majority of libraries is not using a transpiler and will have to support CommonJS/Node imports, so it would be very sad if this is broken