Proposal: Granular Targeting

[yortus]

This proposal is based on a working implementation at:
https://github.com/yortus/TypeScript/tree/granular-targeting
To try it out, clone it or install it with npm install yortus-typescript

Problem Scenario

The TypeScript compiler accepts a single target option of either ES3, ES5 or ES6. However, most realistic target environments support a mixture or ES5 and ES6, and even ES7, often known in advance (e.g. when targeting Node.js, and/or using polyfills).

Using TypeScript with target environments with mixed ES5/5/7 support presents some challenges, many of which have been discussed in other issues. E.g.:

• (Support compile targets between ES5 and ES6 #4389) Support compile targets between ES5 and ES6
• (Normalize our lib files by compiler settings #4168) Normalize our lib files by compiler settings
• (New APIs added to lib.d.ts may break client codes. Allow duplicated members in interfaces? Make lib.d.ts overridable? #3215) New APIs added to lib.d.ts may break client codes. Allow duplicated members in interfaces? Make lib.d.ts overridable?
• (Using ES6 type default library when targetting ES5 output #3005) Using ES6 type default library when targetting ES5 output
• (for-of does not work with some DOM collections when target is ES6 #2695) for-of does not work with DOM collections when target is ES6
• (somewhat related: (Create DOM-Level specific dom-version.d.ts #2481) Create DOM-Level specific dom-version.d.ts)

In summary:

• The default lib either includes all ES6 types and properties, or none of them.
• Specifying --noLib and/or manually maintaining lib.b.ts files brings other problems:
  • separate core typings are a burden to maintain.
  • problems of missing symbols and clashing symbols.
  • burden of manually tracking fixes and additions made in the default libs.
• Targeting ES5 as the 'lowest common denominator' means some language features known to be supported cannot be used (eg generators in Node.js).
• Targeting ES6 (e.g. to take advantage of Node.js' support for many ES6 features) leads to further complications:
  • CommonJS modules won't compile, even though that's the only module system Node supports. (fixed by Support modules when targeting ES6 and an ES6 ModuleKind #4811)
  • The compiler emits ES6 even for features that are known not to be supported, which would fail at runtime.
  • Adding babel.js to the build pipeline adds complexity.

Workarounds

To achieve mixed ES5/ES6 core typings:

• specify --target ES5 and selectively add ES6 typings in separately maintained files (eg from DefinitelyTyped).
• specify --target ES6 and be careful to avoid referencing unsupported ES6 features (the compiler won't issue any errors).
• specify --noLib and manually maintain custom core typings in your own project.

To use ES6 features supported by the target platform

• specify --target ES5 and (a) accept that things will be down-level emitted, and (b) don't use features with no down-level emit yet (ie generators).
• specify --target ES6 and (a) convert everything from CommonJS to ES6 modules (fixed by Support modules when targeting ES6 and an ES6 ModuleKind #4811), (b) add babel.js to the build pipeline, and (c) configure babel.js to do either pass-through or down-level emit on a feature-by-feature basis.

Proposed Solution

This proposal consists of two parts:

1. Support for conditional compilation using #if and #endif directives, so that a single default lib can offer fine-grained typings tailored to a mixed ES3/5/6/7 target environment.

The conditional compilation part is detailed in a separate proposal (#4691) with its own working implementation.

1. A mechanism allowing the default lib to offer fine-grained typings tailored to a mixed ES3/5/6/7 target environment.

This is really an internal compiler detail, so the mechanism is open to debate. It just has to match the granularity supported by the new compiler options below.

The working implementation uses #if...#endif conditional compilation proposed in #4691. But this is overkill for this use case and seems unlikely to be considered.

Several other mechanisms have been discussed (summarized here).

2. Support for additional compiler options allowing the target environment to be described on a feature-by-feature basis.

Under this proposal, the target option remains, but is now interpreted as the 'baseline' target, determining which features the target supports by default. For instance, ES6 symbols and generators are supported by default if target is set to ES6 or higher.

The additional compiler options have the form targetHasXYZ, where XYZ designates a feature. These options are used to override the target for a particular language feature. They instruct the compiler that the target environment explicitly does or does not support a particular feature, regardless of what the target option otherwise imples.

The working implementation currently supports the following additional compiler options (all boolean):

• targetHasArrowFunctions: specify whether the target supports ES6 () => {...} syntax
• targetHasBlockScoping: specify whether the target supports ES6 let and const
• targetHasForOf: specify whether the target supports ES6 for..of syntax
• targetHasGenerators: specify whether the target supports ES6 generators
• targetHasIterables: specify whether the target supports ES6 iterables and iterators
• targetHasModules: specify whether the target supports ES6 modules
• targetHasPromises: specify whether the target supports ES6 promises
• targetHasSymbols: specify whether the target supports ES6 symbols

These options work both on the command line and in tsconfig.json files.

Example tsconfig.json Files and their Behaviour

A.

{
    "target": "es6",
    "targetHasModules": false,
    "targetHasBlockScoping": false,
    "module": "commonjs"
}

Emits ES6 JavaScript, except with CommonJS module syntax, and with let/const down-leveled to var. This might match a Node.js environment.

B.

{
    "target": "es5",
    "targetHasSymbols": true
}

Emits ES5 JavaScript, except with Symbol references emitted as-is, and with full type support for well-known symbols from the default lib.

C.

{
    "target": "es5",
    "targetHasPromises": true
}

Emits ES5 JavaScript, except with full type support for ES6 promises from the default lib. This would work in an ES5 environment with a native or polyfilled Promise object.

Backward Compatibility, Design Impact, Performance, etc

• There is no impact on existing TypeScript projects. The additional options and preprocessor directives only modify the compiler's behaviour if they are explicitly used.
• The preprocessor directives #if and #endif add new language syntax. No existing language features are affected.
• There is negligable impact on compiler performance.
• Only one default lib is needed (lib.es6.d.ts). It contains many conditionally compiled sections (ie with #if and #endif)

Remaining Work and Questions

• Support compiler options for more target features, e.g.:
  • template strings
  • classes
  • Map/Set/WeakMap/WeakSet
  • binary and octal literals
  • destructuring
  • default, rest, and optional parameters
• How granular could/should targets be? Feature support is naturally hierarchical. E.g. block scoping may be separated into (a) let, (b) const and (c) block-level function declaration. This is true of most features and their realistic implementations (the Kangax ES6 compatibility table has a three-level hierarchy down the left side).

[weswigham]

First off, I think @rbuckton has done/is doing some somewhat related work to this in a branch (along with other refactoring). So ping @rbuckton for some input.

Next up: This is probably just bikeshedding, but would it be possible to consider target: "ES3/5/6" to desugar to some collection of these flags, and then make each flag an option like

target: {
  "asyncAwait": false,
  "decorators": false,
  "arrowFunctions": true,
  "blockScoping": true,
  "forOf": true,
  "generators": true,
  "iterables": true,
  "modules": {"emit": "commonjs"}, //or any of our other options or "esm" for ECMAscript module
  "promises": true,
  "symbols": true,
  "templateLiterals": true,
  "destructuring": true,
  "defaultParameters": true,
  "namespaces": false
}

Where the true can additionally be a configuration object for the feature (as "modules" above, which could deprecate the top-level "module" flag), for example, if the runtime supports let but not const, it could be indicated there. (Otherwise, if having configurable features seems wrong, it could simply be an array of strings, like babel's whitelist argument.)

I'd rather like see the old target syntax get deprecated for a purely flags-based one. (Though I imagine the old style will still get used as a shortcut for certain bundles of features.)
Mostly because I feel like using the target: string style alongside all these top-level flags is building a huge amount of conditional complexity within the compiler (you must have seen the many places checking for ScriptVersion.ES6 and the module flag while implementing this), and would like to see the former go away in favor of just the more versatile feature flags scheme to help unify how that feature checking is done internally.

More seriously: I like the proposed end result of this, by and large, but I don't like that it was coupled to the preprocessor feature (for reasons stated in that issue). So I'll help look at alternatives - as far as the lib file issue goes, I think this comes with the how we're trying to represent the lib as dependent on configuration without allowing the standard lib to be configured to an acceptable degree. (We pretty much have two settings right now.)

An alternate solution would be allowing one to specify, internally with each feature, lib.d.ts parts for each, and parts which are dependent on other features to be included. Just like how .es6.d.ts is conditionally included by target right now - Meaning we'd specify lib.d.ts component flag dependencies in code in the compiler, rather than in the .d.ts file with directives.

On top of that, we have to think how this interacts with alternate stdlibs, like the webworker lib, which is actually a bit of a pain to use at present. (Since it's not targetable by the compiler, you include it like any other ref and then it excludes the stdlib.) If we could break down what we'd like to include in our target standard lib with compiler flags in the same way we could feature emit, then we would have all of that information in the project, and let it be granularly targeted as well (and be dependent on emit target flags if need be). For example we could add the compiler options:

"stdlib": {
  "environment": "browser", //or "worker" or "node" or a path to a ".d.ts" file
  "configuration": {  //Set of options used to build/include the correct ".d.ts" files
    "DOMLevel": 3,
    "canvas": true,
    "webGL": true
  }
}

The interesting thing about the stdlib is that from the compiler's perspective, it doesn't necessarily need to correspond to any files (though it does for simplicity's sake at present). See using string[] vs Array<string> to see what I mean (one's compiler intrinsic and defers to the other if possible, the other is stdlib). We could build the (higher-level, not string) contents of this .d.ts - having a stdlib factory rather than an actual stdlib file. (Though for go to definition it would need to be able to generate a file, ofc, just like VM "files" in the chrome dev tools)

Beyond that, it may be acceptable to include some kind of feature-dependency pragma within a triple-slash comment in an entire .d.ts, which sets some kind of conditional inclusion/potential error for the entire file... but I don't feel like that's necessarily the right direction to go with this solution.

[Arnavion]

I agree with @weswigham 's idea of having the feature flags be under their own hash instead of at the top-level, if only to not collide with other top-level properties.

One question - would newly introduced feature flags (say for ES2016 features as they get standardized) default to true or false?

• If true, since I cannot list future feature-flags in my project's current tsconfig.json, so by upgrading the compiler I would be opting in to new features that may not be available on my target platforms.
• If false, when would they start defaulting to true? One minor release later? One major release later?
• Perhaps they wouldn't default to any value. The compiler could require that all feature flags must be explicitly specified, and error out if they're not / prompt at the CLI for the new values. This does make it harder for someone using the CLI without a tsconfig.json, since their commandline would grow longer and longer with every new feature.

[weswigham]

@Arnavion Likely false unless a feature doesn't break back-compat when true (meaning it uses new syntax which doesn't change the interpretation of older code), at which point it would be decided on a per-feature basis, I imagine. Everything presently the default for an empty tsconfig would likely start as true. The defaults would likely be driven by backwards compatibility of config files until TS wanted to take a large breaking change.

[Arnavion]

(meaning it uses new syntax which doesn't change the interpretation of older code)

false is good, but to clarify - the back-compat consideration is not just whether existing TS code gets broken, but whether adding new TS code that uses newly available features is allowed by default or not.

Eg: Say TS 1.7 gets released with support for the proposed bind operator ::. If the switch were to default to true, someone may start using it in their project and not understand why it's not being downlevel-emitted, until they spelunked through the source or release notes to find the magic feature flag they must set to false.

[weswigham]

@Arnavion I think that's something we'd have to configure via flags - I mean, we error on async/await unless you pass the flag to compile it, same with decorators. :: would be the same way - we only recognize it as valid if we've been told to compile it. (Now, weather it should be recognized and not downleveled would need to be a configuration option on the feature - we don't do that at all right now except for features that we don't have a downlevel emit for, like generators)

[LPGhatguy]

I like the concept of the emit, but the addition of #if and #endif kind of scares me a little.

Instead of using conditional compilation in a base .d.ts file, there should just be extra .d.ts files included with each feature level.

[rbuckton]

In the past we discussed pre-processor directives like #if and the general consensus is that we'd like to avoid adding that to the language.

We have been considering future support for "design-time" decorators, which only affect the compiler (and would not be written to the output file):

• @@conditional("ES6") - The body of the function/method is elided unless the named parameter is supplied to the compiler. Other examples could include: @@conditional("DEBUG"), etc.
• @@profile("dom") - The decorated member's type information is only visible when the "dom" profile is selected. Other examples could include: @@profile("es6"), @@profile("webworker"), etc.
• @@obsolete("message") - Use of the decorated member in non-ambient code reports an error at compile time.

We haven't settled on a final design yet.

[yortus]

@weswigham @Arnavion I agree that targets would be better represented by (possibly nestable) sub-hashes. But looking at tsc's commandLineParser.ts, it is clearly oriented toward a flat list of top-level options with simple string/boolean values, which are then used to parse both the command line and tsconfig.json files. So rather than proposing to also re-engineer this mechanism, I took the pragmatic way of just adding top-level options. I think that also overhauling the compiler options mechanism in the same proposal might distract from the key concept of granular targeting. But it would certainly make a good proposal on its own. It does have its own key questions, such as how to specify hierarchical options on the command line, which at this point is equally as capable as tsconfig.json.

[yortus]

@Arnavion regarding default values: under this proposal if a particular option is not explicitly given, the default value is neither true nor false; it is determined by the target option, which is either ES3, ES5, or ES6. So for example, if your project does not explicitly specify an option for targetHasPromises, then the compiler will look at the target option. If that is >= ES6, then it will be as if you had specified targetHasPromises: true, otherwise, it would be as if you had specified targetHasPromises: false.

This means there is no danger of implicitly changing options when you upgrade the compiler.

@weswigham this is also a good reason for keeping (ie not deprecating) the target option. It provides a succinct baseline that imples the value for all the other targetHas... options if they are not explicitly overridden.

As is already the case, if you don't specify a target, the compiler picks ES3 for you. So a blank tsconfig.json file would imply target: ES3 (that is the current compiler behaviour), and hence false for all the other targetHas... options.

Alternatively if you set target: ES6 and nothing else, you would get true for all the ES6 targetHas... options, and false for any ES7+ targetHas... options.

[yortus]

@weswigham @LPGhatguy there no need for this proposal to be wedded to the #if...#endif proposal. The important thing is to get just the core typings needed for the target features specified.

Some solutions to this:

• #if...#endif directives as is currently proposed (probably overkill since we just need conditional inclusion just in the core lib files)
• one big lib.d.ts with triple-slash pragmas for conditionally including parts of the file as @weswigham suggested (effectively a form of conditional compilation)
• the design time decorators that @rbuckton is working on (also a form of conditional compilation?)
• having no physical lib.d.ts file but have the compiler internally piece together the definitions it needs somehow, as suggested by @weswigham (although I suspect the cleanest way to implement this would probably involve a physical file with conditional pragmas behind the scenes)
• Having many small core lib files for each feature as suggested by @LPGhatguy and elsewhere - but I think this is harder than it sounds due to feature inter-relationships.

[yortus]

@rbuckton:

@@Profile("dom") - The decorated member's type information is only visible when the "dom" profile is selected. Other examples could include: @@Profile("es6"), @@Profile("webworker"), etc.

If I understand correctly, this would work at compile time and in ambient contexts like lib.d.ts. Is it effectively a conditional compilation mechanism, or something else? Also could such a decorator be specified on a property within a type? That would be needed to fully modularize the core types.

Eg would something like this be possible?

@@profile("targetHasPromises")
interface PromiseConstructor {

    prototype: Promise<any>;

    new <T>(executor: (resolve: (value?: T | PromiseLike<T>) => void, reject: (reason?: any) => void) => void): Promise<T>;

    all<T>(values: ArrayLike<T | PromiseLike<T>>): Promise<T[]>;

    @@profile("targetHasIterables")
    all<T>(values: Iterable<T | PromiseLike<T>>): Promise<T[]>;

    race<T>(values: ArrayLike<T | PromiseLike<T>>): Promise<T>;

    @@profile("targetHasIterables")
    race<T>(values: Iterable<T | PromiseLike<T>>): Promise<T>;

    reject(reason: any): Promise<void>;

    reject<T>(reason: any): Promise<T>;

    resolve<T>(value: T | PromiseLike<T>): Promise<T>;

    resolve(): Promise<void>;

    @@profile("targetHasSymbols")
    [Symbol.species]: Function;
}