Proposal: Preprocessor Directives

[yortus]

This proposal is based on a working implementation at:
https://github.com/yortus/TypeScript/tree/preprocessor-directives

Problem Scenario

Whilst the TypeScript compiler has some options to control what to emit for a particular source file, it currently has limited support for controlling what is scanned into the compiler from a particular source file. A source file is either included in its entirety, or not at all.

This makes some scenarios difficult. For instance, there are two default lib files, lib.d.ts, and lib.es6.d.ts. A program may be compiled against one, or the other, or neither. If only some ES6 typings are desired, then either they must all be taken (using lib.es6.d.ts), or a custom set of core typings must be maintained with the project.

Even if the core lib files were further subdivided into smaller modular files that could be selectively included in a build, problems would remain. For instance, consider an ES6 built-in, WeakMap, which has members that use the ES6 Symbol spec and the ES6 Iterable spec. How many files must the WeakMap definition be broken into to keep the lib files feature-modular?

Related scenarios have been discussed in other issues:

• (Conditional compilation #4671) Conditional compilation
• (Preprocessor directives proposal #3670) Preprocessor directives proposal
• (Proposal: Conditional Compilation #3538) Proposal: Conditional Compilation
• (Support conditional compilation #449) Support conditional compilation

This proposal focuses on the lib.d.ts modularity problem, since that was the core requirement for the related proposal (#4692) that motivated the working implementation.

Workarounds

With regards to compiling core typings for only some ES6 features, some workarounds are:

• compile with --target ES5 and selectively add ES6 typings in separately maintained files (eg from DefinitelyTyped).
• compile with --target ES6 and be careful to avoid referencing unsupported ES6 features (the compiler won't issue any errors).
• compile with --noLib and manually maintain custom core typings in your own project.

For other scenarios, such as supporting DEBUG builds or IOS builds etc, a common practice is to use a single codebase with conditional execution to differentiate behaviour in different environments. This generally works well, except if conditional require(...)s are needed, as these can be a problem for some module systems that statically analyse module dependencies.

Proposed Solution

This proposal adds a new kind of syntax for preprocessor directives.

Preprocessor Directive Syntax

A preprocesor directive:

• begins with a '#' followed by the directive identifier, eg #if, #endif
• must be the first non-whitespace element on its line
• must not be followed on the same line by anything other than whitespace and/or a single-line comment

Valid:

    #if DEBUG
#endif // end of debug section

Invalid:

}  #if DEBUG
#if X  foo(); #endif 

Directives with Arguments

A preprocessor directive may take an argument. If so, the argument appears after the directive identifier on the same line. The directive identifier and its argument must be separated by at least one whitespace character.

Under this proposal, only #if takes an argument, which must be an identifier. An extended proposal may expand argument syntax to include preprocessor symbol expressions.

Contextual Interpretation

If a syntactically valid preprocessor directive appears inside a multiline comment or a multiline string, it is not considered a preprocessor directive. It remains a normal part of the enclosing comment or string.

/*
   The next line is NOT a preprocessor directive
   #if XYZ
*/

Preprocessor Symbols

A preprocessor symbol is an identifier used with some directives (only #if under this proposal). Preprocessor symbols have no values, they are simply defined or undefined. Under this proposal, the only way to define a preprocessor symbol is using the define compiler option (see below).

Preprocessor symbols are in a completely separate space to all other identifiers in source code; they may have the same names as source code identifiers, but they never clash with them.

#if DEBUG
#if SomeFeature
#if __condition
#if TARGET_HAS_ITERABLES

#if and #endif

The #if and #endif preprocessor directives signal the start and end of a block of conditionally compiled source code. #if must be given a preprocessor symbol as an argument. #endif takes no argument. Each #if in a file must have a matching #endif on a subsequent line in that file.

When the TypeScript compiler encounters an #if directive, it evaluates its preprocessor symbol against a list of defined symbols. If the symbol is defined, then the TypeScript scanner continues scanning the source file normally, as if the directive was not present. If the symbol is not defined, then the compiler skips all the source code down to the matching #endif without compiling it.

#if...#endif blocks may be nested. Inner blocks will be unconditionally skipped if their outer block is being skipped.

#if HAS_FOO
foo();
#endif

#if HAS_FOO
foo();
#if HAS_BAR
foo() + bar();
#endif
#endif

The define Compiler Option

Preprocessor symbols may be defined at compile time using the define compiler option, which takes a comma-separated list of identifiers.

tsc --define FOO,bar

{
    "target": "es6",
    "define": "DEBUG,__foo,ABC"
}

Possible Extensions

This proposal is limited to a small set of features that are useful on their own, but may be expanded.

In particular, #if and #endif alone are sufficient to address the lib.d.ts problem described above, as evidenced in the working implementation of #4692. The ability to nest #if...#endif blocks effectively allows logical ANDing of preprocesor symbols.

Possible extensions include:

• #define and #undef directives to add/remove preprocessor symbols from within source code. However the question of symbol scope then arises.
• Unary and binary expressions involving preprocessor symbols, such as ! (logical NOT), && logical AND, and || logical OR.

Backward Compatibility, Design Impact, Performance, etc

• There is no impact on existing TypeScript projects. The preprocessor directives only modify the compiler's behaviour if they are explicitly used.
• Preprocessor directives introduce new language syntax, but do not affect any existing language features.
• There is negligable impact on compiler performance.
• The hash symbol is only used in shebang trivia at present. There is no syntax clash. But could it be used for something in a future ES standard?

Remaining Work and Questions

The working implementation implements preprocessor directives in the TypeScript scanner, since they are really a filter on incoming tokens. This works fairly well for this limited proposal, but questions arise if extensions were added:

• if #define/#undef were added, how should preprocessor symbols be scoped? Global? Per file? The scanner has very limited control over scoping. Hence currently preprocessor symbols are all globally scoped and provided using a compiler option (or internally generated by the compiler).
• Supporting expressions as preprocessor arguments would add complexity to the scanner, as it would need to parse a mini grammar for the expression, which would be atypical at the lexer stage. But certainly doable.

[weswigham]

I, personally, don't like the idea of adding a preprocessor to TS. #if is almost a language unto itself, and embedding a DSL into the compiler is usually something which should avoided. If you really want a c-style preprocessor, you don't need to integrate it into TS, IMO. (It is just string manipulation, after all.)

But a preprocessor like this will never be in ECMAscript, and it follows no ECMAscript patterns or semantics - so there's no real hope that support for it would broaden in the JS community in the future. While preprocessor directives give great flexibility and power, there's no runtime JS equivalent for them - you're just using TS as a... well... JS preprocessor.

The most common argument I've seen an argument for why people "need" a preprocessor is when they refuse to do dependency injection for dependencies which change with compilation target (test primitives, etc) - using them as a shortcut, a hack, for avoiding refactoring their code - so maybe I'm just a bit bitter at all the bad code I've seen.

@mhegazy mentions here that TS would be more likely to take a [Conditional(bool)] style change, and I can see why - it follow proposed semantics for an ES feature (decorators) and, conceptually, can be desugared into a runtime check, but also be used to indicate to the compiler that it can perform additional typesystem optimizations/removals/additions.

[danquirk]

Historically we have been very resistant to preprocessor directives, especially control flow type directives (in contrast to a more declarative thing like #Deprecated). That said, it's clear that we do need to investigate better ways for people to make the compiler aware of which subset of their runtime environment they're targettng at design time.

[yortus]

Since this proposal is really aimed just at supporting more granular lib.d.ts typings (as proposed in #4692), it could be limited to just that case - i.e. an internal detail of the compiler that only affects how the default lib is scanned during compilation. The syntax would then also be an internal implementation detail, and could be changed to a ///<...> style or [Conditional(...)] style for example.

@weswigham would a [Conditional(bool)] / decorators style approach work with in purely ambient source file like lib.es6.d.ts? I understood it's a runtime mechanism, but the problem here is to be conditional about types rather than values. Or are you suggesting it could be extended for that purpose?

I agree that preprocessor directives have very weak appeal in a JavaScript environment, which has adequate alternatives. However with regard to conditional inclusion of core types to match real-world mixed ES5/6/7 targets (like in #4692), there are no alternative language-level mechanisms. The only other way is to split the core typings into many small files and work out which ones to pass to the compiler, which is just conditional compilation by another means.

[weswigham]

@weswigham would a [Conditional(bool)] / decorators style approach work with in purely ambient source file like lib.es6.d.ts? I understood it's a runtime mechanism, but the problem here is to be conditional about types rather than values. Or are you suggesting it could be extended for that purpose?

With some dead code elimination, yes. If you enable babel's dead code elimination alongside decorators it... kinda does the right thing right now. (Some bits get too aggressively culled while others aren't pruned as much as they could. Both features are experimental, so I'm not expecting perfection.)

For example, you can do this with babel right now:

var debug = true;
function LogSetter(descriptor) {
  if (debug) {
    if (descriptor.set) {
      let oldSet = descriptor.set;
      descriptor.set = function() { console.log(...arguments); oldSet.call(this, arguments); }
    }
  }
}

class Foo {
  @LogSetter
  set name(value) {
    this._name = value;
  }
}

(new Foo()).name = "yes";

When debug = true, your decorator is emitted like so:

function LogSetter(descriptor) {
  if (descriptor.set) {
    (function () {
      var oldSet = descriptor.set;
      descriptor.set = function () {
        console.log.apply(console, arguments);oldSet.call(this, arguments);
      };
    })();
  }
}

and false:

function LogSetter(descriptor) {}

With a little bit more intelligence/fixup it could know to omit the decorator entirely. But anyways, what I'm getting at is that using normal JS constants to control your runtime changes and some good dead code elimination with actual JS conditions can cover "conditional compilation" most if not all of the time. And the best part about it is that you can disable dead code elimination and inspect what options caused what branch eliminations at runtime, making it much easier to debug than a tangle of #if pragmas. TS has a couple extra bits with the extra things we can decorate which need a bit more intelligence on the TS side (For example, if a property decorator always returns {} effectively nullifying the property, does the property need to stay on the type at compile time for type checking? Probably not.), but it is by and large the same concept.

[yortus]

But anyways, what I'm getting at is that using normal JS constants to control your runtime changes and some good dead code elimination with actual JS conditions can cover "conditional compilation" most if not all of the time.

Right, however the problem scenario presented in this proposal (selectively choosing parts of lib.d.ts files) seems to be one of those times that JS constants and dead code elimination won't help.

[yortus]

@weswigham to add to my above comment, as long as other solutions can be found to 'selectively choosing parts of lib.d.ts files', that don't require adding a proprocessor to tsc, I agree that preprocessor directives are unlikely to have other compelling use cases. The other scenarios all seem to have adequate alternatives either at runtime or as you mention through things like dead code elimination.

[tinganho]

@weswigham the purpose of dead code elimination is to optimize code bases by eliminating dead code. Debug code for me isn't dead. Dead code for me is an unused public method in a library. And the trick you describe is being branded as an ugly hack, people began to use the method you described along with different minifiers a long time ago. Dead code elimination has no directives. So when programmers see that code, how do they know it will be eliminated? I think programmers wants a distinct syntax to handle preprocessing.

I still think preprocessor directives makes a lot of sense. JS has the most widest platform usage of all programming languages, and when we programmers now program code using compilers instead of vanilla JS. I think a feature like preprocessor directives makes a lot of sense to target different platforms. This feature will never likely be implemented in JS so it fits TS also.

[rhencke]

D did this very well without preprocessor directives, using what they called 'static if'.

It has roughly the same semantics as 'if', but can be used in type definitions, and is evaluated at compile-time. It cannot slice through arbitrary text, like preprocessor macros can.

See: http://dlang.org/version.html#staticif

[weswigham]

Some languages can accomplish that with hygienic macros. I'm not sure we'd want hygienic macros, though.

[rhencke]

That's true. I was thinking something simpler than most hygienic macros are.
I suppose what I'm suggesting is something more like this hypothetical compile-time if:

// ambient
interface App {
    quit();

    if (false) {
        eraseHardDrive();
    }

    if (Platform === "OS X") {
        setApplicationMenu(menu: Menu);
        getApplicationMenu(): Menu;
    }

    if (Version >= 3) {
        magicVersion3Function();
    }
}

(In this case, Platform and Version are identifiers whose values are known at compile-time, through hand-wavy magic.)

I don't have an exact proposal in mind - my hope in bringing this up is more that if this feature does make it in some form, it can be done leveraging the syntax and concepts already present in TypeScript.

[RichiCoder1]

@rhencke reading that, I'm vaguely reminded of Dlang's static conditions (http://dlang.org/version.html).

Could have something like:

interface App {
    quit();

    static if (false) {
        eraseHardDrive();
    }

    version(OS_X) {
        setApplicationMenu(menu: Menu);
        getApplicationMenu(): Menu;
    }

    version(3) {
        magicVersion3Function();
    }
}

[rhencke]

@RichiCoder1 Not a coincidence. ;) See: #4691 (comment)