Expose public API for transformation.

[rbuckton]

This PR contains the following changes:

• TransformationContext is now public:
  • getEmitResolver and getEmitHost are still marked /*@internal*/.
  • Added assertions to ensure transformation hooks are only modified at the right times during transformation.
• TransformationResult is now public.
• Transformer is now public.
• visitNode(), visitNodes(), and visitEachChild() are now public.
  • NOTE: visitEachChild() does not recursively visit TypeNode nodes, as they have so far not been necessary for emit transformations.
• visitLexicalEnvironment(), visitParameterList(), and visitFunctionBody() helpers are now public.
• Added the ability to attach synthetic leading and trailing comments to a node.
• Added a customTransformers parameter to Program.emit(), to run custom transformations before and after the main transformation pipeline.
• Added a getCustomTransformers() method to LanguageServiceHost to allow implementers to supply custom transformations to be run before and after the main transformation pipeline.
• Added a transform() function in services that can be used to perform arbitrary transformations on source files.
  • NOTE: As the built-in transformers require the EmitResolver and EmitHost, they are not run as part of transform(), as it does not have access to these. If you require this use case consider using Program.emit() with the customTransformers parameter, or create a LanguageService from a custom LanguageServiceHost that implements the getCustomTransformers() method.

Fixes #13764
Fixes #13763

[mhegazy]

remove.

[mhegazy]

//CC @alexeagle, @chuckjaz, @mprobst, @rkirov and @evmar would like to get your feedback on this change. this should allow tools like https://github.com/angular/tsickle to inject a transom either before or after the other TS transforms are applied.

//CC @ivogabe, this should supersede #11561

[evmar]

Thanks for thinking of me! I intend to look at this but I have had a lot of other work recently. If you can't wait I also don't think it'd be too harmful to land this and let us poke at it in a release. I'm not sure you have a mechanism for clearly tagging "experimental" API though.

[rkirov]

Unordered observations from my interaction with the API. Apologies if some stem from my misuse or misunderstanding of the API:

• API is subtle, we can use some minimal documentation. I am confused about hints, partiallyEmittedExpression, and ran into errors of the sort Lexical environment is suspended when playing with the visit* utilities.
• Copying the tests included, I did a mini-tsickle using purely addSynteticLeadingComment. The real implementation would not be so simple, because we always have to check and augment an existing jsdoc. That is the transformer will almost always have to modify the AST, and not just augment.
• Completely failed to write the tsickle pass that rewrites 'export *' (needed because closures module system does not suport that) - https://github.com/angular/tsickle/blob/master/src/tsickle.ts#L402 . Despite context.enableSubstitution(ts.SyntaxKind.ExportDeclaration), I do not see onSubstituteNode being called on any such nodes.

[rbuckton]

@rkirov While I still have some work to do on the API documentation, here are a few pointers based on your bullet points:

• hint - TypeScript uses the same Identifier node for identifiers that are part of an expression and identifiers that are part of a declaration. When performing substitution, it is often important to know which context you are in. You should never modify the hint that is passed in to the onEmitNode or onSubstituteNode hooks, but you can inspect it to understand the context you are in:
  • EmitHint.SourceFile - Only used for a SourceFile node, this hint is used by the emitter to control how comments and source maps are emitted at the top and bottom of a source file. Generally you won't need to use it.
  • EmitHint.IdentifierName - Only used for an Identifier that is the name of a function or class declaration, this hint is used by the emitter to control how source maps are emitted for the name.
  • EmitHint.Expression - Used when emitting an expression, primarily helps to distinguish an Identifier that is in an expression position vs. one used in a declaration.
  • EmitHint.Unspecified - Used for every other node.
• PartiallyEmittedExpression - This is a synthetic node added by transformations to apply comments and source maps for nodes that are partially removed during a transformation. For example, when TypeScript elides a TypeAssertionExpression, we replace it with a PartiallyEmittedExpression to preserve comments and source maps for the original expression.
• visitParameterList/visitFunctionBody - When visiting a function-like declaration, we introduce a new lexical environment for the purposes of hoisting generated variable or function declarations. Due to how lexical environments and scoping rules work in JavaScript, the new lexical environment is introduced when we visit the parameter list of a function-like declaration, and is exited after the end of the function body. To ensure lexical environments are nested correctly, we can suspend any modification of a lexical environment until some point at which it is resumed, allowing us to ensure the correct state is maintained when updating the tree. It is not required to use these helpers when visiting a function, and despite their general utility I may consider removing them from the public surface area to avoid confusion. Generally you want to use visitNode, visitNodes, and visitEachChild.
• onSubstituteNode - It depends on how you are using the transforms API. This hook provides just-in-time substitution during the final print phase, and is primarily used to substitute a relatively small subset of nodes to avoid an expensive full walk of the tree during every transformation. Since this runs during the print phase, it is quite possible some other transformation may have already replaced the ExportDeclaration with some other statement.

[rkirov]

Thanks @rbuckton, I created mini implementations of all the transformations we want to do in tsickle and they seem to work great.

The final two questions:

• how can one remove leading comments (tsickle will need to remove existing ones, before adding the new one)
• how would transformers interact with source maps generation?

and one observation, TS internal API are still not strictNullChecks compatible, I had to use a lot of undefined as any when using ts.create* :)

[rbuckton]

@rkirov:

• how can one remove leading comments (tsickle will need to remove existing ones, before adding the new one)

There is an exported function setEmitFlags that you can use for this purpose. It only affects comments read from the source file, not synthesized comments:

ts.setEmitFlags(node, EmitFlags.NoLeadingComments);

• how would transformers interact with source maps generation?

There is an exported function setSourceMapRange that you can use for this:

// range is a TextRange, either an existing node or some other `{ pos, end }` value.
ts.setSourceMapRange(node, range);

[pflannery]

@mhegazy

The decision whether to elide the module import or not is done by the checker before all the transformations have been done.

When you say "before all transformations have been done", do you mean before custom transformations are executed?

I would like to include new source files in one of my custom transformers.
My current transformer replaces module specifiers but I can't find an API to notify the compiler to reprocess new import\referenced files during a transform

[mhegazy]

I would like to include new source files in one of my custom transformers.
My current transformer replaces module specifiers but I can't find an API to notify the compiler to reprocess new import\referenced files during a transform

Transforms, all of them, happen after the checking phase has happened. you can not add new files at this point.

[EisenbergEffect]

@mhegazy Did this ever get documented?

[mhegazy]

We still have that on our todo list. you can find a lot of samples though in https://github.com/Microsoft/TypeScript/tree/master/src/compiler/transformers

[SerkanSipahi]

@EisenbergEffect right :)

@mhegazy i had written lot of babel-compiler plugins for javascript and postcss but now im coding for a while (because of new company) with typescript and im missing these documentation for typescript.

@mhegazy when will you guys start with the documentation?

We really need documentation like these:
babel:

• (user handbook) https://github.com/jamiebuilds/babel-handbook/blob/master/translations/en/user-handbook.md
• (plugin handbook) https://github.com/jamiebuilds/babel-handbook/blob/master/translations/en/plugin-handbook.md
• (babel types) https://github.com/babel/babel/tree/master/packages/babel-types
• (playground compiler api) https://astexplorer.net (there is no typescript transformer)

postcss:

• https://github.com/postcss/postcss/blob/master/docs/writing-a-plugin.md
• http://api.postcss.org/

[zwwtj2014]

hello @mhegazy ,

The declaration file emit is currently not exposed separately on the API. we intend to rewrite the declaration emitter as an emit transformation in the future; once that is done we will expose that on the API as well.

any plan for this?

[jpike88]

API is still extremely difficult to figure out. Would appreciate a bit of documentation on this

[SerkanSipahi]

@mhegazy any feedback will be good!

[mhegazy]

Sorry have not had chance to get back to adding the docs yet. #14419 tracks doing so. there are a multiple examples for what a transformation looks like in https://github.com/Microsoft/TypeScript/tree/master/src/compiler/transformers

[SerkanSipahi]

@mhegazy what i meant was: is Microsoft planning something like this:

We really need documentation like these:
babel:

• (user handbook) https://github.com/jamiebuilds/babel-handbook/blob/master/translations/en/user-handbook.md
• (plugin handbook) https://github.com/jamiebuilds/babel-handbook/blob/master/translations/en/plugin-handbook.md
• (babel types) https://github.com/babel/babel/tree/master/packages/babel-types
• (playground compiler api) https://astexplorer.net (there is no typescript transformer)

postcss:

• https://github.com/postcss/postcss/blob/master/docs/writing-a-plugin.md
• http://api.postcss.org/

[mhegazy]

@SerkanSipahi We are not exposing these transforms on the commandline compiler (like babel does) at the time being, this proposal is tracked by #16607.
Currentlly this functionality is only exposed as part of the public API. tools that wrap the compiler, be it ts-loader, or tsickle can expose that functionality.
We do have documentation for the compiler API in https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API, but we still missing documentation for how to use the transform pipeline. that is what #14419 is tracking.

[cancerberoSgx]

Here there are three working examples of using transformation APII among other things: https://typescript-api-playground.glitch.me/.

BTW you can modify the code and run it again, have intellisense, etc. ...like a TypeScript Compiler API Playground. hope it helps.

[SerkanSipahi]

@cancerberoSgx thank you.