A minimal custom transformer plugin proposal

[jakebailey]

In the TypeScript compiler, we have the concept of a "transformer". A transformer is a function which "transforms" one AST to another. TypeScript ships many such transforms, including all of the various transformation steps needed to downlevel newer syntax to older syntax, to convert from import/export to require/module.exports, and so on. Even the declaration emitter is a transformer, stripping away function bodies, filing in types, etc.

Since TypeScript 2.3, TypeScript has had the following API:

interface Program {
    emit(
        targetSourceFile?: SourceFile,
        writeFile?: WriteFileCallback,
        cancellationToken?: CancellationToken,
        emitOnlyDtsFiles?: boolean,
        customTransformers?: CustomTransformers, // <--- 👀
    ): EmitResult;
}

interface CustomTransformers {
    before?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    after?: (TransformerFactory<SourceFile> | CustomTransformerFactory)[];
    afterDeclarations?: (TransformerFactory<Bundle | SourceFile> | CustomTransformerFactory)[];
}

This API allows users to provide a set of "custom transformers" at emit time. These transformers run alongside TypeScript's own transformers, either before or after, depending on where they are placed in CustomTransformers.

However, while this API exists and does work, a major gotcha is that custom transformers can only be provided to the API. There is no way to use custom transformers if you want to use tsc alone.

This has been a long-standing issue; #14419 is the fourth most upvoted issue on the repo.

What do people use custom transformers for?

This list can go on and on, but transformers are commonly used these days for:

• Generating serialization / deserialization / runtime type checks
  • typia,ts-runtime-checks, which top the benchmark charts, but there are many more
• Powering internationalization
  • formatjs
• Generating database code
  • ts-graphql-plugin
• Transforming paths (please don't do this one, though 😅)

I also know of many ideas which want to be implemented via plugins, but are not willing to do so without official support, including other runtime type checks, tracing code, and so on.

How do users currently use custom transformers?

Given the current constraints of custom transformers, there are two ways that downstream users have been able to use them:

1. Build TypeScript code via the API, either directly or by using some system which uses the API.
   • webpack, rollup's TS plugins, nx, all provide configuration which allows users to specify a set of custom transformers.
2. Patch TypeScript.
   • ttypescript, ts-patch wholly replace TypeScript in a user's workflow.

The first one is obviously fine. The latter, eek!

What changed?

In TypeScript 5.0, we shipped the biggest architectural / packaging change in TypeScript's history; conversion of the entire project to modules, including the bundling of our code. The relevant side-effects are twofold:

1. The structure of tsc.js (and other outputs) changed massively. This broke anyone who was patching the contents of the TypeScript package.
2. The API is no longer runtime patchable; we use esbuild to bundle our code and the objects it creates are not "configurable", correctly emulating how "real" ES modules would behave. Any runtime patches were already extremely fragile, only functioning due to the structure of our old namespace emit.

In response to this, I did an analysis of as many TS "patchers" as I could find, and found that there are actually very few reasons to patch TypeScript at all:

• ttypescript/ts-patch, which seem to be almost exclusively used to enable the use of custom transformers.
• Language service plugins, mainly injecting their own module resolution and custom source file types (vue/volar, css, etc).
• Build systems like heft.
• Yarn PnP.
• Those who want to tree shake out our parser (prettier).

For 5.0, I was able to fix many of these projects (or at least guide / warn maintainers ahead of time), but many of these patchers have no viable alternative.

We have been seriously considering ways that we can approach the problems that remain in hopes that we can eliminate the need for people to patch TypeScript. This includes (no promises):

• Custom transformers.
• Custom module resolution.
• A future API that actually allows a consumer to tree shake out just a parser (or similar).

This issue intends to address the first bullet.

A conservative, targeted proposal

In short, my proposal is to add to TypeScript the ability to define "custom transformer plugins". This new functionality is targeted and minimal. Its only intent is to satisfy those who just want to be able to add custom transformers.

Many details still have yet to be expanded on, but in short:

• These plugins are placed into the plugins array of compilerOptions, with a discriminator like "type": "transformer".
  • This is the approach taken by ttypescript/ts-patch, and offers up a future base for additional tsc plugins (e.g. file watching plugins, module resolution plugins).
  • The choice in discriminator ("type": "transformer", for now) is intended to avoid conflicting with existing plugins defined by ttypescript or ts-patch.
• Plugins would take a form similar to language service plugin, receiving the ts object, returning a factory that can be used to create a CustomTransformers object. The factory would receive the Program, which is required for any type-using plugins.
  • The ts object must be a part of the API for similar reasons to the language service plugins; the APIs available within the core compiler are very, very limited, and the bundle is not the same as typescript.js.
  • The plugin object entry is passed to the plugin for extra configuration, like LS plugins.
• A new tsclibrary.d.ts file is added, describing the limited API within tsc. This is similar to tsserverlibrary.d.ts.
• When invoking tsc, you must pass --allowPlugins to opt into executing plugins. When using the API or tsserver, plugins are enabled by default.
  • No sandbox is present otherwise. If you currently use a patching-based method to use plugins, you're already taking matters into your own hand. If you currently use webpack or rollup, your config files are executable anyway.
  • --allowPlugins is also a part of the watch plugin PR.
• If multiple plugins are used, their returned CustomTransformers are merged.

There are almost assuredly other things that people want to be able to do with custom transformers or other plugins, however, I believe that we can serve the vast majority of those who want custom transformers with this API, and I do not want to let perfect be the enemy of the good.

In the future, we can expand plugins to do more, allowing the customization of other aspects of TypeScript, or simply add on to what custom transformer plugins can do. Prior research has shown that there are more interested parties, which I believe that we can eventually support.

An example

With the above proposal, a tsconfig.json would look like this:

{
    "compilerOptions": {
        "plugins": [
            { "type": "transformer", "path": "@jakebailey/cool-transformer" }
        ]
    }
}

The transformer would look something like:

import type * as ts from "typescript/tsclibrary";

const factory: CustomTransformersModuleFactory = ({ typescript }) => {
    return {
        create: ({ program }) => {
            return {
                before: [(context) => (file) => {/* ... */}],
            };
        },
    };
};

export = factory;

Of course, the details may change.

Unsolved problems

• It's very unfortunate to have to define yet another random d.ts API used for one thing.
  • Can we do better?
  • What if the executables were ESM?
• Like LS plugins, tsc plugins would need to be CJS.
  • Can we instead make use of import?
  • How do we do async? (This may already be "solved" in a prior WIP for plugins)
• Should we do any sort of performance metric counting for plugins?
  • Like LS plugins, can we just say "no warranty" when plugins are involved?
• How much of a problem is it going to be to need to use the typescript namespace passed into the plugin?
  • What happens to existing transformers which mistakenly import typescript.js's API directly, even though that may not have always worked?
  • Will plugins unknowingly use the typescript.js API and rely on it, assuming users won't use tsc?
  • Should we try and attempt converting our binaries/libraries to ESM first?
• Is the proposed API too factory-y?
• Should we even do this?

Please give feedback!

If you are currently using transformers, creating transformers, or otherwise, I would appreciate your feedback.

[jakebailey]

Preemptive @nonara ping; I suspect you have feedback! 😄

[timocov]

Hey @jakebailey,

Thank you for this proposal! I personally like the approach (it feels similar to ttypescript), but I have a question - it seems it is not possible to provide options to a "transformer" from the config, right? Would it be possible to allow to define something like options/params/config next to type and path fields that the compiler will pass as-is (without any transformation/validation) to the transformer function?

Quick intro where I came from

I'm the author of 2 optimization-related transformers:

• https://github.com/timocov/ts-transformer-minify-privates
• https://github.com/timocov/ts-transformer-properties-rename

They both allow you to minify properties of objects (either private or internal) and it is achieved by adding a prefix to "all properties that should be minified" so you can use these prefixes in tools like terser or uglifyjs to minify i.e. mangle these properties. While the default prefixes (_private_ and _internal_) are used in majority cases, there are few cases when people prefer to use different prefixes for whatever reason (e.g. to sync values in their config files). Thus it would be quite beneficial to have such option.

Thanks!

[jakebailey]

Would it be possible to allow to define something like options/params/config next to type and path fields that the compiler will pass as-is (without any transformation/validation) to the transformer function?

Oops, I should have explicitly defined that; yes, any other options are passed along to create, just like language service plugins.

export type CustomTransformersModuleFactory = (mod: { typescript: typeof ts }) => CustomTransformersModule;

export interface CustomTransformersModuleWithName {
    name: string;
    module: CustomTransformersModule;
}

export interface CustomTransformersModule {
    create(createInfo: CustomTransformersCreateInfo): CustomTransformers;
}

export interface CustomTransformersCreateInfo {
    program: Program;
    config: any;
}

[patdx]

Why wouldn’t it support esm? Is it because the initializing code of tsc is synchronous? It may be nice if it could support esm from the start.