Seeking Guidance: Significant typecheck performance drop post migrating to TS Project References in large monorepo

[sudesh-atlassian]

Acknowledgement

• I acknowledge that issues using this template may be closed without further explanation at the maintainer's discretion.

Comment

Context

We recently implemented TypeScript's project references feature in our monorepo, which has led to significant performance issues compared to our previous TypeScript configuration.

The typecheck process has experienced a dramatic increase in execution time, rising from 469.24 seconds to 4,297.17 seconds when we don't utilize any TypeScript cache (existing .d.ts files). We are keen to identify the root cause of this performance degradation and explore ways to improve or diagnose the typecheck performance with project references.

We are currently using .d.ts cache files to accelerate the typecheck process. However, this solution presents challenges in our integrated development environment (IDE). Emitting these .d.ts on the fly is time-consuming, and the cache files generated on Unix-based systems are incompatible with macOS machines, and vice versa.

We are seeking guidance on:

• Understanding the underlying reasons for the significant performance decline when using project references.
• Identifying methods to improve typecheck performance in our monorepo setup.
• Developing strategies to effectively use project references in our IDE without compromising development speed.
• Finding a solution to the cross-platform incompatibility of cache files between Unix and macOS systems.
• Any insights or recommendations on addressing these challenges would be greatly appreciated.

Setup

Typescript Version

We are using Typescript version 5.4.2

Base tsconfigs common between both tsconfigs with references and earlier typecheck

tsconfig.entry-points.json - This file is used to provide path resolution for custom entry points. Below is a snippet of the configuration; in total, it contains 2,769 paths.

{
  "compilerOptions": {
    "paths": {
      "@af/package-1": [
        "./package-1/src",
        "../package-1/src",
        "../../package-1/src",
        "../../../package-1/src",
        "../../../../package-1/src"
      ],
	  "@af/package-1/some-func": [
        "./package-1/src/some-func",
        "../package-1/src/some-func",
        "../../package-1/src/some-func",
        "../../../package-1/src/some-func",
        "../../../../package-1/src/some-func"
      ],
    }
  }
}   

Base TypeScript Configuration tsconfig.base.json - This file serves as the foundational TypeScript configuration for all packages within our monorepo. It defines common compiler options and settings that are inherited by individual package configurations.

{
  "extends": "./tsconfig.entry-points.json",
  "compilerOptions": {
    "plugins": [
      {
        "name": "tsserver-metrics-plugin"
      }
    ],
    "baseUrl": ".",
    "allowJs": false,
    "strict": true,
    "allowSyntheticDefaultImports": true,
    "declaration": true,
    "emitDeclarationOnly": true,
    "esModuleInterop": true,
    "experimentalDecorators": true,
    "downlevelIteration": true,
    "importHelpers": true,
    "jsx": "react",
    "moduleResolution": "node",
    "noFallthroughCasesInSwitch": true,
    "noUnusedLocals": true,
    "sourceMap": true,
    "skipLibCheck": true,
    "isolatedModules": true,
    "target": "es2019",
    "module": "esnext",
    "lib": [
      "dom",
      "dom.iterable",
      "es5",
      "scripthost",
      "es2015.core",
      "es2015.collection",
      "es2015.symbol",
      "es2015.iterable",
      "es2015.promise",
      "es2016.array.include",
      "es2017"
    ]
  },
  "files": [
    // Global typings go here which are expected in most of the packages - we have 62 such files
    "./typings/json.d.ts",
        ...
  ],
  "ts-node": {
    "transpileOnly": true,
    "files": true,
    "compilerOptions": {
      "module": "CommonJS"
    }
  }
}

Typecheck with project references

To address circular dependency issues, we have split each package's TypeScript configuration into two separate files: one for application code and another for development-related code. Below are sample configurations for both:

tsconfig.app.json - This configuration is used for the main application code of each package.

{
  "extends": "../tsconfig.base.json",
  "include": [
    "./src/**/*.ts",
    "./src/**/*.tsx",
    "./scripts/**/*",
    "definitions.ts",
    "types.ts"
  ],
  "exclude": [
    "**/docs/**/*",
    "**/__tests__/**/*",
    "**/vr-tests/**/*",
    "**/__perf__/**/*",
    "**/*.test.*",
    "**/test.*",
    "**/test-*",
    "**/examples.ts",
    "**/examples.tsx",
    "**/examples/*.ts",
    "**/examples/*.tsx",
    "**/examples/**/*.ts",
    "**/examples/**/*.tsx",
    "**/storybook/**/*",
    "**/constellation/**/*",
    ".storybook/*",
    "./__fixtures__/**/*",
    "./__generated__/**/*",
    "./mocks/**/*",
    "./__mocks__/**/*",
    "**/mock.*",
    "**/codemods/**/*.ts",
    "**/codemods/**/*.tsx"
  ],
  "compilerOptions": {
    "composite": true,
    "lib": ["ES2021.String"],
    "outDir": "../tsDist/package-1/app"
  },
  "references": [
    {
      "path": "../folder-2/foo/bar/tsconfig.app.json"
    },
  ]
}

tsconfig.dev.json - This configuration is used for development-related code, such as tests, mocks and build scripts.

{
  "extends": "../tsconfig.base.json",
  "include": [
    "**/docs/**/*",
    "**/__tests__/**/*",
    "**/vr-tests/**/*",
    "**/__perf__/**/*",
    "**/*.test.*",
    "**/test.*",
    "**/test-*",
    "**/examples.ts",
    "**/examples.tsx",
    "**/examples/*.ts",
    "**/examples/*.tsx",
    "**/examples/**/*.ts",
    "**/examples/**/*.tsx",
    "**/storybook/**/*",
    "**/constellation/**/*",
    ".storybook/*",
    "./__fixtures__/**/*",
    "./__generated__/**/*",
    "./mocks/**/*",
    "./__mocks__/**/*",
    "**/mock.*",
    "**/codemods/**/*.ts",
    "**/codemods/**/*.tsx"
  ],
  "exclude": ["./dist/**/*", "./build/**/*", "./node_modules/**/*"],
  "compilerOptions": {
    "composite": true,
    "lib": ["ES2021.String"],
    "outDir": "../tsDist/package-1/dev"
  },
  "references": [
    {
      "path": "tsconfig.app.json"
    }
  ]
}

tsconfig.project-references.json - This configuration file serves as the central hub for TypeScript project references across our entire monorepo. Its primary purpose is to enable efficient typechecking using TypeScript's project references feature.
In all we have 1226 packages and their app and dev tsconfigs (2452 tsconfigs). We didnt add tsconfig.app.json's as they are already included in tsconfig.dev.json's references.

{
  "files": [],
  "references": [
    { "path": "./folder-1/package1/tsconfig.dev.json" },
    { "path": "./folder-2/package2/tsconfig.dev.json" },
    // ... additional 1,224 package references
  ]
}

Typecheck command

NODE_OPTIONS="--max-old-space-size=16384" tsc --build tsconfig.project-references.json

Extended Diagnostic with project references when we didnt used any existing Cache

Projects in scope:                         2423
Projects built:                            2315
Aggregate Files:                        4064060
Aggregate Lines of Library:            85984227
Aggregate Lines of Definitions:       902941654
Aggregate Lines of TypeScript:          5739519
Aggregate Lines of JavaScript:                0
Aggregate Lines of JSON:                  61189
Aggregate Lines of Other:                     0
Aggregate Identifiers:               1036740113
Aggregate Symbols:                    710515971
Aggregate Types:                       36168921
Aggregate Instantiations:             216983900
Aggregate Memory used:                15641296K
Aggregate Assignability cache size:    12826627
Aggregate Identity cache size:           475304
Aggregate Subtype cache size:            586070
Aggregate Strict subtype cache size:     696811
Aggregate I/O Read time:                  4.28s
Aggregate Parse time:                    49.46s
Aggregate ResolveModule time:           223.61s
Aggregate ResolveTypeReference time:     33.39s
Aggregate ResolveLibrary time:            2.86s
Aggregate Program time:                 823.17s
Aggregate Bind time:                     28.12s
Aggregate Check time:                  1927.31s
Aggregate transformTime time:           735.72s
Aggregate commentTime time:               3.42s
Aggregate printTime time:              1023.13s
Aggregate Emit time:                   1023.78s
Aggregate I/O Write time:                 8.43s
Config file parsing time:                55.87s
Up-to-date check time:                    0.20s
Build time:                            4297.17s

Typehecking with TS Cache

• We run a scheduled job which runs typecheck with references on stable master, then use the emitted .d.ts files as Cache with masters current commit hash as index.
• On branch typecheck we fetch the nearest cache available based on commit history, also we update the file timestamps for files which are modified post the cache generation as to only typecheck them

Typecheck with --project mode

tsconfig.typecheck.json - This is the tsconfig for typecheck with --project mode

{
    {
  "extends": "./tsconfig",
  "include": [
    "./folder-1/**/*",
    "./folder-2/**/*",
    "./folder-3/**/*",
    "./folder-4/**/*",
    "./services/service-1/**/*",
    "./services/service-2/**/*",
    "./services/service-3/**/*",
    "./services/service-4/**/*",
    "./services/service-5/**/*",
    "./services/service-6/**/*",
    "./services/service-7/**/*",
    "./services/service-8/**/*",
    "./tools/**/*"
  ],
  "exclude": [
    "./folder-1/**/**/dist",
    "**/some.d.ts",
    "**/node_modules",
    "node_modules",
    "./services/node_modules",
    "./folder-3/foo/bar/",
    "./folder-3/foo/bar2/",
    "./folder-3/foo2/bar3/0-type.ts",
    "./folder-3/foo2/bar4/0-type.ts",
    "./folder-3/foo3/__generated__/globalTypes.ts"
  ],
  "compilerOptions": {
    "baseUrl": "./",
    "emitDeclarationOnly": false,
    /* Typechecking only, no code generation required */
    "noEmit": true,
    /* Set module to CJS rather than ES so that CJS-only modules that use
     * import/export assignment (export = require(...)) don't break
     */
    "module": "CommonJS"
  }
}

Typecheck command

NODE_OPTIONS="--max-old-space-size=16384" tsc --project tsconfig.typecheck.json

Extended Diagnostic with --project mode

Files:                         59890
Lines of Library:              40963
Lines of Definitions:        2185828
Lines of TypeScript:         4577662
Lines of JavaScript:               0
Lines of JSON:                     0
Lines of Other:                    0
Identifiers:                 7206749
Symbols:                    14278742
Types:                       4368428
Instantiations:             14365921
Memory used:               12876878K
Assignability cache size:    4615911
Identity cache size:          166448
Subtype cache size:           418129
Strict subtype cache size:    510959
I/O Read time:                 2.18s
Parse time:                   21.74s
ResolveModule time:           10.10s
ResolveTypeReference time:     0.05s
ResolveLibrary time:           0.02s
Program time:                 40.06s
Bind time:                    13.03s
Check time:                  416.12s
printTime time:                0.03s
Emit time:                     0.03s
Total time:                  469.24s

Observations

• Compared to --project mode, typecheck with project references has nearly 60x (59890 vs 4064060) more files to typecheck, The number of files difference is mostly because of the way files are calculated (Basically files is aggregate number of files part of each tsconfig) not the files that are typechecked

[RyanCavanaugh]

These two lines point to the problem:

Aggregate Lines of Definitions: 902941654
Aggregate Lines of TypeScript: 5739519

You have a reasonable amount of TS code, but this configuration analyzes a total of 1 billion lines of definition file because every project is loading ~half a million lines of definition files.

The key takeaway is that every project has to load its definition files separately, so the more projects you have (2,200 is far too many to be optimal; you should be targeting something like 50), and the more definition files they load, the more time you're going to pay in overhead. The I/O parsing is shared, but there's still a lot of work to do a bit of up-front analysis of all these top-level files.

To make this faster, you need to either reduce the number of projects you have, reduce the number of definition files they're loading, or ideally both.

A likely culprit here is the lack of types array in your base config. What happens if there's no types array is that everything from node_modules/@types gets loaded automatically, and I'm guessing you have several hundred packages in there (let's call it 500), so you're loading 2,200 x 500 modules per project. A similar team at Microsoft encountered this misconfiguration; the most expedient fix is to put "types": [] in the base tsconfig, rebuild, add back the global types that are needed (usually this is just node and/or jest), and fix up any remaining projects. This should get you a huge win fairly quickly.

If you're spending 1,000s on emit, I would also recommend switching to something like esbuild for JS output. It's much faster and can handle almost all relevant TS configurations.

[DanielRosenwasser]

in total, it contains 2,769 paths.

To add to what Ryan said, I'll also mention that with recent nightly builds, #59048 may slightly improve issues around path mapping lookups. It's probably not the root cause of issues though.

[DanielRosenwasser]

Also, is there any reason why each path has to be mapped in multiple ways? What system are you using where it is possible for a path to map to all of the following?

        "./package-1/src",
        "../package-1/src",
        "../../package-1/src",
        "../../../package-1/src",
        "../../../../package-1/src"

[sudesh-atlassian]

Also, is there any reason why each path has to be mapped in multiple ways? What system are you using where it is possible for a path to map to all of the following?

        "./package-1/src",
        "../package-1/src",
        "../../package-1/src",
        "../../../package-1/src",
        "../../../../package-1/src"

In the monorepo, we have packages at varying depths so to accomodate it we had to add multiple depth paths. We know about this being bottleneck and it also degrades our IDE performance.

We attempted migrating to Bundler moduleResolution but had to revert as the IDE performance deteriorated.

[sudesh-atlassian]

These two lines point to the problem:

Aggregate Lines of Definitions: 902941654
Aggregate Lines of TypeScript: 5739519

You have a reasonable amount of TS code, but this configuration analyzes a total of 1 billion lines of definition file because every project is loading ~half a million lines of definition files.

The key takeaway is that every project has to load its definition files separately, so the more projects you have (2,200 is far too many to be optimal; you should be targeting something like 50), and the more definition files they load, the more time you're going to pay in overhead. The I/O parsing is shared, but there's still a lot of work to do a bit of up-front analysis of all these top-level files.

To make this faster, you need to either reduce the number of projects you have, reduce the number of definition files they're loading, or ideally both.

A likely culprit here is the lack of types array in your base config. What happens if there's no types array is that everything from node_modules/@types gets loaded automatically, and I'm guessing you have several hundred packages in there (let's call it 500), so you're loading 2,200 x 500 modules per project. A similar team at Microsoft encountered this misconfiguration; the most expedient fix is to put "types": [] in the base tsconfig, rebuild, add back the global types that are needed (usually this is just node and/or jest), and fix up any remaining projects. This should get you a huge win fairly quickly.

If you're spending 1,000s on emit, I would also recommend switching to something like esbuild for JS output. It's much faster and can handle almost all relevant TS configurations.

Thanks @RyanCavanaugh, we did the changes in tsconfig.base.json to include only few types by default "types": ["node", "jest", "@testing-library/jest-dom"], post this change and fixing missing the types the typecheck time has improved considerably from 70 mins to now nearly 33 mins.

Below is the updated diagnostic information

Projects in scope:                        2423
Projects built:                           2315
Aggregate Files:                       1535787
Aggregate Lines of Library:           85339821
Aggregate Lines of Definitions:      416501200
Aggregate Lines of TypeScript:         5739519
Aggregate Lines of JavaScript:               0
Aggregate Lines of JSON:                 61189
Aggregate Lines of Other:                    0
Aggregate Identifiers:               469412125
Aggregate Symbols:                   361212674
Aggregate Types:                      32278672
Aggregate Instantiations:            172315031
Aggregate Memory used:               11625463K
Aggregate Assignability cache size:   12221436
Aggregate Identity cache size:          473960
Aggregate Subtype cache size:           585216
Aggregate Strict subtype cache size:    696813
Aggregate I/O Read time:                 3.57s
Aggregate Parse time:                   32.53s
Aggregate ResolveModule time:           63.89s
Aggregate ResolveTypeReference time:     2.32s
Aggregate ResolveLibrary time:           1.80s
Aggregate Program time:                255.40s
Aggregate Bind time:                    16.57s
Aggregate Check time:                 1185.78s
Aggregate transformTime time:          329.86s
Aggregate commentTime time:              2.60s
Aggregate printTime time:              390.34s
Aggregate Emit time:                   390.75s
Aggregate I/O Write time:                4.69s
Config file parsing time:               37.30s
Up-to-date check time:                   0.15s
Build time:                           1985.91s

Regarding emit we have added emitDeclarationsOnly as we need the .d.ts cache to improve typecheck performance. We are currently using parcel / webpack to transpile code to js.

[typescript-bot]

This issue has been marked as "Question" and has seen no recent activity. It has been automatically closed for house-keeping purposes.

[DanielRosenwasser]

@sudesh-atlassian did you find improvements using the nightly builds of TypeScript at all?

[sudesh-atlassian]

@sudesh-atlassian did you find improvements using the nightly builds of TypeScript at all?

@DanielRosenwasser - We will spike TS5.6 this week and update findings here upon completion.

This week, we spiked moduleResolution Bundler to eliminate tsconfig.entry-points.json. Findings: CI typecheck time remained similar, but IDE performance regressed by nearly 2x for our team.

[typescript-bot]

This issue has been marked as "Question" and has seen no recent activity. It has been automatically closed for house-keeping purposes.