Compiler incorrectly caches module resolution if we use a custom ts.SourceFile & ts.Program cache

[frigus02]

Bug Report / Question

At Google we use a "custom tsc", which integrates with Bazel. It also implements something similar to --watch using persistent workers. The main difference is that the responsibility to watch for file changes lives in a separate process, which notifies a long running compiler process via IPC. The same compiler process is used to build many different projects. Projects may share dependencies and therefore read the same .d.ts files.

To make repeated builds fast, we use 2 caches:

• A ts.SourceFile cache lets us reuse ASTs across different projects
• A ts.Program cache lets use cache module resolution and type checking per project (by passing an oldProgram to ts.createProgram)

The TS compiler (used through the compiler API) sometimes incorrectly caches module resolution and reports "TS2307: Cannot find module", even if the corresponding file exists.

🔎 Search Terms

TS2307, compiler API, cache, oldProgram

🕗 Version & Regression Information

• This is the behavior in every version I tried (4.0 - 4.7.4)

💻 Repro

We managed to create a minimal repro: https://github.com/frigus02/test-ts-oldprogram-moduleresolution

$ git clone https://github.com/frigus02/test-ts-oldprogram-moduleresolution.git
$ cd test-ts-oldprogram-moduleresolution
$ npm ci
$ node ./index.js

The repro has the 2 caches mentioned above and performs 3 builds, one after another, on the same project/list of files:

1. Successful build
2. Delete file /module/b.d.ts. Build fails with error "TS2307: Cannot find module './module/b'"
3. Restore file /module/b.d.ts. Build still fails with the same error. But I would expect it to succeed again

🙁 Actual behavior

The 3rd build fails, seemingly because module resolution and/or diagnostics have been cached.

🙂 Expected behavior

The 3rd build succeeds.

🕵️ Investigation

I assume we're violating an assumption in the compiler with the 2 different caches. In the repro, do you see any obvious misuses of the compiler API?

We found some solutions/workarounds. They either make the caches less effective or rely on @internal TypeScript APIs:

• Implement the @internal hasInvalidatedResolution() function in the CompilerHost.
• Only add dependency .d.ts files to the ts.SourceFile cache. Bazel should ensure that those are always without type errors.
• Remove one of the caches. It would have to be the ts.Program cache. The ts.SourceFile cache has a much bigger performance impact. We can't afford to remove that.

From your perspective, is anyone of those the right thing to do?

[frigus02]

We did some further investigation. It turns out the following solutions are not sufficient:

• Only add dependency .d.ts files to the ts.SourceFile cache. Bazel should ensure that those are always without type errors.

Module resolution for an import to ./a can change from ./a/index.d.ts to ./a.d.ts just based on the presence of those files. Bazel doesn't know the module resolution logic, so it can only be of limited help for us here.

I think this leaves us with the following options that don't rely on @internal APIs:

• Bazel knows the full list of input files. If the file names in the list have changed, drop either the ts.Program or ts.SourceFile cache.
• Implement module resolution ourselves entirely, as described in https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API#customizing-module-resolution. Although since failedLookupLocations is @internal we might not be able to use ts.resolveModuleName. I also saw that Running with custom module resolver is 3x slower, as resolution is invalidated by TS compiler every time #48057 suggested custom module resolution is slower. Not sure if that would apply for us.

If we use @internal APIs, these would also work:

• If the list of input files has changed, return true from hasInvalidatedResolution() for all files.
• Implement hasInvalidatedResolution(). For each import statement, call program.getResolvedModuleWithFailedLookupLocationsFromCache(). If any failedLookupLocation now exists, return true.

Would you be open to making some of these internal APIs public?

[sheetalkamat]

@frigus02 what you are looking for is actually supported using Watch Program API where we track the changes to keep track of what all changes to update the program correctly.

const host = createWatchCompilerHost(...)
// Override your watchFile and watchDirectory to handle your own watches: 
// host.watchFile = 
const watchProgram = createWatchProgram(host);
watchProgram.getProgram() // Should give you Program at any point
// Invoking watch callbacks for the files should update the program

[frigus02]

Thanks @sheetalkamat. The WatchHost doesn't quite fit into the Bazel model. Our long running TypeScript compiler process can't actually watch files on the filesystem. It waits for Bazel to setup a (kind of sandboxed) working directory and send a build request. This request contains a full list of input files for this build.

I suppose we could diff the input files against the ones from the previous build and issue FIleWatcherCallbacks and DirectoryWatcherCallbacks. Maybe that is the right thing to do.

I'll see if I can prototype this.

[frigus02]

I read through the docs at https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API#writing-an-incremental-program-watcher and changed the repro to use the watch API:

https://github.com/frigus02/test-ts-oldprogram-moduleresolution/tree/watchhost

This definitely solves the module resolution cache issue. This is great!

The way I'm using it right now comes with a new issue. It seems compilation happens asynchronously or at least delayed. When I call watchProgram.getProgram().getSemanticDiagnostics() right after issuing file watcher events, I'm seeing (at least partially) old diagnostics. They do update eventually. Is there a way to synchronously force type checking after we issued all file watcher events?

[mprobst]

@frigus02 we do know which files have changed at the start of a build (bazel tells us the hashes of source files, we could track which have changed).

However I wonder if using the program watcher has other performance consequences. In our bazel build, a typical application often has hundreds or thousands of ts.Program objects that each represent an individual compilation action. When building, we know which of these programs needs re-building, and selectively only re-check those that can have changed.

@sheetalkamat does such a model fit the assumptions made by createWatchProgram? In particular, how does it compare in memory and other resource usage (open file descriptors? file watches?) to a plain ts.Program?

[frigus02]

We noticed anther possibly larger issue with our approach. I'll try to illustrate it with an example:

repo
|- input_component
|  |- input.ts
|  `- tsconfig.json (depends on: validation_lib)
|                   (input files: input.ts, validation.d.ts, assertion.d.ts)
|- button_component
|  |- button.ts
|  `- tsconfig.json (depends on: validation_lib)
|                   (input files: button.ts, validation.d.ts, assertion.d.ts)
|- validation_lib
|  |- validation.ts
|  `- tsconfig.json (depends on: assertion_lib)
|                   (input files: validation.ts, assertion.d.ts)
|- assertion_lib
|  |- assertion.ts
|  `- tsconfig.json (no dependencies)
|                   (input files: assertion.ts)

In this example we would have 4 ts.Program instances.

As Martin mentioned, in reality a typical application consists of hundreds or thousands of ts.Program instances. Building all of those in a single process is too slow. So Bazel spawns multiple processes and/or runs some builds remotely on shared build VMs.

We observed that many ts.Program instances have to parse and type-check the same source file. In the example assertion.d.ts has to be processed 3 times. To avoid that, we introduced a ts.SourceFile cache.

The ts.SourceFile cache lets us share ts.SourceFile instances between ts.Program instances within a single process. But that, it turns out, seems to be an issue. TypeScript stores the module resolution cache in ts.SourceFile instances (SourceFile#resolvedModules).

Example issue (triggered by our use of rootDirs):

• "input_component" loads assertion.d.s from path input_deps/assertion_lib/assertion.d.ts and stores this resolution in the validation.d.ts source file.
• "button_component" would load assertion.d.s from path button_deps/assertion_lib/assertion.d.ts, but the shared validation.d.ts source file already has a module resolution cache pointing to input_deps/. This path doesn't exist for the "button_component" build. The build fails.

Sharing ts.SourceFile instances between programs is a huge performance win for us. I assume we would have the same issue, even if we switched to the Watch Program API. @sheetalkamat, is there a way we can safely share ts.SourceFile instances between programs?

[sheetalkamat]

You can use DocumentRegistry for getting/creating source files since that handles when source files can be shared (depends on module resolution options as well as some other important options that if they differ between programs cannot share sourceFile)

[frigus02]

I had a look at the DocumentRegistry. If I understand correctly, it would give us 2 benefits:

• It caches source files by fileName & source file affecting compiler options. We could use getKeyForCompilationSettings to add this feature to our custom source file cache.
• It has a way to update a source file AST rather then re-creating from scratch after an edit.

In my test the document registry did not entirely fix the module resolution cache issue. That makes sense to me. It's possible that builds use the same compiler options and only the available source files on disk change.

Maybe combining DocumentRegistry with the Watch Program API would work. I will need some more time to test that.

[dmichon-msft]

The way I'm using it right now comes with a new issue. It seems compilation happens asynchronously or at least delayed. When I call watchProgram.getProgram().getSemanticDiagnostics() right after issuing file watcher events, I'm seeing (at least partially) old diagnostics. They do update eventually. Is there a way to synchronously force type checking after we issued all file watcher events?

If you also provide implementations of setTimeout and clearTimeout on the WatchCompilerHost you can manage the timing yourself.