Portable .tsbuildinfo

[bweggersen]

Is it possible to make the tsbuildinfo-file portable so it can be shared between different machine environments?

Search Terms

tsbuildinfo, portable, cache

Suggestion

Looking at a tsbuildinfo-file, it contains absolute paths, both in fileInfos and options. I don't know how the hashes in version and signature are created, and if they contain elements of the absolute path in some way. If not, then maybe it's enough to make the paths relative?

Use Cases

We want to cache outDir on the CI and on local dev machines to enable instant incremental builds. But we can't currently bring back a tsbuildinfo-file from a different environment since it will be discarded as it contains absolute paths. In this case, the project will have to be rebuilt.

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

[bweggersen]

@sheetalkamat

[ajafff]

Seems to be implemented by #31985

[Adjective-Object]

Currently looking at porting build outputs between machines to speed up our project build in CI and stumbled onto this issue.

tsbuildinfo files don't track the hashes of the build outputs, and instead only track the most recently changed file. If moved between multiple machines, that the individual file mtimes for both the input source files and the copied cache files are going to be updated to the time the cache changed.

I think that means that the cache-verification process will likely be incorrect on the first several runs if tsbuildinfo files + build outputs are copied to a new machine with a new checkout of whatever repo they come from.

• Some projects may be incorrectly marked as up-to-date if the cache is transferred after checkout. This may lead to correctness issues
• Some may be incorrectly marked as out-of-date if the cache is transferred before checkout, and rebuilt. This won't lead to correctness issues, but it will lead to overbuilding.

It looks like the cache is being verified by a single file's timestamp because its faster than checking content hashes. On a project I work on, we solve this by storing both the hashes of all the expected output files, as well as a timestamp + machine ID to do quick reverification to work around this exact issue.

I think I can work around this by also tracking and artificially updating the mtime on files in our repo's caching setup script, but I figured I'd resurrect this long-closed thread to bring up the issue and offer to help on a solution, if portable caches is something the project is still interested in.

[sheetalkamat]

While we are making it easier to port tsbuildInfo, we are not looking to support this scenario natively which means if you are porting tsbuildinfo, you know what you are doing and will handle all the timestamp etc stuff correctly.

[Adjective-Object]

On further investigation, moving mtimes between machines isn't super feasible either -- this would require performing content-hash checks while restoring the mtimes on new machines -- which is approaching just writing a parallel reimplementation of tsbuildinfo verification.

That makes using the tsc cache across machines in its current state prohibitively cumbersome :/

If you want me to jump in and make the changes to tsc to make this happen, I'm happy to. Given the past history of the changes you made in support of porting tsbuildinfo I'm curious why it's an explicit non-goal of the project now. Has the philosophy of around cache-portability changed since 2022?

[sheetalkamat]

As i mentioned, we dont want to add complexity to tsbuildinfo. We do not want to encode extra information and checks as it comes with the cost. We have done the changes to support portability in past where it was simple and not costly but we do not want to be encoding hashes of output or machine ids etc

[JavaScriptBach]

While we are making it easier to port tsbuildInfo, we are not looking to support this scenario natively which means if you are porting tsbuildinfo, you know what you are doing and will handle all the timestamp etc stuff correctly.

In that case @sheetalkamat would you consider adding documentation on how tsbuildinfo works so users can work around it?

I have a similar situation where:

1. .d.ts files are shared in a remote cache, but not the tsbuildinfo file
2. Projects are compiled using incremental mode.

Developers are reporting that sometimes tsc is giving incorrect results. I'm guessing this issue is the culprit because the mtimes of the .d.ts files will not necessarily match up with the mtimes on the source files or the tsbuildinfo, but without documentation it's not clear what I should do to work around it.