No way to use composite project where outDir is determined by build tool

[alexeagle]

In #37257 I discussed one way to host TS project references under Bazel, where each build step gets only .d.ts and tsconfig.json files from the referenced projects. This is broken because TS module resolution fails to locate the .d.ts files in their outDir.

In this issue, let's try a different approach for Bazel, to satisfy current TS module resolution, each build step should get the .ts sources of its referenced projects, and the .tsbuildinfo output as well.

Bazel is picky about the setting of --outDir, because it supports cross-compiled builds in distinct output locations. For example if you're on a Mac and building for local platform, the outDir is <workspace root>/bazel-out/darwin-fastbuild/bin. But if you're on a Mac and building for a docker image, you need native binaries to be for the target platform so outDir is <workspace root>/bazel-out/k8-fastbuild/bin and if you want binaries with debug information it's <workspace root>/bazel-out/k8-dbg/bin.

Since the outDir is platform-specific, we don't want to check in a config file that hard-codes it. So we prefer to always have Bazel pass the --outDir option on the command-line to tsc, rather than include it in the tsconfig files.

Now we come to the problem. TypeScript "composite" setting enables up-to-dateness checking in tsc. With this input structure, and trying to tsc -p b

a/a.ts
a/tsconfig.json (no outDir specified; a was compiled with tsc --outDir=bazel-out/darwin-fastbuild/bin/a)
b/b.ts
b/tsconfig.json
bazel-out/darwin-fastbuild/bin/a/a.d.ts
bazel-out/darwin-fastbuild/bin/a/tsconfig.tsbuildinfo

we get b/b.ts(1,17): error TS6305: Output file '/private/var/tmp/_bazel_alx/df60115ea7f2a64e10fb4aa64b7d827f/sandbox/darwin-sandbox/54/execroot/ts_composite/a/a.d.ts' has not been built from source file '/private/var/tmp/_bazel_alx/df60115ea7f2a64e10fb4aa64b7d827f/sandbox/darwin-sandbox/54/execroot/ts_composite/a/a.ts'.

This indicates that TS is looking for a.d.ts next to a.ts. If we hard-code the platform-dependent outDir into a/tsconfig.json like I've done here
https://github.com/alexeagle/ts_composite/pull/5 it solves this problem, but not in a way we can ship.

Note that the compilation of a/tsconfig.json produces a .tsbuildinfo file with a correct "outDir" (based on the --outDir flag that was passed to compilation of a

 "options": {
      "project": "../../../../a/tsconfig.json",
      "outDir": "./",
  }

So it seems like the behavior for compiling b is to read the outDir setting from a/tsconfig.json rather than trust the options.outDir reflected in the .tsbuildinfo output.

[sheetalkamat]

when doing tsc -p b you are only building the project b and not a so there is no way to tell if a was built with extra options or not. Whatever is present in the config file of a is the only truth compiler can assume.

[alexeagle]

Yes, I can see that. The only way tsc could do what I'm asking here is if the --outDir setting of b was considered to match a - that is, my output directory scheme always preserves the same directory structure as the sources, so if a was imported from ../a and b is compiled into --outDir=bazel-out/b then we could know that a may be in bazel-out/a

I think I should take @DanielRosenwasser up on a suggestion that we have a brief design discussion. After studying this problem for a week, it seems that Bazel's output directory is simply incompatible with typescript project references and one of them needs to change. Do you have a design meeting this week? Thanks!

[alexeagle]

I hoped that --incremental false would disable this up-to-date checking (Bazel already does it) but I get

error TS6379: Composite projects may not disable incremental compilation.

is there any other way to avoid this checking?

[alexeagle]

@RyanCavanaugh thanks for the discussion today. I noted that you said TS repo itself is affected by this issue, requiring that you have multiple tsconfig files to express the difference between two different build options.

I have another proposal based on some more experimentation.

Take this scenario. We've already compiled lib and now want to compile app.

lib/
    tsconfig.json   // composite: true
bazel-out/mac/lib
    index.js
    index.d.ts
    tsconfig.tsbuildinfo
app/
    tsconfig.json   // "references": [  {"path": "../lib"}  ]}
    index.ts

tsc -p app --outDir bazel-out/mac/app

As we know with TS today, this fails because the reference goes to the lib/tsconfig which doesn't specify an outDir, so TS will look for lib/index.d.ts and not find it.

Let's say the semantics for references were expanded (maybe under some new flag) so that path: ../lib could be resolved relative to the outDir of the current build, meaning bazel-out/mac/app/../lib
(I think you suggested in the meeting that this makes sense to you)

This can work, but only if we also add a build step to copy the tsconfig file to the output directory - and we know that's complex because the paths need to be made relative to the new location.

So instead, since there is no tsconfig.json file in the output directory, we'd want this to resolve one step further to bazel-out/mac/lib/tsconfig.tsbuildinfo.

I could imagine that a reference to a tsbuildinfo file could be made to work in place of a reference to a tsconfig file? If so this seems simple and in my basic testing it works.

[alexeagle]

#37509 shows the change to resolving reference paths that I'm currently using for prototyping. Along with a hack in Bazel to copy tsconfig into the outDir, this seems to work.

[alexeagle]

@sheetalkamat FYI we shipped the new TS support for Bazel with a bunch of ugly caveats about this and related bugs
https://bazelbuild.github.io/rules_nodejs/TypeScript#ts_project

What do you or Ryan think of the proposal to resolve reference paths into the outDir?

[sheetalkamat]

c:: @RyanCavanaugh @DanielRosenwasser
I am not sure if that's what we are looking for.. Resolving project reference from two different places means that (not only project but when processing dependencies and everything) I remember that design mentting we needed more general approach rather than solving just outDir issue?