Compile multiple packages in a mono-repo in a single compilation step

[bajtos]

In https://github.com/strongloop/loopback-next, we are developing a set of Node.js modules using TypeScript as the language and lerna convention for the monorepo layout.

I am not able to figure out how to configure TypeScript to compile all our packages in a single step, also allowing us to use --watch mode. As a result, whenever we need to transpile our code, we have to run tsc for each package, which adds a lot of friction to our red-green-refactor loop. (I suspect tsc is wasting a lot of time by re-reading type definitions produced by previous tsc runs, which would have been avoided if everything was compiled in one tsc run.)

Let me show a small example to illustrate our project setup.

Source files committed to git:

packages/
  helper/
    src/
      helper.ts
      util.ts

  core/
    src/
      app.ts
      core.ts

Transpiled files that will be eventually published to npmjs.org in two packages (foo and bar):

packages/
  helper/
    lib/
      helper.js
      util.js

  core/
    lib/
      app.js
      core.js

Essentially, I need to configure TypeScript compiler to use outDir=packages/helper/lib when compiling files from packages/helper/src, but use a different outDir value packages/core/lib when compiling files from packages/core/src.

Additional caveats:

• We have interdependencies between our packages, e.g. core depends on helper. lerna is taking care of running the build command in such order that dependencies are built first, dependants second.
• I considered starting multiple tsc --watch processes (one for each package), however think this will not trigger rebuild when a dependency was changed (and rebuilt by another tsc --watch process).

The proposal

Allow outDir option to take a map value configuring different output directories for different source file locations. Alternatively, define a new option (e.g. outDirs or outDirMap).

{
  "outDir": {
    "packages/foo/src/**": "packages/foo/lib",
    "packages/bar/src/**": "packages/bar/lib",
  }
}

Another alternative that comes to my mind is to allow rootDir and outDir to contain a pattern matcher, where the value of the matcher from rootDir will be applied to outDir.

{
  "rootDir": "packages/*/src",
  "outDir": "packages/*/lib",
}

I understand this may be out of scope of TypeScript project (as mentioned in #7752). In which case, is there a compiler API allowing users to control the mapping from source to output files in such fine-grained way?

[ksjogo]

We use webpack and it's entrypoint functionality to compile multiple packages from one repo (and have one watch process). I am not sure if/how types are cached with multiple entrypoints though. The created bundles are consumed by the server, so no third-party is requiring them, so we are fine with bundling everything. You might need to look into the externals configuration option if you want to exclude further things.

[breathe]

I'm pondering this same issue ... I want to convert a monorepo project to use lerna, but I don't want to have a separate compilation context for each package (less project setup, more type checking, more better!).

I'm considering just turning off outDir and letting the typescript compiler generate the .js files adjacent to the source .ts files ...

The only thing I don't like about that solution (that I can think of) is not being able to do a 'safe clean' ... I could define 'safe clean' as remove all .js files from all src/ directories -- but that would prevent incrementally converting js -> ts when necessary ...

Other than that issue -- do think this would work? I haven't thought through @types/ discovery -- not sure if that would magically work right ...

[aluanhaddad]

@breathe if you are incrementally converting, why not leave --outDir enabled and add --allowJs which will be safe. TypeScript will then process both kinds of sources.

[breathe]

@aluanhaddad -- I don't necessarily want to hijack this issue -- but I normally do use --outDir. My above contemplation was about turning off --outDir so that compilation artifacts from child projects in a mono-repo would end up in the correct place.

E.g. -- instead of the parent's desire:

packages/
  helper/
    src/
      helper.ts
      util.ts

  core/
    src/
      app.ts
      core.ts

With transpiled files that will be eventually published to npmjs.org in two packages (foo and bar):

packages/
  helper/
    lib/
      helper.js
      util.js

  core/
    lib/
      app.js
      core.js

Not using --outDir and organizing as follows:

packages/
  helper/
    lib/
      helper.ts
      helper.js
      util.ts
      util.js

  core/
    lib/
      app.ts
      app.js
      core.ts
      core.js

The benefit being a (maybe?) workable strategy for getting a single compilation context for a mono-repo -- with the drawback of making '--allowJs' less safe -- or at least making clean operations less safe.

[DanielRosenwasser]

Hey @bajtos, sorry for the delay here. We've been trying to revisit this sort of scenario a little bit more closely. #3469 documents some of the ideas in play here. While we don't have a concrete solution at this time, we're definitely trying to scope out how to solve this scenario.

[bajtos]

Hello @DanielRosenwasser, thank you for chiming in! I think project references should address our needs pretty well. I am proposing to keep this issue open until #3469 is released and I can verify that it does support what we need. Thoughts?

[jonaskello]

I think a simple solution for monorepo compiles and watch would be to allow multiple -p path/to/tsconfig.json on tsc.

You can have a base tsconfig.json in the root and one in each package that is empty except it extends the one in the root.

If -p would also support globbing it would be really simple to compile and watch all packages with at single invocation of tsc:

tsc --watch -p ./packages/*/tsconfig.json

In case of globbing there is a need to establish compilation order, this could be done by looking for a package.json for each tsconfig.json and then sort out which package is a dependency on another package.

[harrysolovay]

Any progress on this? It would be great to simplify monorepo packages' ts compilation by having a single root-level tsconfig, and dev and build scripts.

[dvdzkwsk]

First-class support would be wonderful, but in case it helps anybody else, here's a workaround we implemented recently:

A project I'm working on (a fairly large monorepo) is not yet able to upgrade to use project references, and until very recently we were running tsc in each package (parallelized, to a degree). This was, as you've all pointed out, quite slow.

Because all of the packages extend same tsconfig, what we did was update the root tsconfig to build to a dist folder in the root, using rootDir so that it mirrored the monorepo directory structure. Once built, a simple script traveresed the directory and copied the emitted files into their respective workspace's own dist folder.

.
├── dist
│   └── packages
│       ├── foo            # this moves to packages/foo/dist
│       │  └── index.js
│       └── bar            # this moves to packages/bar/dist
│          └── index.js
└── packages
        ├── foo
        |   └── index.ts
        └── bar
            └── index.ts

This change dropped build times from 13 minutes for all packages to 1.5 minutes. It required some rewriting of sourcemap paths, but that was a fairly similar process (read all sourcemaps and rewrite sources to be relative to the workspace instead of rootDir). Unfortunately I'm not able to share the exact code because it's for internal build systems, but hopefully that helps someone.