Skip to content

Commit c5ad1de

Browse files
build: one ts.Program per DTS pass — patch tsup's bundled rollup-plugin-dts grouping (spec 4997 → 882 MB live heap) (#20499)
Fixes #20419 Clause-②: no ## What changes This is round 2 of the card: the root-cause fix. The seat ruled route A, a pnpm patch of tsup's bundled rollup-plugin-dts, and the maintainer's veto window runs until this PR lands. Round 1 (#20483) landed the `noCheck` mitigation. - **`patches/tsup@8.5.1.patch`** changes 5 lines in `dist/rollup.js`, which is tsup's bundled rollup-plugin-dts 6.1.1. When tsup passes a tsconfig path, every entry is now grouped by the tsconfig's directory. Before, an entry that hit the config cache kept its own directory, so each entry directory got its own `ts.Program`. The patch was written with `pnpm patch` / `pnpm patch-commit`. - **`pnpm-workspace.yaml`** gets `patchedDependencies: tsup@8.5.1`, with a comment on why the patch exists and what a tsup bump owes. `patch-commit` wrote the key into `package.json`'s `pnpm` field; it was moved here, and `package.json` is unchanged. - **`pnpm-lock.yaml`** was regenerated by `pnpm install`. The diff is the `patchedDependencies` block plus the `patch_hash` on each tsup resolution (+16 / −11). No other resolution moved. The lockfile was byte-identical before and after the key moved from `package.json` to the yaml. - **`packages/spec/tsup.config.ts`** changes only its docblock: the one-program mechanism, the tsup-bump duty, today's table, and the wider equality needed to compare trees. The 6144 ceiling is unchanged. No package source changes. ## Census: which packages build more than one program **Static census.** All 67 workspace packages whose scripts run tsup were checked. Each config was loaded the way tsup loads it (`bundle-require`), with the build script's `--config` and positional entries applied, and the plugin's grouping simulated. 12 packages had more than one program; the other 55 have one DTS entry. **Empirical census.** The declaration pass of every package in the census closure (31 packages) was built with a `ts.createProgram` probe preloaded into the DTS worker. It confirms the 12. Programs before → after the patch: | package | before | after | |---|---|---| | `@objectstack/spec` | 18 | 1 | | `@objectstack/platform-objects` | 11 | 1 | | `@objectstack/metadata` | 3 (roots 1+3+1) | 1 | | `@objectstack/metadata-core` (two config items) | 1+2 | 1+1 | | `@objectstack/service-cluster` (two config items) | 1+2 | 1+1 | | `@objectstack/core`, `lint`, `objectql`, `plugin-auth`, `plugin-webhooks`, `service-datasource`, `types` | 2 each | 1 each | | the 18 single-entry packages in the closure | 1 | 1 | Round 1's estimate was off in both directions. It listed `create-objectstack`, whose DTS pass has one entry (`dts.entry`), and it missed `platform-objects`, whose object-form `entry` gave 11 programs. ## Measurements: spec at the 6144 ceiling in an 8192 MB cgroup **Method.** - The DTS pass ran exactly as the build script spells it, plus `--trace-gc`. - It ran in a cgroup-v1 memory cgroup capped at 8192 MB. Round 1 proved that cap kills first. - The tree is `8113763026` (with `noCheck`) against the same tree plus the patch (`78cbf5dfbf`). Spec's source is identical in both. - Wall times are shared-box readings. | | programs | live heap after a mark-compact | largest heap before one | peak RSS | wall | |---|---|---|---|---|---| | DTS pass, unpatched | 18 | 4997 MB | 5425 MB | 5694 MB | 153 s | | DTS pass, patched | 1 (18 roots, 396 emits) | 882 MB (4 mark-compacts, so a lower bound) | 1677 MB | 3537 MB | 41 s | | whole `build` script, unpatched | | | | 5740 MB | 189 s (DTS 148 s) | | whole `build` script, patched | | | | 3526 MB | 87 s (DTS 45 s) | The ceiling now has about 4.5 GB of headroom. It stays at 6144; lowering it is a separate change. ## The patch cannot rot silently (measured) All three cases ran with the repo's pnpm 10.31.0, from `packageManager`. - **tsup moves off 8.5.1** (a scratch project with the same patch file and key, and tsup 8.5.0 installed): `pnpm install` exits 1 with `ERR_PNPM_UNUSED_PATCH The following patches were not used: tsup@8.5.1`, and `--frozen-lockfile` exits 1 with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. - **Positive control** (the same scratch project on tsup 8.5.1): the install exits 0, and the installed `dist/rollup.js` carries the patch marker once. - **The yaml key is what pnpm reads** (in this tree): with the `patchedDependencies` block removed, `pnpm install --frozen-lockfile` exits 1 with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`, which is what CI's frozen install would hit. The removal was done with `scripts/ablation-replace.mjs`, and the restore was proven blob-equal to HEAD. ## The emitted declarations are the same types Per the seat's Q2 answer: canonical equality with negative controls replaces byte identity. **Trees compared.** All 31 packages of the census closure were built twice, in dependency order: unpatched at `8113763026`, then patched at `78cbf5dfbf`. Only tsup differs between the two builds. Every `dist/**/*.d.{ts,mts,cts}` was saved each time. **Canonical form.** Each file is re-printed through the TypeScript printer with: - union members sorted; - type literals made only of property signatures sorted; - top-level statements sorted; - `import` / `export` specifiers sorted; - rollup's chunk hashes stripped. Shared chunks are paired by chunk-name-agnostic content, and the pairing must be unique on both sides. **Results.** - 29 of 31 packages are byte-identical before and after, including 11 of the 12 census packages. - `@objectstack/spec`: 130 files, 0 differ canonically. One chunk is paired by content: `data-engine` (unpatched) is `analytics.zod` (patched). The files differ only in the ways round 1 recorded. - `@objectstack/metadata-protocol` has one entry, so the patch does not change its program count. It differs in bytes only, through union order inlined from spec, and is canonically equal even with statement sorting off. - The other 29 packages are canonically equal as well: 0 differing files, 0 pairing problems. **Negative controls.** In a copy of each patched tree, the first real `string` type node became `number`, located by the TypeScript parser and never inside a comment. The control ran on: - one entry file in each of the 12 census packages (for `platform-objects`, `identity/index.d.ts`, because its `index.d.ts` only re-exports); - one chunk file in each census package that has chunks: `lint`, `metadata-core`, `objectql`, and spec's renamed `analytics.zod` chunk. Every control was caught, and each one named exactly the mutated file. A mutated chunk also fails the content pairing ("no partner"). One control (`platform-objects`) was first pointed at a directory, errored, and was re-run on a real file. That first attempt is not counted. ## Gates, at the merged head (`57e5191f21` = this branch + `origin/main` `fc0db22bcf`) `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derived 60 commands over 4 changed paths. **Derived set.** All 60 derived gates exited 0. `--ran` reconciliation: "60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED". That includes: - `check:override-consistency`, `check:vendor-export-contract-resolve`, `check:workspace-manifest-cycles` and `check:osv-exemptions`; - `check:dts-closure`: 72 built packages, 166/166 declared declaration files present; - `check:dual-build-cjs-loads`; - `check:lean-entry-closure`. **Builds.** - `turbo run build --force` over the census closure: 29/29 tasks, 0 cached. - The rest of the workspace: `turbo run build --filter=!@objectstack/docs`, 72/72 tasks. - `pnpm --filter @objectstack/spec build` inside the 8192 MB cgroup: exit 0, peak RSS 3680 MB, 77.6 s, DTS 38.7 s. - `check-dts-emitted`: 36/36. - `check-dts-references`: 130 files, 394/394. - `pnpm --filter @objectstack/spec check:generated`: exit 0, all 15 artifacts up to date. - `pnpm install --frozen-lockfile`: exit 0. **Tests.** The 8 spec test files that read `tsup.config.ts` as text: 130 passed. **Scope.** The patch reaches only the DTS worker: tsup's `index.js` loads `dist/rollup.js` in one place, `new Worker(… "./rollup.js")`. The JS passes are untouched. **Changeset.** None; `skip-changeset`. Per Q2: canonical equality holds for every census package, no JS output or shipped source moves, and 29 of 31 packages are byte-identical. ## Acceptance notes - **The census counts DTS entries, not `entry`.** A tsup package whose `dts.entry` narrows to one file (`create-objectstack`) has one program, whatever its `entry` says. Object-form `entry` counts too (`platform-objects`). - **Chunk names and statement order moved in spec only.** The other 11 census packages emit identical bytes. Those are the kinds of drift two unpatched builds of spec already show, so anything that compares spec's declaration trees by byte digest reads phantom changes (round 1's note stands). - **Every tsup bump now owes the patch a decision.** pnpm refuses an unused patch, as measured above, so the bump PR is where the patch is re-derived or retired. Upstream: rollup-plugin-dts 6.5.1 still has the same code. The report below is for the maintainer to file. - **Upstream reporting stays with the maintainer.** Nothing was written outside this org. ## Draft upstream report (for the maintainer to file at rollup-plugin-dts) > **Title:** `createPrograms` builds one `ts.Program` per entry directory when a `tsconfig` path is passed > > **Version:** rollup-plugin-dts 6.5.1 (also 6.1.1, as bundled by tsup 8.5.1). > > **What happens:** In `getCompilerOptions`, when `overrideConfigPath` (the plugin's `tsconfig` option) is set, the config cache key is that path. The first entry misses the cache, and `dirName` becomes the config's directory. Every later entry hits the cache, and `dirName` stays `path.dirname(input)`. `createPrograms` starts a new program whenever `dirName` changes between consecutive inputs, so entries in different directories each get their own `ts.Program`. Each of those programs parses, binds, and declaration-emits its whole reachable graph again. > > **Impact:** In a package with 18 entries in 18 directories, the declaration pass built 18 programs. It needed a 4997 MB live heap and 153 s. With one program it needed an 882 MB live heap and 41 s, and the emitted declarations were the same types. tsup always passes `tsconfig`, so every multi-entry tsup build takes this path. > > **Suggested fix:** On a cache hit with `overrideConfigPath` set, return the config's directory. Equivalently: `if (overrideConfigPath) dirName = path.dirname(path.resolve(process.cwd(), overrideConfigPath));` before the cached options are read. > > **Reproduction:** Two entries, `src/a/index.ts` and `src/b/index.ts`, passed with `tsconfig: 'tsconfig.json'`. Count the `ts.createProgram` calls: 2. With the line above: 1. --- _Generated by [Claude Code](https://claude.ai/code/session_014EJ1ED8X4MMrT18BhVx4tx)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4a1df19 commit c5ad1de

4 files changed

Lines changed: 89 additions & 41 deletions

File tree

‎packages/spec/tsup.config.ts‎

Lines changed: 38 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -180,55 +180,63 @@ const swapServerOnlyGrammarArm: Plugin = {
180180
* it, and the pass's non-heap overhead measures ~250 MB, so the bound is
181181
* ~6.4 GB inside an 8 GB container.
182182
*
183-
* WHY THE PASS IS THIS HEAVY: ONE `ts.Program` PER ENTRY. tsup 8.5.1 runs the
184-
* pass through its bundled rollup-plugin-dts 6.1.1, whose `createPrograms`
185-
* groups entries by a directory key. tsup always passes the tsconfig path, and
186-
* on that path every entry after the first is keyed by its OWN directory (the
187-
* same code is in rollup-plugin-dts 6.5.1). So each entry above gets its own
188-
* program, which parses, binds and emits its whole reachable graph again: the
189-
* peak grows with entries × graph, not with the graph. ⇒ Every entry added to
190-
* `entries` adds a program to this pass.
183+
* ONE `ts.Program` FOR ALL ENTRIES — held by a patch, not by tsup. tsup 8.5.1
184+
* runs this pass through its bundled rollup-plugin-dts 6.1.1, whose
185+
* `createPrograms` groups entries by a directory key. tsup always passes the
186+
* tsconfig path, and on that path every entry after the first was keyed by its
187+
* OWN directory (the same code is in rollup-plugin-dts 6.5.1). So each entry
188+
* above got its own program, which parsed, bound and emitted its whole
189+
* reachable graph again: the peak grew with entries × graph, not with the
190+
* graph. `patches/tsup@8.5.1.patch` (wired in `pnpm-workspace.yaml`'s
191+
* `patchedDependencies`) keys every entry by the tsconfig's directory, so the
192+
* 18 entries share one program. The patch names the exact tsup version, and
193+
* `pnpm install` refuses a patch that matches no installed package, so a tsup
194+
* bump cannot drop it silently. ⇒ On a tsup bump, re-derive the patch or
195+
* retire it, then re-measure this table. If this pass ever shows 18 programs
196+
* again, the entries are back to one program each.
191197
*
192-
* `noCheck`: rollup-plugin-dts forces `noEmitOnError`, which made every one of
193-
* those programs also semantically CHECK each file it emitted — a type check
198+
* `noCheck`: rollup-plugin-dts forces `noEmitOnError`, which makes the program
199+
* also semantically CHECK each file it emits — a type check
194200
* the `typecheck` script (`tsc --noEmit` over this same tsconfig, run by the
195201
* required `TypeScript Type Check` job) already performs once. `noCheck` drops
196202
* only that duplicate. Syntactic, option, global and declaration diagnostics
197203
* still fail the pass, so a declaration that cannot be emitted still stops the
198204
* build; a plain type error in `src/` is `typecheck`'s to report, not this
199205
* pass's.
200206
*
201-
* Measured on 8cdbe0c6e5's source, DTS pass alone at the 6144 ceiling, inside a
207+
* Measured on 8113763026's source, DTS pass alone at the 6144 ceiling, inside a
202208
* cgroup capped at 8192 MB. Live heap is the largest heap V8 kept after a
203209
* mark-compact (`--trace-gc`); peak RSS is the peak anonymous RSS of the whole
204-
* process tree:
210+
* process tree; wall times are shared-box readings:
205211
*
206-
* pass live heap peak RSS wall
207-
* duplicate check on (before) 5658 MB 6177 MB 181-194s
208-
* noCheck (this config) 5083 MB 5889 MB 134s
209-
* one program, grouping patched 1379 MB 3749 MB 53s
210-
* `tsc --noEmit`, whole package 1103 MB 1149 MB 18s
212+
* pass (both with noCheck) programs live heap peak RSS wall
213+
* one program per entry (unpatched) 18 4997 MB 5694 MB 153s
214+
* one program (patched, this tree) 1 882 MB 3537 MB 41s
211215
*
212-
* The third row was measured with the grouping patched in a copy of tsup
213-
* outside this tree. It shows what cutting the program count is worth.
216+
* The patched run did only 4 mark-compacts, so its live-heap figure is a
217+
* lower bound; its largest heap before a mark-compact was 1677 MB. The whole
218+
* `build` script went 189s → 87s. For scale, `tsc --noEmit` over the whole
219+
* package peaked at 1103 MB live on 8cdbe0c6e5.
214220
*
215221
* NOT BYTE-STABLE: this pass does not emit the same bytes twice. TypeScript
216222
* prints union members, and the members of object types derived from them, in
217-
* type-creation order, and that order follows emit order. Three runs of the
218-
* same commit gave three different trees, and rollup's content-hashed chunk
219-
* names moved with them. Once union and property-signature order are
220-
* normalised, all three runs and the `noCheck` run are the same tree (128
221-
* files, 30389539 bytes). ⇒ Compare two declaration trees in such an
222-
* order-insensitive form, never by a byte digest.
223+
* type-creation order, and that order follows emit order. Three runs of one
224+
* commit gave three different trees, and rollup's content-hashed chunk names
225+
* moved with them. The one-program pass also reorders top-level statements
226+
* and `import`/`export` specifiers, and names one shared chunk after a
227+
* different module (`data-engine` → `analytics.zod`). Normalise all of those
228+
* (sort union and property-only type-literal members, statements and
229+
* specifiers; strip chunk hashes; pair chunks by content), and the unpatched
230+
* and patched trees are the same tree: 130 files, 0 differing. ⇒ Compare two
231+
* declaration trees in that order-insensitive form, never by a byte digest.
223232
*
224233
* If this pass starts failing with `ERR_WORKER_OUT_OF_MEMORY`, the live type
225234
* graph has outgrown 6144 — that is a loud, actionable failure and the point of
226235
* the ceiling. ⛔ Do not "fix" it by raising the number past what the build
227-
* container has; that trades this error back for the silent exit 137. Cut the
228-
* program count, shrink the graph, or split the pass across entries. Cutting
229-
* the count moves statement order and one chunk name beyond the noise above. A
230-
* split redraws the shared chunks. Either one changes what publishes, so it
231-
* needs that reviewed first.
236+
* container has; that trades this error back for the silent exit 137. First
237+
* check that the pass still builds ONE program (above). Then shrink the graph,
238+
* or split the pass across entries. A split redraws the shared chunks and
239+
* changes what publishes, so it needs that reviewed first.
232240
*/
233241
const isDts = process.env.BUILD_DTS === 'true';
234242

‎patches/tsup@8.5.1.patch‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
diff --git a/dist/rollup.js b/dist/rollup.js
2+
index e128b61b9558318b9f86dc11d72c31f09a8eb7db..35f85e39c2af6c645fb2d3da115742df445d2343 100644
3+
--- a/dist/rollup.js
4+
+++ b/dist/rollup.js
5+
@@ -4915,6 +4915,11 @@ var require_rollup_plugin_dts = _chunkTWFEYLU4js.__commonJS.call(void 0, {
6+
} else {
7+
logCache("HIT", cacheKey);
8+
}
9+
+ if (overrideConfigPath) {
10+
+ // objectstack patch: a cache HIT kept dirname(input), so each entry
11+
+ // got its own ts.Program. One tsconfig for all entries => one key.
12+
+ dirName = path__namespace.dirname(path__namespace.resolve(process.cwd(), overrideConfigPath));
13+
+ }
14+
const { fileNames, options, errors } = configByPath.get(cacheKey);
15+
dtsFiles = fileNames.filter((name) => DTS_EXTENSIONS.test(name));
16+
if (errors.length) {

‎pnpm-lock.yaml‎

Lines changed: 16 additions & 11 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pnpm-workspace.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,25 @@ onlyBuiltDependencies:
1818
- msw
1919
- sharp
2020

21+
# Patched dependencies. Written with `pnpm patch` / `pnpm patch-commit`, never
22+
# by hand. `patch-commit` writes this key into package.json's `pnpm` field;
23+
# move it here, beside `overrides` and `onlyBuiltDependencies`, then run
24+
# `pnpm install`.
25+
# - tsup@8.5.1: its bundled rollup-plugin-dts gave every DTS entry its own
26+
# ts.Program. tsup always passes the tsconfig path, and on that path a
27+
# config-cache hit keyed each later entry by its own directory. The patch
28+
# keys every entry by the tsconfig's directory, so all entries share one
29+
# program. The same code is in rollup-plugin-dts 6.5.1. Twelve packages
30+
# here have more than one DTS entry; @objectstack/spec had 18 programs
31+
# and now has one, and its DTS pass went from 4997 MB to 882 MB of live
32+
# heap. Every one of the twelve emits the same types as before.
33+
# The key names the exact version. When the installed tsup is any other
34+
# version, the patch goes unused and `pnpm install` fails
35+
# (ERR_PNPM_UNUSED_PATCH), so a tsup upgrade cannot drop the patch
36+
# silently. Re-derive the patch, or retire it, on every tsup bump.
37+
patchedDependencies:
38+
tsup@8.5.1: patches/tsup@8.5.1.patch
39+
2140
# Transitive-dependency version pins. pnpm v10 reads `overrides` from THIS file
2241
# — the `pnpm.overrides` block in package.json is silently ignored — so all
2342
# pins must live here (previously orphaned in package.json: minimatch, tar).

0 commit comments

Comments
 (0)