Repository navigation
Commit c5ad1de
build: one ts.Program per DTS pass — patch tsup's bundled rollup-plugin-dts grouping (spec 4997 → 882 MB live heap) (#20499)
Fixes #20419
Clause-②: no
## What changes
This is round 2 of the card: the root-cause fix. The seat ruled route A,
a pnpm patch of tsup's bundled rollup-plugin-dts, and the maintainer's
veto window runs until this PR lands. Round 1 (#20483) landed the
`noCheck` mitigation.
- **`patches/tsup@8.5.1.patch`** changes 5 lines in `dist/rollup.js`,
which is tsup's bundled rollup-plugin-dts 6.1.1. When tsup passes a
tsconfig path, every entry is now grouped by the tsconfig's directory.
Before, an entry that hit the config cache kept its own directory, so
each entry directory got its own `ts.Program`. The patch was written
with `pnpm patch` / `pnpm patch-commit`.
- **`pnpm-workspace.yaml`** gets `patchedDependencies: tsup@8.5.1`, with
a comment on why the patch exists and what a tsup bump owes.
`patch-commit` wrote the key into `package.json`'s `pnpm` field; it was
moved here, and `package.json` is unchanged.
- **`pnpm-lock.yaml`** was regenerated by `pnpm install`. The diff is
the `patchedDependencies` block plus the `patch_hash` on each tsup
resolution (+16 / −11). No other resolution moved. The lockfile was
byte-identical before and after the key moved from `package.json` to the
yaml.
- **`packages/spec/tsup.config.ts`** changes only its docblock: the
one-program mechanism, the tsup-bump duty, today's table, and the wider
equality needed to compare trees. The 6144 ceiling is unchanged.
No package source changes.
## Census: which packages build more than one program
**Static census.** All 67 workspace packages whose scripts run tsup were
checked. Each config was loaded the way tsup loads it
(`bundle-require`), with the build script's `--config` and positional
entries applied, and the plugin's grouping simulated. 12 packages had
more than one program; the other 55 have one DTS entry.
**Empirical census.** The declaration pass of every package in the
census closure (31 packages) was built with a `ts.createProgram` probe
preloaded into the DTS worker. It confirms the 12. Programs before →
after the patch:
| package | before | after |
|---|---|---|
| `@objectstack/spec` | 18 | 1 |
| `@objectstack/platform-objects` | 11 | 1 |
| `@objectstack/metadata` | 3 (roots 1+3+1) | 1 |
| `@objectstack/metadata-core` (two config items) | 1+2 | 1+1 |
| `@objectstack/service-cluster` (two config items) | 1+2 | 1+1 |
| `@objectstack/core`, `lint`, `objectql`, `plugin-auth`,
`plugin-webhooks`, `service-datasource`, `types` | 2 each | 1 each |
| the 18 single-entry packages in the closure | 1 | 1 |
Round 1's estimate was off in both directions. It listed
`create-objectstack`, whose DTS pass has one entry (`dts.entry`), and it
missed `platform-objects`, whose object-form `entry` gave 11 programs.
## Measurements: spec at the 6144 ceiling in an 8192 MB cgroup
**Method.**
- The DTS pass ran exactly as the build script spells it, plus
`--trace-gc`.
- It ran in a cgroup-v1 memory cgroup capped at 8192 MB. Round 1 proved
that cap kills first.
- The tree is `8113763026` (with `noCheck`) against the same tree plus
the patch (`78cbf5dfbf`). Spec's source is identical in both.
- Wall times are shared-box readings.
| | programs | live heap after a mark-compact | largest heap before one
| peak RSS | wall |
|---|---|---|---|---|---|
| DTS pass, unpatched | 18 | 4997 MB | 5425 MB | 5694 MB | 153 s |
| DTS pass, patched | 1 (18 roots, 396 emits) | 882 MB (4 mark-compacts,
so a lower bound) | 1677 MB | 3537 MB | 41 s |
| whole `build` script, unpatched | | | | 5740 MB | 189 s (DTS 148 s) |
| whole `build` script, patched | | | | 3526 MB | 87 s (DTS 45 s) |
The ceiling now has about 4.5 GB of headroom. It stays at 6144; lowering
it is a separate change.
## The patch cannot rot silently (measured)
All three cases ran with the repo's pnpm 10.31.0, from `packageManager`.
- **tsup moves off 8.5.1** (a scratch project with the same patch file
and key, and tsup 8.5.0 installed): `pnpm install` exits 1 with
`ERR_PNPM_UNUSED_PATCH The following patches were not used: tsup@8.5.1`,
and `--frozen-lockfile` exits 1 with
`ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`.
- **Positive control** (the same scratch project on tsup 8.5.1): the
install exits 0, and the installed `dist/rollup.js` carries the patch
marker once.
- **The yaml key is what pnpm reads** (in this tree): with the
`patchedDependencies` block removed, `pnpm install --frozen-lockfile`
exits 1 with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`, which is what CI's
frozen install would hit. The removal was done with
`scripts/ablation-replace.mjs`, and the restore was proven blob-equal to
HEAD.
## The emitted declarations are the same types
Per the seat's Q2 answer: canonical equality with negative controls
replaces byte identity.
**Trees compared.** All 31 packages of the census closure were built
twice, in dependency order: unpatched at `8113763026`, then patched at
`78cbf5dfbf`. Only tsup differs between the two builds. Every
`dist/**/*.d.{ts,mts,cts}` was saved each time.
**Canonical form.** Each file is re-printed through the TypeScript
printer with:
- union members sorted;
- type literals made only of property signatures sorted;
- top-level statements sorted;
- `import` / `export` specifiers sorted;
- rollup's chunk hashes stripped.
Shared chunks are paired by chunk-name-agnostic content, and the pairing
must be unique on both sides.
**Results.**
- 29 of 31 packages are byte-identical before and after, including 11 of
the 12 census packages.
- `@objectstack/spec`: 130 files, 0 differ canonically. One chunk is
paired by content: `data-engine` (unpatched) is `analytics.zod`
(patched). The files differ only in the ways round 1 recorded.
- `@objectstack/metadata-protocol` has one entry, so the patch does not
change its program count. It differs in bytes only, through union order
inlined from spec, and is canonically equal even with statement sorting
off.
- The other 29 packages are canonically equal as well: 0 differing
files, 0 pairing problems.
**Negative controls.** In a copy of each patched tree, the first real
`string` type node became `number`, located by the TypeScript parser and
never inside a comment. The control ran on:
- one entry file in each of the 12 census packages (for
`platform-objects`, `identity/index.d.ts`, because its `index.d.ts` only
re-exports);
- one chunk file in each census package that has chunks: `lint`,
`metadata-core`, `objectql`, and spec's renamed `analytics.zod` chunk.
Every control was caught, and each one named exactly the mutated file. A
mutated chunk also fails the content pairing ("no partner"). One control
(`platform-objects`) was first pointed at a directory, errored, and was
re-run on a real file. That first attempt is not counted.
## Gates, at the merged head (`57e5191f21` = this branch + `origin/main`
`fc0db22bcf`)
`node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack
--commands` derived 60 commands over 4 changed paths.
**Derived set.** All 60 derived gates exited 0. `--ran` reconciliation:
"60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED". That
includes:
- `check:override-consistency`, `check:vendor-export-contract-resolve`,
`check:workspace-manifest-cycles` and `check:osv-exemptions`;
- `check:dts-closure`: 72 built packages, 166/166 declared declaration
files present;
- `check:dual-build-cjs-loads`;
- `check:lean-entry-closure`.
**Builds.**
- `turbo run build --force` over the census closure: 29/29 tasks, 0
cached.
- The rest of the workspace: `turbo run build
--filter=!@objectstack/docs`, 72/72 tasks.
- `pnpm --filter @objectstack/spec build` inside the 8192 MB cgroup:
exit 0, peak RSS 3680 MB, 77.6 s, DTS 38.7 s.
- `check-dts-emitted`: 36/36.
- `check-dts-references`: 130 files, 394/394.
- `pnpm --filter @objectstack/spec check:generated`: exit 0, all 15
artifacts up to date.
- `pnpm install --frozen-lockfile`: exit 0.
**Tests.** The 8 spec test files that read `tsup.config.ts` as text: 130
passed.
**Scope.** The patch reaches only the DTS worker: tsup's `index.js`
loads `dist/rollup.js` in one place, `new Worker(… "./rollup.js")`. The
JS passes are untouched.
**Changeset.** None; `skip-changeset`. Per Q2: canonical equality holds
for every census package, no JS output or shipped source moves, and 29
of 31 packages are byte-identical.
## Acceptance notes
- **The census counts DTS entries, not `entry`.** A tsup package whose
`dts.entry` narrows to one file (`create-objectstack`) has one program,
whatever its `entry` says. Object-form `entry` counts too
(`platform-objects`).
- **Chunk names and statement order moved in spec only.** The other 11
census packages emit identical bytes. Those are the kinds of drift two
unpatched builds of spec already show, so anything that compares spec's
declaration trees by byte digest reads phantom changes (round 1's note
stands).
- **Every tsup bump now owes the patch a decision.** pnpm refuses an
unused patch, as measured above, so the bump PR is where the patch is
re-derived or retired. Upstream: rollup-plugin-dts 6.5.1 still has the
same code. The report below is for the maintainer to file.
- **Upstream reporting stays with the maintainer.** Nothing was written
outside this org.
## Draft upstream report (for the maintainer to file at
rollup-plugin-dts)
> **Title:** `createPrograms` builds one `ts.Program` per entry
directory when a `tsconfig` path is passed
>
> **Version:** rollup-plugin-dts 6.5.1 (also 6.1.1, as bundled by tsup
8.5.1).
>
> **What happens:** In `getCompilerOptions`, when `overrideConfigPath`
(the plugin's `tsconfig` option) is set, the config cache key is that
path. The first entry misses the cache, and `dirName` becomes the
config's directory. Every later entry hits the cache, and `dirName`
stays `path.dirname(input)`. `createPrograms` starts a new program
whenever `dirName` changes between consecutive inputs, so entries in
different directories each get their own `ts.Program`. Each of those
programs parses, binds, and declaration-emits its whole reachable graph
again.
>
> **Impact:** In a package with 18 entries in 18 directories, the
declaration pass built 18 programs. It needed a 4997 MB live heap and
153 s. With one program it needed an 882 MB live heap and 41 s, and the
emitted declarations were the same types. tsup always passes `tsconfig`,
so every multi-entry tsup build takes this path.
>
> **Suggested fix:** On a cache hit with `overrideConfigPath` set,
return the config's directory. Equivalently: `if (overrideConfigPath)
dirName = path.dirname(path.resolve(process.cwd(),
overrideConfigPath));` before the cached options are read.
>
> **Reproduction:** Two entries, `src/a/index.ts` and `src/b/index.ts`,
passed with `tsconfig: 'tsconfig.json'`. Count the `ts.createProgram`
calls: 2. With the line above: 1.
---
_Generated by [Claude
Code](https://claude.ai/code/session_014EJ1ED8X4MMrT18BhVx4tx)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4a1df19 commit c5ad1de
4 files changed
Lines changed: 89 additions & 41 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
191 | 197 | | |
192 | | - | |
193 | | - | |
| 198 | + | |
| 199 | + | |
194 | 200 | | |
195 | 201 | | |
196 | 202 | | |
197 | 203 | | |
198 | 204 | | |
199 | 205 | | |
200 | 206 | | |
201 | | - | |
| 207 | + | |
202 | 208 | | |
203 | 209 | | |
204 | | - | |
| 210 | + | |
205 | 211 | | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
211 | 215 | | |
212 | | - | |
213 | | - | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
214 | 220 | | |
215 | 221 | | |
216 | 222 | | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
223 | 232 | | |
224 | 233 | | |
225 | 234 | | |
226 | 235 | | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
232 | 240 | | |
233 | 241 | | |
234 | 242 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
21 | 40 | | |
22 | 41 | | |
23 | 42 | | |
| |||
0 commit comments