Skip to content

fix(unplugin): the transform cache never hits, so every module recompiles the whole project #1245

Description

@samchon

Problem

Every bundler module delivery recompiles the whole project. @ttsc/unplugin's transform cache never hits in any project that imports an npm package whose types resolve through a .d.ts, which is every real application. A build's cost is therefore modules x whole-project compile instead of one compile, and that is the residual stall reported in #970 (it survives #980, #1009, #1017, #1224 and #1228).

Current behaviour, measured on master (1250f0c70, 0.28.0), vite 8.2.1 + typia 14.0.0 + typescript 7.0.2, an 11-module project:

lifecycle modules delivered whole-project compiles
vite serve (the mode vitest runs in) 11 11
vite build 11 11

Expected: 1 compile per build, the invariant experimental/unplugin-perf claims to guard.

Affected users: every @ttsc/unplugin consumer (all adapters) and @ttsc/metro, on every platform. The larger the project, the worse: the reporter's react-router application never completes.

Evidence

Reproduction

A project of 8 modules, 5 chained type files, 3 typia.createValidate calls, a ~/* tsconfig+vite alias, typia@14.0.0 installed. Drive the real @ttsc/unplugin/vite plugin object over the modules with vite.createServer(...).transformRequest(id) (what vite-node/vitest does) and count TtscCompiler.prototype.transform calls. One compile is expected; one compile per delivered module is observed.

Root cause

Instrumented run of matchesCachedSource / transformProject:

generation walk {beforeComplete:true, afterComplete:true, sameHashes:true, sameDirectories:true}
external snapshot incomplete {count:11, total:1185}
graph proofs: member/proof mismatch {conflicts:[], proofs:123, members:134}
generation {stable:false, graphProofs:false, externalComplete:false}
validate {snapshotComplete:false} -> "complete: no stable generation snapshot"

The 11 unproven graph members are all host-owned resolution candidates:

node_modules/typia/lib/index.ts
node_modules/typia/lib/index.tsx
node_modules/@standard-schema/spec/dist/index.ts
node_modules/@standard-schema/spec/dist/index.tsx
node_modules/node_modules/@typia/interface/package.json
node_modules/typia/node_modules/@standard-schema/spec/package.json
src/node_modules/typia/package.json
...

Two facts meet:

  1. The host cannot prove a candidate. driver.SupersedingModuleCandidates (packages/ttsc/driver/resolution_candidates.go) enumerates candidates speculatively and touches the filesystem through raw os.Stat / os.ReadFile (sameExistingPath, packageManifestCandidates), never through inputObservationFS. Most candidates are pure path arithmetic the compiler never probed at all: a candidate is by construction strictly ahead of the target that won, so the resolver stopped before reaching it. TransformGraph.attachInputProof skips every input inputObserver.proof cannot answer, so graph.inputHashes has no entry for them.

  2. The consumer demands a proof for every member. matchesCompilerGraphInputProofs in packages/unplugin/src/core/transform.ts rejects unless graph.inputProofs.size === graph.members.size, and captureExternalInputSnapshot marks the whole external snapshot incomplete for a member without a proof. graph.members includes every candidate.

Requiring a compile-time content proof for a path the compiler deliberately never read is a category error, and it makes projectSnapshotComplete permanently false.

Why that costs a compile per module

projectSnapshotComplete !== true closes all three validation paths at once - the build-scoped first-delivery shortcut, matchesNarrowPersistentInputs, and matchesCompleteInputSnapshot (which returns false at its first gate). A refusal evicts the generation, the next module recompiles the project, that generation is unstable for exactly the same structural reason, and the loop never converges. The absence of a proof is treated as evidence of a change, against the rule the narrow path states for itself ("losing the proof is not evidence of a change").

Version boundary

NewTransformGraph is host-owned and stamped for every transform, so this is not a typia defect. typia only decides which candidates exist: 13.1.19+/14 added the @standard-schema/spec and @typia/interface dependencies, whose resolution produces the unproven candidate set. That is exactly the typia 13.1.1-vs-13.2/14 boundary the reporter measured. The proof gate itself landed in 0.27.0 (bca474df7), which is where the reporter's suite went from 13 s to 40 s with old typia and to "never completes" with typia 14.

Why no test caught it

experimental/unplugin-perf asserts plugin runs == 1, and tests/test-unplugin/src/features/transform/* asserts cache hits, but both drive a hand-written synthetic Go sidecar whose envelope carries a complete inputHashes map and no missing resolution candidates. No fixture in the repository resolves a real node_modules package through .d.ts, so the one shape that fails was never exercised.

Consequence surface

surface effect
@ttsc/unplugin vite/rollup/rolldown/webpack/rspack/esbuild/farm/next/turbopack cache permanently off, one project compile per module
@ttsc/metro same core, same effect
Vite serve / vitest worst case: no build boundary, so every module pays a full compile
Vite/rollup build same, the build-scoped shortcut is also gated on projectSnapshotComplete
watch and HMR each rebuild repeats it
persistent bundler caches unaffected in correctness, only in cost

Every project with at least one dependency typed by a .d.ts is affected; a project with no npm dependency at all is not, which is why the fixtures pass.

Approach

The invariant to restore: a generation the compiler produced from a coherent input state stays reusable for every module of that project until one of its inputs changes.

  • Distinguish the two evidence classes the envelope already carries. A realized input (an edge source or target, a global, a config) is a file the compiler read and must keep its compile-time proof. A member the envelope reports only under graph.candidates is speculative: it has no compile-time read to prove, so it is validated against the state recorded when the envelope was produced, exactly like a plugin-declared dependency path. A candidate that is also a realized input keeps the realized standard, and a contradictory proof (inputProofConflicts) still rejects.
  • Do not let the absence of a proof void a whole generation. Degrade per member, never globally.
  • Make the residual case loud: when a generation still cannot be reused, say so once on stderr with the reason, instead of silently multiplying the build by the module count.

Architectural owner: packages/unplugin/src/core/transform.ts (envelopeGraphIndexes, matchesCompilerGraphInputProofs, captureExternalInputSnapshot). The host contract in packages/ttsc/driver/graph.go is already honest about what it can prove; the consumer's reading of it is what must change. If the host is changed instead, it must not attach a post-compile probe as if it were a compile-time proof.

Acceptance and verification

  • Positive: a project that resolves a node_modules package through .d.ts delivers N modules with exactly 1 whole-project compile, in both a build-scoped host and a persistent (serve) host.
  • Negative: a realized graph member without a proof still refuses reuse; a conflicting proof still refuses; a candidate that appears on disk still invalidates the generation and forces a recompile; an edited project source still invalidates.
  • Boundary: an envelope with no inputHashes at all (legacy sidecar) keeps its current behaviour; an envelope where a candidate is also an edge target keeps the realized standard; a candidate under the project walk and one outside it behave the same.
  • Regression fixture: the repository needs one that exercises the real host graph, not a synthetic envelope - a fixture project whose node_modules contains a package typed by index.d.ts - asserting one compile across many deliveries. Without it this recurs.
  • Commands: pnpm --filter @ttsc/test-unplugin start, pnpm --dir experimental/unplugin-perf start, pnpm test:go.

Measured with a prototype of the classification above, same fixture:

lifecycle before after
serve, 11 modules 11 compiles / 167 s 1 compile / 48 s
build, 11 modules 11 compiles 1 compile

Coordination

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions