Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion experimental/unplugin-perf/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,15 @@ For `N` TypeScript files, the harness measures per simulated build:
- **`fs.statSync` calls** — synchronous validation work. Scenario D adds 100 unrelated nested directories and requires generation-scoped membership notification rather than one directory-stat pass per module.
- **fs identity probes:** the `existsSync`/`realpathSync.native` call volume paid by the shared path-identity resolver on the real host platform. Correct watch-input derivation pays it once per distinct graph path per generation, not once per module delivery.

The guarded invariants are **`plugin runs == 1`**, **`stats/file` within the membership budget** (a missing resolution candidate is proven by notification rather than re-probed, samchon/ttsc#1261), **`lstats/file` within the budget the scenario's own envelope justifies** (declared by the serve scenarios whose shape bounds it, D and F), and, for the graph scenario, **bounded probes per module** — the harness exits non-zero when any of them breaks.
The guarded invariants are **`plugin runs == 1`**, **`plugin runs == 1` across repeated passes** (scenario G), **`stats/file` within the membership budget** (a missing resolution candidate is proven by notification rather than re-probed, samchon/ttsc#1261), **`lstats/file` within the budget the scenario's own envelope justifies** (declared by the serve scenarios whose shape bounds it, D and F), and, for the graph scenario, **bounded probes per module** — the harness exits non-zero when any of them breaks.

- **Scenario A — output keys under the project root.** The cache hits, so the project is transformed once. (`reads` still grow with `N`: validating a cache hit re-hashes the project to detect a sibling-file change — bounded work that the existing invalidation contract requires.)
- **Scenario B — one output key outside the validator's directory walk** (a `node_modules/**` path, exactly what the native host emits for program dependencies). Before the fix the store-time and validate-time hash key sets diverged, the cache _never_ hit, and the whole project was re-transformed once per module (`plugin runs == N`); now the cache hits and `plugin runs == 1`.
- **Scenario C — a graph-bearing envelope (the typia >= 13.1.19 shape).** The sidecar stamps `graph` (every module edges to its next sibling plus `K` external `node_modules` declarations, and one missing resolution candidate) and per-file `dependencies`. Before the #1007 fix every cache-hit delivery re-walked the whole graph: probes per module grew linearly with the edge count (6.7k/12k/22.8k probes per module at E=2.6k/5.1k/10.1k, ~76-95 s for 99 deliveries), the O(modules x edges) syscall storm behind the #970 residual stall on macOS. The gate is 64 probes per module; the fixed code derives once per generation and stays flat regardless of `E`.
- **Scenario D: the same envelope without a build boundary** (the Vite development server's persistent-validation mode). Each module owns one disjoint external input, and the project contains 100 unrelated nested directories. The gates require reads to stay bounded by that file's inputs and synchronous stats not to grow with either the whole-envelope union or project directory count.
- **Scenario E — the same serve mode over a _shared_ closure.** Every module reaches the same externals and the same `graph.globals` (the shape a real program produces, where the globals are whatever `@types/*` packages declare). Scenario D's partition hides this: with a shared closure the pre-#1222 code re-read and re-hashed the whole closure for every delivered module, so reads/file grew with the closure instead of staying flat. The per-file read gate is the same one Scenario D uses, and it now holds only because an unchanged nanosecond metadata signature stands in for the content comparison. Its stat gate is its own: a missing resolution candidate cannot be proven absent by metadata, so each one reachable from the delivered file costs one failed `stat`, and a shared closure makes that set grow with the module count rather than staying flat.
- **Scenario F: the same serve shape from a producer that declares completeness.** The sidecar stamps `dependenciesComplete` for every file it reports, which is what `@ttsc/banner` and `@ttsc/strip` do today and what samchon/typia#2357 asks typia for. The delivered file's derived set collapses from the whole closure to the reported dependencies plus the config chain, which is the sound way to stop validating what a transform never consulted: measured on this fixture it is the difference between 127.5 and 54.0 `lstat` calls per delivery.
- **Scenario G: repeated build passes over an unchanged project.** Scenarios A-C do open two passes, a warm-up and a measured one, and D-F open none. Their `plugin runs == 1` held only because the second pass threw the first pass's generation away, so the harness was measuring the per-pass clear rather than gating against it, and a boundary that discarded a valid compile on every rebuild could never have failed one of them (samchon/ttsc#1302). This scenario discards the warm-up's generation explicitly and then runs three passes that change nothing, so the whole run must cost one compile; a per-pass clear makes it cost one per pass, which is the shape samchon/ttsc#1300 reported from a webpack watch session.

The adapter source is bundled on the fly with esbuild (with `ttsc` and `unplugin` kept external), so the production code path runs unmodified — no rebuilt `lib` required.

Expand Down
86 changes: 81 additions & 5 deletions experimental/unplugin-perf/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,21 @@ async function main(): Promise<void> {
}),
);

console.log("\nScenario G: repeated build passes over an unchanged project");
console.log(
" invariant: plugin runs == 1 across every pass, not one per pass;",
);
console.log(
" a pass boundary is a statement about deliveries, not about whether the",
);
console.log(" compiled program is still correct\n");
for (const count of [25, 50]) {
recordFailure(
failures,
await measureRepeatedPasses(adapter, { count, emitExternalKey: false }),
);
}

if (failures.length !== 0) {
console.error(
`\nFAIL: a scenario violated its invariant:\n ${failures.join("\n ")}`,
Expand All @@ -173,8 +188,8 @@ async function main(): Promise<void> {
return;
}
console.log(
"\nOK: every build ran exactly one whole-project transform and watch-input" +
" derivation stayed bounded per module.",
"\nOK: every build ran exactly one whole-project transform, repeated passes" +
" reused it, and watch-input derivation stayed bounded per module.",
);
}

Expand All @@ -189,6 +204,7 @@ interface Adapter {
createTtscTransformCache(
operations?: Record<string, unknown>,
): Map<string, Promise<unknown>>;
resetTtscTransformCache(cache: Map<string, Promise<unknown>>): void;
resolveOptions(options?: unknown): unknown;
transformTtsc(
id: string,
Expand Down Expand Up @@ -374,7 +390,11 @@ async function measure(

// Warm-up build: pays the one-time Go plugin compile + native program load so
// the timed run reflects steady-state per-module cost, not toolchain startup.
// Its generation is discarded, because a delivery pass keeps one now: without
// the reset the measured build would reuse the warm-up's compile and this
// scenario's `plugin runs == 1` would read 0 (samchon/ttsc#1300).
await runBuild(harness, project, runLog);
harness.adapter.resetTtscTransformCache(harness.cache);

resetCounters(harness);
fs.writeFileSync(runLog, "");
Expand All @@ -401,6 +421,59 @@ async function measure(
: `scenario ${scenario} N=${options.count}: pluginRuns=${pluginRuns} (expected 1)`;
}

/**
* Gate the dimension every other scenario is blind to: cost _across_ passes.
*
* Scenarios A-C do open two passes, a warm-up and a measured one, and D-F open
* none. But their `plugin runs == 1` held only _because_ the second pass threw
* the first pass's generation away, so the harness was measuring the per-pass
* clear rather than gating against it, and a boundary that discarded a valid
* compile on every rebuild could never have failed one of them
* (samchon/ttsc#1302).
*
* Nothing changes between the passes here, so the whole run must cost one
* compile. A per-pass clear makes it cost one per pass, which is the shape
* samchon/ttsc#1300 reported from a webpack watch session.
*/
async function measureRepeatedPasses(
adapter: Adapter,
options: MeasureOptions,
): Promise<string | undefined> {
const passes = 3;
const project = createProject(options);
const harness = createTransformHarness(adapter, project);
const runLog = pluginRunLog(project);

// Warm-up pass: pays the one-time Go plugin compile and native program load.
// Its generation is discarded so the measured passes start from an empty
// cache and the count below is theirs alone.
await runBuild(harness, project, runLog);
harness.adapter.resetTtscTransformCache(harness.cache);

resetCounters(harness);
fs.writeFileSync(runLog, "");
const timings: number[] = [];
for (let pass = 0; pass < passes; pass += 1) {
const started = process.hrtime.bigint();
await runBuild(harness, project, runLog);
timings.push(Number(process.hrtime.bigint() - started) / 1e6);
}

const pluginRuns = fs.existsSync(runLog)
? fs.readFileSync(runLog, "utf8").length
: 0;
console.log(
` N=${String(options.count).padStart(3)} ` +
`passes=${passes} ` +
`pluginRuns=${String(pluginRuns).padStart(3)} ` +
`reads=${String(harness.counters.reads).padStart(7)} ` +
`perPassMs=${timings.map((value) => value.toFixed(0)).join("/")}`,
);
return pluginRuns === 1
? undefined
: `scenario G N=${options.count}: pluginRuns=${pluginRuns} across ${passes} unchanged passes (expected 1)`;
}

/**
* An fs probe pair (`existsSync` + `realpathSync.native`) is what one
* `pathIdentityKey` call costs on macOS. A bounded watch-input derivation pays
Expand All @@ -427,6 +500,9 @@ async function measureGraphBuild(
const runLog = pluginRunLog(project);

await runBuild(harness, project, runLog);
// See the note in `measure`: the warm-up's generation now survives a pass, so
// it is discarded before the measured build.
harness.adapter.resetTtscTransformCache(harness.cache);

const modules = projectModules(project);
const context = {
Expand All @@ -437,9 +513,9 @@ async function measureGraphBuild(
};
fs.writeFileSync(runLog, "");
process.env.PLUGIN_RUN_LOG = runLog;
adapter.beginTtscTransformBuild(harness.cache);
harness.adapter.beginTtscTransformBuild(harness.cache);
const [first, ...rest] = modules;
await adapter.transformTtsc(
await harness.adapter.transformTtsc(
first!,
fs.readFileSync(first!, "utf8"),
harness.options,
Expand All @@ -451,7 +527,7 @@ async function measureGraphBuild(
resetCounters(harness);
const started = process.hrtime.bigint();
for (const id of rest) {
await adapter.transformTtsc(
await harness.adapter.transformTtsc(
id,
fs.readFileSync(id, "utf8"),
harness.options,
Expand Down
Loading
Loading