@@ -412,41 +412,45 @@ jobs:
412412 # shard, because every shard gets its own runner.
413413 #
414414 # SIX specifically, and not four or five: sharding is BY PACKAGE, so no
415- # shard can finish faster than the heaviest serial task it carries. On the
416- # current dataset (scripts/test-shard-timings.json, the 21-run refresh at
417- # 040184752c) that task is `@objectstack/cli` at 1738.88s, which no shard
418- # count fixes by itself: whole, it is 2.52x the mean predicted shard wall at
419- # six. So it is cut into 3 file-level slices whenever a run's own split
420- # needs it (#22075; FILE_SHARDED_PACKAGES and planShards() in the script),
421- # and the split is graded on predicted shard WALL, not on summed weights --
422- # a shard runs its whole packages four at a time (`--concurrency=4` below)
423- # and each slice after them, so a bin's sum is not its wall. Re-derived with
424- # the partitioner's own planShards(), partition() and shardWalls():
415+ # shard can finish faster than its single heaviest package. On the
416+ # current dataset (scripts/test-shard-timings.json as #21826 refreshed it,
417+ # provenance run 37262126122) that package is `@objectstack/cli` at
418+ # 1702.69s, ahead of `@objectstack/spec` at 1134.86s. At five shards or
419+ # fewer the partitioner must co-schedule it with others (heaviest bin
420+ # 1957s at five); at six it fills bin 1 alone and the other five sit at
421+ # 1615-1617s. Six is therefore the smallest count that isolates the
422+ # heaviest indivisible suite; past six, the CLI's shard cannot improve,
423+ # only the others can. (spec held that place until #21826 measured the
424+ # CLI whole: the dataset before it recorded the CLI at 733.33s, summed
425+ # from two file-level slices, and #21487 had already retired the slicing.)
425426 #
426- # 4 shards -> max 1071s mean 1011s ratio 1.06x
427- # 5 shards -> max 910s mean 816s ratio 1.12x
428- # 6 shards -> max 667s mean 663s ratio 1.01x
429- # 7 shards -> max 580s mean 568s ratio 1.02x
430- # 8 shards -> max 580s mean 517s ratio 1.12x
431- # 10 shards -> max 580s mean 455s ratio 1.27x
432- #
433- # ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS, then in summed
434- # seconds. #10472 replaced the count (a proxy ~2.8x off on the CLI: six
435- # count-balanced bins ran 5.0/6.2/6.3/13.6/6.3/9.0 min in run 32428961038);
436- # #22075 replaced the sum, which read 1.00-1.47x on the runs where shard
437- # 1/6 -- the CLI alone -- ran 2.3-7.6x the other five. Each time the
438- # conclusion (six) survived and only the units it is argued in changed.
427+ # ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins
428+ # 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now
429+ # made in measured seconds, because #10472 replaced the weight input: the
430+ # file count was a proxy that ran ~2.8x off on @objectstack/cli, and the
431+ # six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/
432+ # 9.0 min in run 32428961038. The conclusion (six) survived the re-
433+ # measurement; only the units it is argued in changed. Weights now come
434+ # from scripts/test-shard-timings.json — see partition-test-shards.mjs.
439435 #
440436 # AND WHY NOT MORE THAN SIX (#10472 asked for 6 -> 8 to be considered).
441- # The floor is the heaviest serial task -- one CLI slice, 580s, beside
442- # `@objectstack/spec`'s 573s -- and it does not move when shards are added,
443- # while the mean falls with every shard. A seventh shard would take the
444- # predicted maximum from 667s to that floor; past seven the maximum does not
445- # move at all, so more shards buy only more runners' fixed overhead and a
446- # worse ratio. The partitioner's self-test pins the half of this that can
447- # fail: at SHARD_COUNT the heaviest serial task must sit within 1.3x the mean
448- # shard wall, and when it stops fitting, slicing it further is the remedy
449- # that failure names.
437+ # The floor is the heaviest single package, and it does not move when
438+ # shards are added, while the mean falls with every shard — so max/mean,
439+ # which is what the acceptance bound is written in, gets WORSE past the
440+ # point where the floor becomes the max. Re-derived on the current dataset
441+ # with the partitioner's own partition() and balanceOf():
442+ #
443+ # 6 shards -> max 1703s mean 1630s ratio 1.04x
444+ # 7 shards -> max 1703s mean 1397s ratio 1.22x
445+ # 8 shards -> max 1703s mean 1223s ratio 1.39x (past the 1.3x bound)
446+ # 10 shards -> max 1703s mean 978s ratio 1.74x
447+ #
448+ # Past six the critical path does not move at all — the CLI's 1703s is
449+ # the maximum at every count — so more shards buy only more runners'
450+ # fixed overhead and a worse ratio. The partitioner's self-test pins the
451+ # half of this that can fail: at SHARD_COUNT the heaviest package must
452+ # sit within 1.3x the mean, and when it stops fitting, slicing it below
453+ # package granularity is the remedy that failure names.
450454 #
451455 # ⚠ THE COST, stated because it is real: per-shard fixed overhead
452456 # (checkout + pnpm/Turbo cache restore + install, ~60s measured on run
@@ -474,50 +478,28 @@ jobs:
474478 # margin for a cold Turbo cache; the old 45 left a hung job "running" for
475479 # half an hour past any plausible healthy finish.
476480 #
477- # ⚠ RAISED 30 -> 45 (#16173, the #16445 "temporary" raise): shard 5/6 was
478- # being killed at the 30-minute wall, and after the dataset refreshes
479- # (#20388, #21826) the split ran the whole CLI alone on shard 1/6, whose job
480- # wall then reached 35m43s (run 37453598388).
481- #
482- # HELD AT 45 BY #22075; THE RE-SIZE IS OWED AFTER IT LANDS, carried by
483- # #22075. That change stops running the whole CLI on one shard (it is cut
484- # into 3 slices whenever a run carries it), which removes the reason for
485- # this raise -- but a wall is re-sized from a MEASURED distribution, and
486- # the post-change one does not exist until runs execute the new split.
487- # What the re-size will start from, and nothing here has been applied:
488- #
489- # INPUT 1, measured before the change. Window: the 14 runs after the 21-run
490- # dataset refresh (040184752c), 6 pull_request and 8 merge_group,
491- # 37870616843 to 37876969409 (2026-10-09T00:03Z-02:57Z), from the jobs API:
492- #
493- # Test Core job holding the whole CLI 20.5-39.1 min (12 runs)
494- # every other Test Core job <= 20.8 min (72 jobs)
495- # `Build this shard's dependency closure` <= 5.6 min; fixed setup <= 2.4 min
496- #
497- # INPUT 2, PREDICTED, not measured. The CLI's test step read 18.7-32.2 min
498- # whole, so its heaviest slice (vitest's hash split puts 1.14x of an even
499- # third on slice 2/3) should read at most ~12.2 min, plus its own
500- # `cli#build`, the closure and setup: <= ~24 min of job. The busiest
501- # whole-package shard of a FULL run will carry ~2640s of windows, a size no
502- # run has executed; at the 2.0x packing measured on ~1400s shards that is
503- # <= ~22 min of tests, <= ~30 min of job. The wall model reads LOW on
504- # whole-package legs (run 37872770181: predicted 573/442/442/442/491s,
505- # stepped 905/737/704/668/388s), so these two figures are a floor for the
506- # reading, never a substitute for it.
507- #
508- # THE RE-SIZE: once at least 3 pull_request and 2 merge_group runs print
509- # `slicing: @objectstack/cli: sliced x3` in "Compute this shard's package
510- # set", take the slowest Test Core JOB wall of each from the jobs API and
511- # size this value from that distribution, with the margin rule below.
481+ # ⚠ RAISED 30 -> 45 (#16173), and the raise is still load-bearing. It was
482+ # made because shard 5/6 was being killed at the 30-minute wall (12+
483+ # observations at 30:16-30:21 with every other job green; the green band
484+ # topped out at 28:56, a ~1 minute margin), and so the tail could be
485+ # measured uncensored before anyone re-derived
486+ # `scripts/test-shard-timings.json`. Both are done: the dataset was
487+ # refreshed (#20388, then #21826) and the split now runs the whole CLI
488+ # alone on shard 1/6 (#21487).
512489 #
513- # ⛔ REVERT CONDITION, against measured wall time, not a card: back to `30`
514- # only when the slowest Test Core JOB wall stays at or under 24 minutes (80%
515- # of 30) on every scheduled run for a week. A miss here kills merge_group
516- # runs for every lane, so this value moves on a reading, never on a
517- # prediction. `partition-test-shards.mjs` prints each shard's predicted
518- # wall in the "Compute this shard's package set" step, so the prediction
519- # sits beside the measurement it is checked against. The stall guard (a
520- # 20-minute cap) stays the primary hang detector.
490+ # ⛔ REVERT CONDITION, restated against measured wall time, not a card:
491+ # back to `30` only when the slowest shard's JOB wall time stays at or
492+ # under 24 minutes (80% of 30, so the margin the old wall lacked is
493+ # there) on every scheduled run for a week. It does not today. On the 16
494+ # main runs after the #21826 refresh (37413386379 to 37467882762), shard
495+ # 1/6 — the CLI alone — read 6m14s-35m43s of job wall time, over 30
496+ # minutes on 9 of them (34m39s run 37415122516, 35m43s run 37453598388,
497+ # 34m08s run 37460325624), so the condition as first written ("once
498+ # #16173 lands") would now kill that shard; and its slowest reading is
499+ # already 79% of this 45. Read the `Test Core (1/6)` job's started and
500+ # completed times before touching this value. It carries no other expiry
501+ # — a raise with no revert condition beside it becomes permanent by
502+ # forgetting — and the stall guard stays the primary hang detector.
521503 timeout-minutes : 45
522504 permissions :
523505 contents : read
@@ -680,8 +662,8 @@ jobs:
680662 # ⛔ THIS SHARD'S `^build` CLOSURE IS BUILT HERE, ON THE TURBO REMOTE
681663 # CACHE (#22077), so `Run this shard's tests` REPLAYS it from the local
682664 # cache instead of building it. Before this step the closure was built
683- # inside the test step's own turbo run (the slice step below ran zero
684- # iterations from #21487 to #22075 ). Merge-queue run 37623284168, `Test Core
665+ # inside the test step's own turbo run (the slice step below runs zero
666+ # iterations since #21487). Merge-queue run 37623284168, `Test Core
685667 # (3/6)`: `Tasks: 72 successful, 72 total` / `Cached: 1 cached, 72
686668 # total` / `Time: 11m43.373s` for the tests AND their closure in one
687669 # run. That run cannot read the remote, because `test` / `test:repo`
@@ -695,9 +677,8 @@ jobs:
695677 # Build Core's hash. A `test` task that `dependsOn: ["build"]` (cli and
696678 # metadata in turbo.json) also needs its OWN package's build. This filter
697679 # leaves that out on purpose, so a shard never builds a package its tests
698- # do not need; the test step builds it as before. Today that is
699- # `cli#build`, on each shard carrying a CLI slice: the slice step below
700- # builds it there.
680+ # do not need; the test step builds it as before. Today that is one task,
681+ # `cli#build` on shard 1/6.
701682 #
702683 # Its own guarded step, for the reason the slice step below gives: a
703684 # guarded SITE is (file, job, step). No `--summarize`: `.turbo/runs/`
@@ -778,13 +759,11 @@ jobs:
778759 # and `pnpm check:stall-guard-headroom` both read this step, so it keeps
779760 # its own `--stall-minutes` and its own headroom row.
780761 #
781- # A shard with no slice runs zero iterations here. Since #22075 a run
782- # that carries `@objectstack/cli` cuts it into 3 slices on 3 shards
783- # whenever its own split needs them (partition-test-shards.mjs
784- # planShards(), which prints its decision in "Compute this shard's
785- # package set"), so those three shards run one iteration each. Every
786- # shard still reaches the step, so its name is a stable site for those
787- # two gates.
762+ # A shard with no slice runs zero iterations here, and since #21487
763+ # emptied FILE_SHARDED_PACKAGES that is every shard: no package is sliced
764+ # today, so the paragraphs above describe the mechanism a slice would
765+ # use, not anything this job currently runs. Every shard still reaches
766+ # the step, so its name is a stable site for those two gates.
788767 - name : Build the sliced package's dependency closure
789768 env :
790769 NODE_OPTIONS : --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
@@ -903,11 +882,9 @@ jobs:
903882 # an unmoved `PKG#test` across a change outside the package; the
904883 # completeness guard below reads both tasks' summaries. A slice leg
905884 # stays `test`-only: the CLI, the one package whose slice wiring is
906- # in the tree, is not split. Since #22075 the CLI is cut into 3
907- # slices whenever a run's own split needs it, so the three shards
908- # carrying one run the whole-package leg and then the slice leg, in
909- # that order -- which is why the partitioner weighs a slice as
910- # holding its shard for its entire window.
885+ # in the tree, is not split. No package is sliced today
886+ # (FILE_SHARDED_PACKAGES is empty since #21487), so every shard runs
887+ # the whole-package leg alone and the slice branch below is idle.
911888 STATUS=0
912889 LOGS=""
913890 for LEG in __whole__ $SLICES; do
@@ -1079,15 +1056,9 @@ jobs:
10791056 # with no packages runs no turbo and writes no summary; that is NOT
10801057 # MEASURED here rather than a usage error from the script.
10811058 #
1082- # The dataset it reads is scripts/test-shard-timings.json as the 21-run
1083- # refresh left it (040184752c): the whole CLI at 1738.88s. Since #22075 a
1084- # shard carrying one of its 3 slices is predicted a third of that, from
1085- # the slice count the shard's own summary records (the OS_TEST_SHARD
1086- # digest) rather than from the config, so the ratio stays one shard's
1087- # executed windows against their own prediction on every run, whatever
1088- # the affected set -- the meaning the 1.5x red and the 1.3x warning were
1089- # written against. Measured before the slicing, on merge_group
1090- # 37875522518: shard 1/6 (the whole CLI) 1.07x, shard 2/6 0.74x.
1059+ # The dataset it reads is scripts/test-shard-timings.json as #21826
1060+ # refreshed it (provenance run 37262126122): the whole CLI at 1702.69s,
1061+ # alone on shard 1/6 since FILE_SHARDED_PACKAGES went empty (#21487).
10911062 # The second sample taken before wiring this, executed windows only, from
10921063 # the `Test Core` timing tables of the main runs after that refresh:
10931064 # shard 1/6 read 0.61x-1.05x on all eight scheduled runs (37416453417 to
0 commit comments