Skip to content

Commit b49b022

Browse files
committed
Revert "ci(test-shards): grade the Test Core split on predicted shard wall and slice the CLI per run (#22415)"
This reverts commit c64130b. Why: PR #22415 packs about 2000-3000 s of predicted windows onto each whole-package Test Core shard. The dataset's per-package windows were measured on shards carrying about 1,400 s, and under 4-wide concurrency each package's window inflates on the denser shards (objectql 1.84x, plugin-security 1.80x, rest 1.61x, plugin-auth 1.87x). That carries the shard past the "Check this shard's timing drift" step's 1.5x red while the test step itself passes. Red full runs on that partition model: - main push 37894048074 (e02833c; shards 4/6 and 6/6; 6/6 read 3992.4s measured vs 2467.1s predicted = 1.62x) - merge_group builds 37894050587, 37893672824, 37894053453 and 37892033675 The revert restores .github/workflows/ci.yml and scripts/partition-test-shards.mjs byte-for-byte to their content at f05649f, the reverted commit's single parent. A contention-aware redesign is the next round, not a hotfix. Claude-Session: https://claude.ai/code/session_0115N1oNnQS5WqofZ2DzaT3q Co-authored-by: Claude <noreply@anthropic.com>
1 parent e02833c commit b49b022

2 files changed

Lines changed: 184 additions & 576 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 72 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -412,41 +412,45 @@ jobs:
412412
# shard, because every shard gets its own runner.
413413
#
414414
# SIX specifically, and not four or five: sharding is BY PACKAGE, so no
415-
# shard can finish faster than the heaviest serial task it carries. On the
416-
# current dataset (scripts/test-shard-timings.json, the 21-run refresh at
417-
# 040184752c) that task is `@objectstack/cli` at 1738.88s, which no shard
418-
# count fixes by itself: whole, it is 2.52x the mean predicted shard wall at
419-
# six. So it is cut into 3 file-level slices whenever a run's own split
420-
# needs it (#22075; FILE_SHARDED_PACKAGES and planShards() in the script),
421-
# and the split is graded on predicted shard WALL, not on summed weights --
422-
# a shard runs its whole packages four at a time (`--concurrency=4` below)
423-
# and each slice after them, so a bin's sum is not its wall. Re-derived with
424-
# the partitioner's own planShards(), partition() and shardWalls():
415+
# shard can finish faster than its single heaviest package. On the
416+
# current dataset (scripts/test-shard-timings.json as #21826 refreshed it,
417+
# provenance run 37262126122) that package is `@objectstack/cli` at
418+
# 1702.69s, ahead of `@objectstack/spec` at 1134.86s. At five shards or
419+
# fewer the partitioner must co-schedule it with others (heaviest bin
420+
# 1957s at five); at six it fills bin 1 alone and the other five sit at
421+
# 1615-1617s. Six is therefore the smallest count that isolates the
422+
# heaviest indivisible suite; past six, the CLI's shard cannot improve,
423+
# only the others can. (spec held that place until #21826 measured the
424+
# CLI whole: the dataset before it recorded the CLI at 733.33s, summed
425+
# from two file-level slices, and #21487 had already retired the slicing.)
425426
#
426-
# 4 shards -> max 1071s mean 1011s ratio 1.06x
427-
# 5 shards -> max 910s mean 816s ratio 1.12x
428-
# 6 shards -> max 667s mean 663s ratio 1.01x
429-
# 7 shards -> max 580s mean 568s ratio 1.02x
430-
# 8 shards -> max 580s mean 517s ratio 1.12x
431-
# 10 shards -> max 580s mean 455s ratio 1.27x
432-
#
433-
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS, then in summed
434-
# seconds. #10472 replaced the count (a proxy ~2.8x off on the CLI: six
435-
# count-balanced bins ran 5.0/6.2/6.3/13.6/6.3/9.0 min in run 32428961038);
436-
# #22075 replaced the sum, which read 1.00-1.47x on the runs where shard
437-
# 1/6 -- the CLI alone -- ran 2.3-7.6x the other five. Each time the
438-
# conclusion (six) survived and only the units it is argued in changed.
427+
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins
428+
# 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now
429+
# made in measured seconds, because #10472 replaced the weight input: the
430+
# file count was a proxy that ran ~2.8x off on @objectstack/cli, and the
431+
# six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/
432+
# 9.0 min in run 32428961038. The conclusion (six) survived the re-
433+
# measurement; only the units it is argued in changed. Weights now come
434+
# from scripts/test-shard-timings.json — see partition-test-shards.mjs.
439435
#
440436
# AND WHY NOT MORE THAN SIX (#10472 asked for 6 -> 8 to be considered).
441-
# The floor is the heaviest serial task -- one CLI slice, 580s, beside
442-
# `@objectstack/spec`'s 573s -- and it does not move when shards are added,
443-
# while the mean falls with every shard. A seventh shard would take the
444-
# predicted maximum from 667s to that floor; past seven the maximum does not
445-
# move at all, so more shards buy only more runners' fixed overhead and a
446-
# worse ratio. The partitioner's self-test pins the half of this that can
447-
# fail: at SHARD_COUNT the heaviest serial task must sit within 1.3x the mean
448-
# shard wall, and when it stops fitting, slicing it further is the remedy
449-
# that failure names.
437+
# The floor is the heaviest single package, and it does not move when
438+
# shards are added, while the mean falls with every shard — so max/mean,
439+
# which is what the acceptance bound is written in, gets WORSE past the
440+
# point where the floor becomes the max. Re-derived on the current dataset
441+
# with the partitioner's own partition() and balanceOf():
442+
#
443+
# 6 shards -> max 1703s mean 1630s ratio 1.04x
444+
# 7 shards -> max 1703s mean 1397s ratio 1.22x
445+
# 8 shards -> max 1703s mean 1223s ratio 1.39x (past the 1.3x bound)
446+
# 10 shards -> max 1703s mean 978s ratio 1.74x
447+
#
448+
# Past six the critical path does not move at all — the CLI's 1703s is
449+
# the maximum at every count — so more shards buy only more runners'
450+
# fixed overhead and a worse ratio. The partitioner's self-test pins the
451+
# half of this that can fail: at SHARD_COUNT the heaviest package must
452+
# sit within 1.3x the mean, and when it stops fitting, slicing it below
453+
# package granularity is the remedy that failure names.
450454
#
451455
# ⚠ THE COST, stated because it is real: per-shard fixed overhead
452456
# (checkout + pnpm/Turbo cache restore + install, ~60s measured on run
@@ -474,50 +478,28 @@ jobs:
474478
# margin for a cold Turbo cache; the old 45 left a hung job "running" for
475479
# half an hour past any plausible healthy finish.
476480
#
477-
# ⚠ RAISED 30 -> 45 (#16173, the #16445 "temporary" raise): shard 5/6 was
478-
# being killed at the 30-minute wall, and after the dataset refreshes
479-
# (#20388, #21826) the split ran the whole CLI alone on shard 1/6, whose job
480-
# wall then reached 35m43s (run 37453598388).
481-
#
482-
# HELD AT 45 BY #22075; THE RE-SIZE IS OWED AFTER IT LANDS, carried by
483-
# #22075. That change stops running the whole CLI on one shard (it is cut
484-
# into 3 slices whenever a run carries it), which removes the reason for
485-
# this raise -- but a wall is re-sized from a MEASURED distribution, and
486-
# the post-change one does not exist until runs execute the new split.
487-
# What the re-size will start from, and nothing here has been applied:
488-
#
489-
# INPUT 1, measured before the change. Window: the 14 runs after the 21-run
490-
# dataset refresh (040184752c), 6 pull_request and 8 merge_group,
491-
# 37870616843 to 37876969409 (2026-10-09T00:03Z-02:57Z), from the jobs API:
492-
#
493-
# Test Core job holding the whole CLI 20.5-39.1 min (12 runs)
494-
# every other Test Core job <= 20.8 min (72 jobs)
495-
# `Build this shard's dependency closure` <= 5.6 min; fixed setup <= 2.4 min
496-
#
497-
# INPUT 2, PREDICTED, not measured. The CLI's test step read 18.7-32.2 min
498-
# whole, so its heaviest slice (vitest's hash split puts 1.14x of an even
499-
# third on slice 2/3) should read at most ~12.2 min, plus its own
500-
# `cli#build`, the closure and setup: <= ~24 min of job. The busiest
501-
# whole-package shard of a FULL run will carry ~2640s of windows, a size no
502-
# run has executed; at the 2.0x packing measured on ~1400s shards that is
503-
# <= ~22 min of tests, <= ~30 min of job. The wall model reads LOW on
504-
# whole-package legs (run 37872770181: predicted 573/442/442/442/491s,
505-
# stepped 905/737/704/668/388s), so these two figures are a floor for the
506-
# reading, never a substitute for it.
507-
#
508-
# THE RE-SIZE: once at least 3 pull_request and 2 merge_group runs print
509-
# `slicing: @objectstack/cli: sliced x3` in "Compute this shard's package
510-
# set", take the slowest Test Core JOB wall of each from the jobs API and
511-
# size this value from that distribution, with the margin rule below.
481+
# ⚠ RAISED 30 -> 45 (#16173), and the raise is still load-bearing. It was
482+
# made because shard 5/6 was being killed at the 30-minute wall (12+
483+
# observations at 30:16-30:21 with every other job green; the green band
484+
# topped out at 28:56, a ~1 minute margin), and so the tail could be
485+
# measured uncensored before anyone re-derived
486+
# `scripts/test-shard-timings.json`. Both are done: the dataset was
487+
# refreshed (#20388, then #21826) and the split now runs the whole CLI
488+
# alone on shard 1/6 (#21487).
512489
#
513-
# ⛔ REVERT CONDITION, against measured wall time, not a card: back to `30`
514-
# only when the slowest Test Core JOB wall stays at or under 24 minutes (80%
515-
# of 30) on every scheduled run for a week. A miss here kills merge_group
516-
# runs for every lane, so this value moves on a reading, never on a
517-
# prediction. `partition-test-shards.mjs` prints each shard's predicted
518-
# wall in the "Compute this shard's package set" step, so the prediction
519-
# sits beside the measurement it is checked against. The stall guard (a
520-
# 20-minute cap) stays the primary hang detector.
490+
# ⛔ REVERT CONDITION, restated against measured wall time, not a card:
491+
# back to `30` only when the slowest shard's JOB wall time stays at or
492+
# under 24 minutes (80% of 30, so the margin the old wall lacked is
493+
# there) on every scheduled run for a week. It does not today. On the 16
494+
# main runs after the #21826 refresh (37413386379 to 37467882762), shard
495+
# 1/6 — the CLI alone — read 6m14s-35m43s of job wall time, over 30
496+
# minutes on 9 of them (34m39s run 37415122516, 35m43s run 37453598388,
497+
# 34m08s run 37460325624), so the condition as first written ("once
498+
# #16173 lands") would now kill that shard; and its slowest reading is
499+
# already 79% of this 45. Read the `Test Core (1/6)` job's started and
500+
# completed times before touching this value. It carries no other expiry
501+
# — a raise with no revert condition beside it becomes permanent by
502+
# forgetting — and the stall guard stays the primary hang detector.
521503
timeout-minutes: 45
522504
permissions:
523505
contents: read
@@ -680,8 +662,8 @@ jobs:
680662
# ⛔ THIS SHARD'S `^build` CLOSURE IS BUILT HERE, ON THE TURBO REMOTE
681663
# CACHE (#22077), so `Run this shard's tests` REPLAYS it from the local
682664
# cache instead of building it. Before this step the closure was built
683-
# inside the test step's own turbo run (the slice step below ran zero
684-
# iterations from #21487 to #22075). Merge-queue run 37623284168, `Test Core
665+
# inside the test step's own turbo run (the slice step below runs zero
666+
# iterations since #21487). Merge-queue run 37623284168, `Test Core
685667
# (3/6)`: `Tasks: 72 successful, 72 total` / `Cached: 1 cached, 72
686668
# total` / `Time: 11m43.373s` for the tests AND their closure in one
687669
# run. That run cannot read the remote, because `test` / `test:repo`
@@ -695,9 +677,8 @@ jobs:
695677
# Build Core's hash. A `test` task that `dependsOn: ["build"]` (cli and
696678
# metadata in turbo.json) also needs its OWN package's build. This filter
697679
# leaves that out on purpose, so a shard never builds a package its tests
698-
# do not need; the test step builds it as before. Today that is
699-
# `cli#build`, on each shard carrying a CLI slice: the slice step below
700-
# builds it there.
680+
# do not need; the test step builds it as before. Today that is one task,
681+
# `cli#build` on shard 1/6.
701682
#
702683
# Its own guarded step, for the reason the slice step below gives: a
703684
# guarded SITE is (file, job, step). No `--summarize`: `.turbo/runs/`
@@ -778,13 +759,11 @@ jobs:
778759
# and `pnpm check:stall-guard-headroom` both read this step, so it keeps
779760
# its own `--stall-minutes` and its own headroom row.
780761
#
781-
# A shard with no slice runs zero iterations here. Since #22075 a run
782-
# that carries `@objectstack/cli` cuts it into 3 slices on 3 shards
783-
# whenever its own split needs them (partition-test-shards.mjs
784-
# planShards(), which prints its decision in "Compute this shard's
785-
# package set"), so those three shards run one iteration each. Every
786-
# shard still reaches the step, so its name is a stable site for those
787-
# two gates.
762+
# A shard with no slice runs zero iterations here, and since #21487
763+
# emptied FILE_SHARDED_PACKAGES that is every shard: no package is sliced
764+
# today, so the paragraphs above describe the mechanism a slice would
765+
# use, not anything this job currently runs. Every shard still reaches
766+
# the step, so its name is a stable site for those two gates.
788767
- name: Build the sliced package's dependency closure
789768
env:
790769
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
@@ -903,11 +882,9 @@ jobs:
903882
# an unmoved `PKG#test` across a change outside the package; the
904883
# completeness guard below reads both tasks' summaries. A slice leg
905884
# stays `test`-only: the CLI, the one package whose slice wiring is
906-
# in the tree, is not split. Since #22075 the CLI is cut into 3
907-
# slices whenever a run's own split needs it, so the three shards
908-
# carrying one run the whole-package leg and then the slice leg, in
909-
# that order -- which is why the partitioner weighs a slice as
910-
# holding its shard for its entire window.
885+
# in the tree, is not split. No package is sliced today
886+
# (FILE_SHARDED_PACKAGES is empty since #21487), so every shard runs
887+
# the whole-package leg alone and the slice branch below is idle.
911888
STATUS=0
912889
LOGS=""
913890
for LEG in __whole__ $SLICES; do
@@ -1079,15 +1056,9 @@ jobs:
10791056
# with no packages runs no turbo and writes no summary; that is NOT
10801057
# MEASURED here rather than a usage error from the script.
10811058
#
1082-
# The dataset it reads is scripts/test-shard-timings.json as the 21-run
1083-
# refresh left it (040184752c): the whole CLI at 1738.88s. Since #22075 a
1084-
# shard carrying one of its 3 slices is predicted a third of that, from
1085-
# the slice count the shard's own summary records (the OS_TEST_SHARD
1086-
# digest) rather than from the config, so the ratio stays one shard's
1087-
# executed windows against their own prediction on every run, whatever
1088-
# the affected set -- the meaning the 1.5x red and the 1.3x warning were
1089-
# written against. Measured before the slicing, on merge_group
1090-
# 37875522518: shard 1/6 (the whole CLI) 1.07x, shard 2/6 0.74x.
1059+
# The dataset it reads is scripts/test-shard-timings.json as #21826
1060+
# refreshed it (provenance run 37262126122): the whole CLI at 1702.69s,
1061+
# alone on shard 1/6 since FILE_SHARDED_PACKAGES went empty (#21487).
10911062
# The second sample taken before wiring this, executed windows only, from
10921063
# the `Test Core` timing tables of the main runs after that refresh:
10931064
# shard 1/6 read 0.61x-1.05x on all eight scheduled runs (37416453417 to

0 commit comments

Comments
 (0)