perf(ci): lower the integration suite floor — split the shared integration collection

[mforce]

AMENDED 2026-10-01 — re-scoped again; supersedes the 2026-09-30 amendment
below. That amendment ruled out splitting the integration collection and
said to reopen it "only if a later measurement shows a durable shared-only
tail". That condition is now met, locally.

Two merged changes removed both objections to the split:

• perf(test): reuse one Postgres server for integration databases #1002 (one Postgres server; each fixture clones its own database from a
  migrated template) made an extra collection cost a database clone instead of
  a container start plus 18 migrations.
• perf(test): serialize the five full-seed integration classes #1003 (the five full-demo-seed classes serialized in their own
  demo-seed collection) stopped the heaviest nonshared work contending with
  itself, so the nonshared side now finishes well before the shared side.

Measured on three clean local runs of current main (ad258d8f; 245 classes,
94 in integration), from the full per-class timing file: the shared
collection ends at 316.9–320.4 s of a 317.6–321.0 s suite; the last
substantive nonshared class (DemoSeedAttributionTests, the end of the
demo-seed lane) ends at 227.8–241.1 s. The shared collection runs alone
for 79–93 s at the end of every run. AppDbContextDesignTimeFactoryTests
finishes later but with ~0 s of work — it is a DisableParallelization
collection — so it is not the floor.

New scope: split the shared integration collection, starting at width
two, within the correctness boundary in §D6 of
docs/plans/863-integration-collection-split/01-measurement.md.
Projection, not measurement: two balanced halves of ~308 s of shared work
would end around 165–170 s each, leaving the demo-seed lane (~230–240 s) as
the new floor — a suite near 240 s, roughly 25% below today's ~320 s. Width
three buys nothing past that floor unless the seed lane also shrinks. Two
halves running side by side also add CPU load, so the real figure may land
higher; measure it.

CI must be measured directly. CI's integration time swings ~140 s on
identical code — larger than any change here — so compare ~10 CI runs per
side. The earlier finding that CI's two sides finish within ~23 s of each other
predates #1002 and #1003 and must be re-established, not assumed.

Shipped under this issue so far: #1000 (repaired a DemoSeedTests false
pass), #1002 (~11.5% local), #1003 (~6.9% local).

---

AMENDED 2026-09-30 — re-scoped. The premise below is measured false; do not
plan from it. #863 was "split the serialized integration collection". A
measurement pass (PR #1000,
docs/plans/863-integration-collection-split/01-measurement.md, and
this comment)
established that the serialized collection is no longer the critical path,
so splitting it at any width now buys 0–23.5 s, about 4%. The body's claim
that the other 142 classes "fit inside the block's shadow" held on a 12-core
host; on current CI the nonshared side ends after the shared side, or within
23.5 s of it. The original 331 s → 90 s estimate has no support on current main.

The real floor is a handful of very slow nonshared classes, and two of them
are single-test classes each running a full demo seed:
DemoSeedDisabledOwnerTests 190.9 s for one test and
DemoSeedAttributionTests 182.8 s for one test — 373.7 s of the 1,450.9 s
nonshared work in two tests. Then SeedCommandTests 151.9 s / 8,
ProcessRoleGuardTests 81.4 s, OtlpSubprocessExporterTests 69.5 s / 11,
AccountScopedIdentityMigrationTests 59.0 s, OneShotVerbMinimalConfigTests
58.6 s. These launch the app as a subprocess or create databases. A class
cannot be split, so the tallest of them is the suite's floor no matter what
happens to collections.

New scope: lower the cost of those classes. Share or cache the demo-seed
work the two single-test classes each pay in full; reduce redundant subprocess
launches and per-class database creation. Measure first, exactly as this issue
already required — the lesson that killed the original premise applies again.

Out of scope, unchanged: test deletion (#775 decision 1) and higher worker
counts. Also out of scope now: splitting the integration collection. Reopen
that only if a later measurement shows a durable shared-only tail; the
correctness boundary for doing so safely is recorded in §D6 of the measurement
document and should be re-read rather than re-derived.

Already shipped by the measurement pass: a real false pass in DemoSeedTests
(asserted only IsSuccess, so it passed in 0.25 s without seeding — 18.55 s of
actual work) and database-wide count coupling in AccountProvisioningTests.

---

Split out of #839, which merged as a variance fix (#861). This is the lever that moves the floor.

The problem

dotnet test overlaps collections but runs tests within a collection serially. The shared integration collection is one collection holding 95 classes and 1,024 of 1,815 tests — 316.1s of test work that cannot overlap itself.

That 316s serial block is the suite's wall-clock floor. Measured on the #861 run: 331.5s total wall clock, of which the shared collection occupied 316.1s end to end. #861 stops the block landing at the end of the timeline (the 434s outlier); it cannot shrink the block.

Everything else — 142 classes, 852.4s of test work — already runs in parallel across 12 cores and fits inside the block's shadow. Splitting the shared collection is the only change that lowers the floor.

First task: measure, do not split

#839's own lesson applies. Its stated premise (per-class container startup) was already solved before anyone looked. Before designing a split, establish:

1. What state the 95 classes actually share. The default account, base reference data (grades, roles, packed-unit conversions), Simulation:* config, and any advisory-lock assumptions. The shared fixture exists because they share something; find out what, per class, before assuming two classes are separable.
2. Which classes are already isolated in practice — those that never write, or write only rows nobody else reads. Those are the candidates.
3. What the floor would become at 2, 3, and 4 collections of the resulting groups. A split that produces two 158s blocks halves the floor; one that produces 300s + 16s does nothing.

Constraints

• AGENTS.md requires real Postgres via Testcontainers, never SQLite.
• The Version concurrency-token races, Write guard trusts OriginalValue as DB provenance; detached Update/Remove can bypass the tenant theft check #562's detached tenant writes, and Sales: the Orders list cannot show which orders are unpaid #769's paging defects are only observable against real SQL — a split must not make two racing tests share a database they previously had alone.
• StealLossConnectionReleaseTests already keeps a dedicated factory and one-slot pool; do not fold it back in.
• Any class relying on the shared collection's serialization to pass is a false pass waiting to happen. If a test only passes because nothing else runs concurrently, say so in the split's notes rather than preserving the accident.

Explicitly not this issue

Test deletion and higher worker counts. #775 decision 1 argues against deletion; #776 landed coverage to answer "what is untested", which is never "what is redundant".

Known remaining cost, for whoever sizes this

The parallel work is lumpy: six classes hold 374s of the 852s. OtlpSubprocessExporterTests 80.5s for 11 tests, ProcessRoleGuardTests 71.7s, SeedCommandTests 61.0s, OneShotVerbMinimalConfigTests 58.7s. These repeatedly launch the app as a subprocess. Lowering the collection floor below roughly that height makes this the next floor, so measure both before committing to a split width.

Refs #839

[mforce]

Measurement pass is in PR #1000: #1000

On current main, two local runs passed 1,873 tests. The shared collection has 96 classes and 1,056 tests, with 382-402 s of test work and 383-404 s end-to-end spans; total test wall time was 397-417 s. The other 148 classes contributed 1,350-1,451 s of summed test work. The issue's #861 baseline was 95 classes, 1,024 tests, 316 s shared work, 852 s other work, and 331.5 s wall time.

A concrete two-collection packing is about 201 s per group locally. Three and four collections lower the modeled group maximum to 134 s and 101 s. Independent work finished around 222 s locally, so width two reaches that floor. In two recent main CI runs, the independent side finished only 87 s and 45 s before the suite. With that schedule held fixed, splitting would yield only about 1.20x and 1.09x, before another fixture and worker slot change contention.

I found a reproducible false pass. DemoSeedTests.DemoSeed_PopulatesEveryScreen_AndIsIdempotent passes in roughly 0.24 s after another fact seeded its shared database, but takes 18.55 s when run alone. Its first SeedAsync() result accepts AlreadySeeded, so the full-suite pass does not test the seeding path.

Recommendation: do not split yet. Fix that test and remeasure CI's independent tail. If a split later earns its cost, try two separate Postgres-backed collections first. The PR changes documentation only and does not close #863.

[mforce]

The repaired measurement and fix are together in PR #1000, with the updated diagnosis.

The DemoSeedTests population fact now requires Seeded on its first call and AlreadySeeded on its second. The new first-call assertion went red on the old shared fixture (AlreadySeeded), then passed after the boot and population facts received separate Postgres fixtures. AccountProvisioningTests also received a dedicated fixture for its database-wide count checks. Full local and four PR CI integration runs each passed 1,873 tests.

The issue's premise that the 852 s of parallel work fits inside the serialized block's shadow does not hold on current CI. In four repaired PR runs, the nonshared side finished after the shared collection or within 23.5 s before it. CI test steps ranged from 390.4 to 605.9 s, showing substantial runner variance. Widths two through four have a fixed-schedule modeled gain of 0–23.5 s, about 4% at most, before any extra fixture cost. The document therefore recommends no split now. #863 stays open for a future decision if repeated CI timing shows a durable shared-only tail.

[mforce]

Spike result: PR #1002 carries the one-server change and measurement record. All final-head checks pass.

The exact (271, 1) worker advisory lock conflicts within one database and does not conflict across two databases of the same real Postgres server, in either direction. The prior 138-container total comprised 128 Postgres, nine Redis, and one Ryuk; fixture logs showed 82 factory container starts and 81 host migration calls. Cloning removes those container starts and migration replay, although hosts still check schema state. The new suite creates one Postgres / 11 total. At the default 100-connection limit, the sampled peak was 39 clients across 13 active databases. Twenty parallel template clones completed without failure. The first-run lock-release check was made database-scoped, went red against a genuinely held same-database key (Expected: 0; Actual: 1), and has a committed positive control.

Three local full-suite walls were 388.7 / 388.6 / 384.4 s before and 344.1 / 343.8 / 346.7 s with one server: median improvement 44.5 s (11.5%), with non-overlapping ranges. A further final-source run passed all 1,873 tests in 337.6 s. Three CI integration Test steps on the code commit passed in 535 / 487 / 424 s (median 487); the final documentation-only head passed in 360 s on the same test code. The four #1000 steps were 605 / 606 / 390 / 596 s (median about 601). CI ranges overlap, so this supports a likely win without a precise CI saving estimate.

The proposed DemoSeedTests class split did not improve suite wall: B-only was 393.5 / 368.6 / 387.3 s and A+B was 352.7 / 347.5 / 350.9 s, versus A-only 344.1 / 343.8 / 346.7 s. The shared collection still finished last locally. The PR therefore ships Part A and leaves Part B out. #863 remains available for the other slow nonshared classes.

[mforce]

Demo seed measurement is in PR #1003 and 03-demo-seed-profile.md. The PR contains documentation only; all profiling code was removed.

Three isolated fresh seeds took 16.36–16.54 s inside SeedAsync. The two daily-entry loops took 15.32–15.50 s. Each seed made 981 SaveChanges calls and 4,389 EF commands containing 11,784 SQL statements. It wrote 977 audit rows. A temporary no-audit counterfactual saved a median 1.13 s locally (6.8%); audit output is required by #500.

EF ran DetectChanges 1,962 times per seed, twice per save. Those passes took 10.30–10.51 s in isolation. Under full-suite contention, twelve overlapping seeds across three runs took 121.0–174.2 s each; DetectChanges accounted for 103.6–148.6 s (85–89%) while EF command execution accounted for 7.8–14.9 s (6–9%). All three full runs passed 1,873 tests, with suite walls of 352.6, 342.7, and 343.0 s. Warm demo factory database creation took 0.06–0.11 s and host migration checks 0.98–1.76 s; repeated boot is not the class floor.

Recommendation: there is substantial change-detection work to investigate, with the same demo rows and audits preserved. Removing one of two detection passes projects roughly 5 s per isolated seed, but no CI wall saving has been measured. The seed --profile demo verb is user-facing and fail-loud, so a later code change needs correctness checks and repeated CI wall measurements. #863 remains open for that decision.

[mforce]

Tested one hypothesis from 03-demo-seed-profile.md: since the demo seed is CPU-bound on EF change detection, running the five full-seed classes (DemoSeedTests, DemoSeedBootTests, DemoSeedDisabledOwnerTests, DemoSeedAttributionTests, SeedCommandTests) one after another, instead of letting xUnit's default parallelism overlap them, should reduce the CPU contention they cause each other while every other class keeps running in parallel.

It wins, but the margin is smaller than a naive "four sequential 16-second seeds beat four overlapping 150-second ones" estimate. Three load-verified clean runs at each of three configurations, all on the same 12-core host:

• Baseline: 342.0, 343.9, 345.0, 345.6 s (median 344.5).
• Four classes serialized (SeedCommandTests left parallel): 322.3, 325.5, 326.2 s (median 325.5, saves 19.0 s / 5.5%).
• All five classes serialized: 317.6, 320.7, 321.0 s (median 320.7, saves 23.8 s / 6.9% total).

The clean ranges do not overlap between configurations. Serialized, the four in-process seeds ran 17-31 s each (close to the 16.4 s isolated figure from the profile), down from 89-147 s under overlap in the same run. SeedCommandTests itself did not shrink (~120 s in both configurations), since its cost is mostly three spawned seed --profile demo subprocesses rather than in-process EF work, but removing it from the contention pool still bought a further 4.8 s.

Correctness: DemoSeedCollection is a bare [CollectionDefinition] with no ICollectionFixture, so it only reorders these five classes against each other. Each keeps its own factory subclass and its own migrated Postgres database, exactly as required after #1000's false-pass repair — verified by grep and by reading each factory's declaration, not assumed.

A note on method: an unrelated local agent loaded this host intermittently during part of the measurement session. Runs taken during that window are recorded and excluded from the comparison above; every reported run above was taken with that agent confirmed idle via its own status API, with load average recorded at the start of each run. Full detail, including the excluded contaminated runs and per-class breakdowns, is in docs/plans/863-integration-collection-split/04-seed-serialization.md on #1003.

Shipped as a perf(test) change on #1003, alongside the profile doc that motivated it. CI update: two CI runs of the new test code passed 1,873 tests with VSTest times of 361.9 s and 505.7 s, a 143.8 s spread on identical code. That brackets rather than beats the three post-#1002 CI baselines in 02-shared-postgres-spike.md (385.7, 442.4, 483.3 s), so CI evidence is inconclusive on its own, not corroborating as I first said from the single early run. CI on this repo is documented to swing by 100+ seconds run to run; two runs cannot separate a real CI-side change from that noise. The local result stands regardless, since it comes from load-verified clean runs on a controlled host, not CI. Refs #863, a broader issue about the integration suite's wall-clock floor; this comment does not mean that issue is resolved.

[mforce]

Tested the width-two split this issue's 2026-10-01 amendment re-scoped in after #1002 and #1003 removed the two objections that killed the original attempt. Short answer: the local win is real, but CI shows no measurable benefit, so the split is reverted.

The change. IntegrationCollection (94 classes, one shared factory and database) split into IntegrationCollectionA/IntegrationCollectionB, each with its own factory and database, so the two halves run concurrently with each other. Classes assigned by measured test time (152.7 s / 152.7 s) via greedy longest-processing-time packing, moving whole classes only; the three default-account classes and the two global-purge classes each stayed together as required.

Correctness, confirmed rather than assumed. Both halves pass 100% completely alone, in a fresh database, with no other class in the test binary: 487/487 and 548/548, exact matches to the expected counts. A vacuous-pass audit (the #1000 failure shape: a check that now passes because the data it implicitly expected moved to the other database) covered every Assert.All/Assert.Empty/.CountAsync()/IgnoreQueryFilters() call site across all 94 classes; every one is scoped to an ID, slug, or account the test itself created, so none could pass by matching nothing.

Local: a clean win. Three interleaved clean baseline runs (312.6-331.6 s) versus three clean split runs (273.2-277.7 s): non-overlapping ranges, a median saving of 47.3 seconds (14.6%). Every run was paired with a build fingerprint (SHA-256 plus the embedded collection-name strings) proving which code actually ran, after an earlier bug in this same measurement series showed how easy it is to silently measure the wrong build.

CI: no measurable win. Ten samples per side, re-running only the Tests (integration) job (never the whole workflow or workflow_dispatch), one side on this PR and the other on a throwaway draft PR from an empty commit on main. Split: 405-586 s (median 518.0, mean 513.1). Baseline: 433-571 s (median 545.5, mean 515.5). The mean difference is 2.4 seconds against a 27-second standard error, about 0.09 standard errors from zero: indistinguishable from no effect. Both sides show a bimodal pattern (split especially: five runs cluster 405-483 s, five cluster 553-586 s, nothing between), consistent with GitHub-hosted runner variance dominating over the code change on both sides equally.

Why the local win doesn't show up on CI, most likely: the local measurement ran on a 12-core host, where splitting one 305-second serial block into two ~153-second halves genuinely lets them run on separate cores at the same time. A standard ubuntu-26.04 GitHub-hosted runner has far fewer cores (2-4, not 12). With that few already saturated by the rest of the suite's existing parallelism, two collections competing for the same scarce cores may gain little from no longer being one serial block, while still paying for a second factory boot and a second database clone. This is consistent with the data, not independently confirmed; the runner's actual core count and a core-constrained local run would confirm it.

The split code is reverted. The method, both halves' class lists, the correctness audit, and the full local and CI measurement stay recorded in docs/plans/863-integration-collection-split/05-collection-split.md on #1004, which is now a documentation-only PR. The owner's original skepticism about a local-only win is the right call on this evidence: #1002 (~11.5% local) and #1003 (~6.9% local) both shipped without CI confirmation and may deserve the same ten-sample CI check this spike used, if that hasn't already happened.

[mforce]

Closing #863 — outcome and where the record lives

Shipped

PR	What	Local effect
#1000	Repaired a DemoSeedTests false pass — it asserted only IsSuccess, so it passed in 0.25 s without seeding (18.55 s of real work alone).	Correctness fix, no speed change
#1002	One Postgres server; each fixture clones its own database from a migrated template. 128 containers per run became 1.	−44.5 s, −11.5%
#1003	The five full-demo-seed classes serialized in a demo-seed collection with no shared fixture, so their CPU-bound seeds stop contending.	−23.8 s, −6.9%

Net local suite: ~389 s → ~320 s, about 18% faster.

Measured and dropped

• Merging the demo-seed tests — cuts total work but not wall clock; the floor is set by the tallest single class.
• Splitting DemoSeedTests alone — no saving, and slower combined with perf(test): reuse one Postgres server for integration databases #1002.
• Splitting the shared integration collection (docs(ci): measure the integration collection split (no CI win; reverted) #1004) — −47.3 s / −14.6% locally, non-overlapping across three clean runs per side, but no CI effect: ten CI samples per side, mean difference 2.4 s against a 27 s standard error. Reverted; docs(ci): measure the integration collection split (no CI win; reverted) #1004 records it.

Why this closes the issue

• On a 12-core dev host the suite is bound by its longest serial lane. About 1,200 s of test work could finish in ~100 s across 12 cores, yet takes ~320 s, so rearranging lanes helps.
• On CI it is bound by work on too few cores. GitHub's standard public-repo runner has 4. The same ~1,200 s needs at least ~300 s across 4 cores before any lane matters. Rearranging lanes barely registers there; only doing less work helps CI.

So rearranging is exhausted. #1003 and the collection split help developer machines, not CI. #1002 is the only change here that removed real work and is the one most likely to show on CI.

CI variance — read before claiming any CI timing result

CI integration time swings ~140 s on identical code. #1004's split samples were bimodal (five at 405–483 s, five at 553–586 s), consistent with GitHub assigning two machine classes. A single CI run, or two, cannot confirm a change smaller than that; use ~10 samples per side.

The remaining lever — not filed, pending owner approval

Reduce work in the save path. Per 03-demo-seed-profile.md, EF DetectChanges is ~63% of an isolated 16.4 s demo seed: TenantStampInterceptor.StampTenant calls ChangeTracker.Entries() on every save, which triggers a full detection pass, and EF runs another inside SaveChanges. The seeder's tracked graph grows to 7,897 entities and save time doubles across the run. That is production code: the interceptor runs on every write in the app, and seed --profile demo is a user-facing verb under #280. It is a different risk class from this issue, so it belongs in its own.

Documentation — docs/plans/863-integration-collection-split/

• 01-measurement.md — the per-class sharing map and the §D6 correctness boundary any future split must respect.
• 02-shared-postgres-spike.md — perf(test): reuse one Postgres server for integration databases #1002.
• 03-demo-seed-profile.md — where demo-seed time goes.
• 04-seed-serialization.md — perf(test): serialize the five full-seed integration classes #1003.
• 05-collection-split.md — the collection split; lands with docs(ci): measure the integration collection split (no CI win; reverted) #1004.

[mforce]

Update: the remaining lever named in the closing comment above is now filed, with owner approval, as two issues:

• perf(seed): release tracked entities between days in DemoDataSeeder #1007 — release tracked entities between days in DemoDataSeeder (seeder-only, low risk; do first).
• perf(tenancy): avoid the extra change-detection pass in TenantStampInterceptor #1008 — avoid the extra change-detection pass in TenantStampInterceptor (touches the tenant isolation guard; may become unnecessary after perf(seed): release tracked entities between days in DemoDataSeeder #1007).