test(frontend): two flaky tests are producing red checks unrelated to the change under test #14613

Activity

1. mrveiss

   added

   bugSomething isn't workingSomething isn't working

   on Aug 18, 2026
2. github-actions

   added

   frontend

   on Aug 18, 2026
3. mrveiss commented on Aug 18, 2026

   

   mrveiss

   on Aug 18, 2026

   OwnerAuthor

   More actions

   Third instance, and it narrows the hypothesis considerably.

   OrgChart.people.test.ts > Team grouping is honest about the data that exists (#13938) > groups the user under the team that claims them failed in a combined run of src/components/workflow src/components/llc src/views/llc src/composables/llc (46 files / 466 tests), passed alone (29/29), and passed on an immediate re-run of the identical combined command.

   That is now two different tests in the same file — the earlier instance was says the people could not be loaded, not that the company has none — both failing only in a larger run and both passing in isolation.

   Same file, different tests, order-dependent: this points at state shared across files that OrgChart.people.test.ts reads, not at a slow mount or a bad assertion in either test. Likely candidates, in the order I would check them:

   1. A module-level ref or cache in an llc composable that another test file populates first (useCompanyPeople, useLlcCompanyContext, and the org-chart composables all hold module-scope state).
   2. A vi.mock factory whose returned object is shared rather than recreated per file.
   3. A timer or pending promise from a neighbouring file resolving into this one's mount.

   Reproduction for whoever picks this up: run the four paths above together; it does not reproduce every time, so run it several times or use --sequence.shuffle to force ordering variation.

   The ChatInterface timeout in the original report may well be a separate cause; these two share a file and a shape, that one does not.
4. mrveiss commented on Aug 18, 2026

   

   mrveiss

   on Aug 18, 2026

   OwnerAuthor

   More actions

   Fourth instance, and a workaround that names the cause

   OrgChart.roleLens.test.ts > never fetches or posts as a result of a lens selection failed in a combined run of src/components/workflow src/views/llc src/composables/llc src/components/llc (53 files / 534 tests), passed alone (9/9), and passed on an immediate re-run of the identical combined command (534/534).

   That is now four different tests across three files — ChatInterface, OrgChart.people (twice), and now OrgChart.roleLens — all failing only under parallel file execution and all passing in isolation.

   The diagnostic that matters: --no-file-parallelism makes the whole set stable. An independent run of the same suites hit the same intermittency and needed the same flag, naming CostDashboard.contract.test.ts, OrgChart.people.test.ts and ProjectBrowserView.findings.test.ts as its casualties.

   Different files, different tests, same trigger. This is not four separate bugs — it is one piece of state shared across worker boundaries, and file parallelism decides who loses. Candidates, narrowing:

   1. Module-scope state in an llc composable that survives between files in the same worker (useCompanyPeople, useLlcCompanyContext, and the org-chart composables all hold module-level refs).
   2. A vi.mock factory returning a shared object rather than a per-file one.
   3. A timer or unresolved promise from one file settling inside another's mount.

   The lens test is a useful probe: it asserts "no fetch happens", so it fails when someone else's pending fetch resolves into its window. That points at (3) more strongly than (1).

   Reproduction: run those four paths together repeatedly; it reproduces roughly one run in three. --no-file-parallelism suppresses it.

   Do not "fix" this by adding --no-file-parallelism to the CI command. That hides a real leak and slows every run; the leak will still be there for anyone running a subset locally.
5. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Root cause, both halves — PR #14874

   1. ChatInterface.test.ts > renders the main chat interface

   A genuinely slow mount, not an unresolved promise. The file proves it
   itself: every wait in it goes through waitFor(), which carries its own 1s
   budget and fails with "unable to find element", not with a timeout. So a
   10000ms failure has always meant the render was still running when the budget
   expired — there is no promise left hanging to fix.

   All 23 tests in the file render the full chat tree (sidebar, session list,
   message list, composer, settings) with a Pinia store and a router. Two costs
   were removed and one budget was set:

   • renderComponent() in the shared test helper built a fresh createI18n
     instance on every render. Now one per module (test(frontend): mount helpers rebuild a ~400KB i18n instance on every mount #14860 is the same defect,
     suite-wide).
   • The file gets vi.setConfig({ testTimeout: 20_000 }) — per file, because
     frontend Unit & Integration Tests times out on a different test each run — 10s ceiling against 3x environment overhead #14842 recorded a different test failing each run, so naming one would be
     pretending we know which. The global testTimeout stays at 10000 so a
     genuinely hung test elsewhere still fails fast.

   2. OrgChart.people.test.ts — passes alone, fails in the larger run

   It is not another file interfering. It cannot be. Vitest resolves
   pool: "forks" with isolate: true (both defaults, neither overridden in
   vitest.config.ts), and an isolated task never shares a runner — the pool
   throws "Isolated tasks should not share runners" on that path. Every test file
   gets its own worker process and its own module registry, so a leaked mock or
   shared module state from a sibling file has no route in. Bisecting the file set
   would have found nothing.

   The accumulation is inside the file, and it is real:

   • OrgChart.people.test.ts mounts the full OrgChart tree 31 times and
     unmounts none of them.
   • The global beforeEach in the vitest setup file resets
     document.body.innerHTML. That detaches each tree's DOM but never unmounts
     the app — component instances, watchers and effects stay live for the rest of
     the file.
   • So the last tests in the file run with ~30 live trees still reacting to every
     shared ref they touch. The file's own beforeEach sets
     i18n.global.locale.value = 'en' on a shared instance, which every one of
     those live trees re-renders against.

   The cost therefore grows as the file progresses, and whether a late test crosses
   10s depends on how contended the runner is — which is exactly "passes alone,
   fails in a 367-test run, passes again on a re-run".

   This was already diagnosed once, for a single file:
   OrgChart.canvasFilters.test.ts carries a private teardown whose comment
   measures the same accumulation at ~3.4s quiet vs. a timeout under load, and
   notes that "no global enableAutoUnmount is configured". It was the only file
   that had the teardown; every heavy file had the problem.

   Fix: enableAutoUnmount(afterEach) in the global setup, so every mounted
   tree is destroyed after the test that made it. The private copy in
   OrgChart.canvasFilters.test.ts is removed in favour of it.

   Reproductions

   cd autobot-frontend
   npx vitest run src/components/__tests__/ChatInterface.test.ts
   npx vitest run src/composables/llc src/components/llc src/views/llc
   

   Stated honestly: the second is a contention reproduction, not a
   deterministic one — it did not fail on every run when it was reported either.
   The session that produced #14874 was not permitted to execute the suite, so the
   verification behind that PR is CI on the branch, not a local before/after.
6. mrveiss

   added a commit that references this issue on Aug 23, 2026

   fix(frontend): stop rebuilding i18n and leaking mounted trees in the …

   ...

   d70a1f9
7. mrveiss commented on Aug 23, 2026

   

   mrveiss

   on Aug 23, 2026

   OwnerAuthor

   More actions

   Delivered and merged.

   • PR: fix(frontend): stop rebuilding i18n and leaking mounted trees in the test suite (#14860, #14842, #14854, #14613) #14874 — squash-merged as d70a1f99e
   • CI: green on the exact head sha 603a17b95 — 53 check-runs, 42 success / 11 skipped / 0 failure. Verified against the sha directly rather than the rollup, because auto-update-pr-branches pushed a base merge mid-run and GITHUB_TOKEN pushes are known to leave a refreshed head with no checks (ci: auto-update-pr-branches leaves refreshed PRs with ZERO checks — GITHUB_TOKEN pushes do not trigger workflows #12823). The trap did not bite here: the checks are genuinely on the merged head.
   • Review: independent code-reviewer pass. No blocking issues. It traced all 37 files for i18n mutation APIs, confirmed pool: "forks" + isolate: true are unmodified defaults, and verified the removed private teardown was an exact functional subset of the new global one.

   Verified in base:

   origin/Dev_new_gui:autobot-frontend/src/test/utils/i18n-cache.ts        PRESENT
   origin/Dev_new_gui:autobot-frontend/src/test/vitest-setup.ts:25         enableAutoUnmount(afterEach)
   

   Two real causes, neither of which was the timeout:

   1. 32 test files rebuilt a fresh createI18n carrying the ~400KB en bundle on every mount. 20 hoisted to module scope, 11 memoized per locale, 1 deliberately left per-call because two RTL cases mutate locale.value.
   2. Nothing was ever unmounted — the global beforeEach reset document.body.innerHTML, which detaches the DOM but never unmounts the app, so watchers and effects stayed live for the rest of each file. Now enableAutoUnmount(afterEach) suite-wide.

   Only one file-scope timeout was raised (ChatInterface.test.ts, 20s). vitest.config.ts stays at 10000.

   Evidence: four consecutive runs of Unit & Integration Tests, all 259 files green, two of them the same commit re-run. On the slowest run the slowest unbudgeted test was 5500ms against the 10s ceiling; rtl.spec.ts went from a case timing out at 10043ms to an 18-test file in 3.2-4.8s.

   Honest caveat: no speedup is claimed from run totals. Files touched moved 1.67x/0.78x vs base and files untouched moved 1.70x/0.76x — identical, so the contended singleton runner dominates and these runs cannot separate the change from load. What they show is no regression; the removed work is provable by reading.
8. mrveiss

   closed this as completedon Aug 23, 2026