CI reports green on tests that cannot fail

[TortoiseWolfe]

The last E2E run before this repo went dormant (3cca5af, 2026-04-20) was 18/18 green. That number does not mean what it looks like.

Six independent reasons, all verified against the source at 3cca5af. Two are fixed in this issue's companion commit; four are open.

---

1. Half the messaging E2E suite has never run in CI — ✅ FIXED

.github/workflows/e2e.yml — each browser's matrix listed:

- { group: msg-, shard: 1/4 }
- { group: msg-, shard: 2/4 }

Playwright's --shard=1/4 and --shard=2/4 run the first two quarters of the matched tests. Shards 3/4 and 4/4 were never dispatched as jobs, in any of the three browsers. Roughly half of tests/e2e/messaging/** and tests/e2e/employer/** has never executed in CI — and it was counted as passing.

Fixed by adding msg- 3/4 and msg- 4/4 to all three matrices. The run goes from 21 jobs to 27.

This was safe to switch on: tests/e2e/global-setup.ts:274-331 self-provisions e2e-s{N}-* users for whatever E2E_SHARD_INDEX it is handed, and the E2E_ENCRYPTION_KEYS secret covers shards 1-6, so 3 and 4 get pre-baked keys rather than falling back to the 60-180s Argon2id path.

Expect new failures. They are not regressions — they are the first results this code has ever produced.

2. Three assertions in the only a11y spec CI runs could never fail — ✅ FIXED

pnpm test:a11y:ci (package.json:66) runs exactly one file: tests/e2e/tests/accessibility.spec.ts.

line	assertion	why it could not fail
:71	expect(alt).toBeDefined()	getAttribute('alt') returns null when the attribute is absent. null is defined. An <img> with no alt at all passed.
:119	expect(outline).toBeTruthy() where outline = styles.outline || styles.border	computed style is always a non-empty string — an unfocused element reports "rgb(0, 0, 0) none 0px". Always truthy. Also wrapped in if (elementCount > 0).
:86	label check	every assertion sat inside if (inputId). An unlabelled input with no id — the most likely real defect — was skipped entirely.

Fixed: assert .not.toBeNull() on the alt attribute; drop the if (inputId) gate and accept label[for] / aria-label / aria-labelledby / ancestor <label>; assert a real focus affordance (outlineStyle !== 'none' && outlineWidth > 0, or a box-shadow ring) instead of a string that is always truthy.

Verified by mutation: with the repaired assertions, an injected alt-less <img>, an id-less unlabelled <input>, and an element with outline:none; box-shadow:none each produce a failure. Before the fix, all three passed.

---

3. Product failures are converted into skips — ❌ OPEN

tests/e2e/messaging/complete-user-workflow.spec.ts:

:421  test.skip(true, 'Friend request did not persist to DB — send failed silently');
:497  test.skip(true, 'Connection exists in DB but ConnectionManager UI did not render it');

Both are real product bugs reported as green. :421 is literally the "silent loss" symptom class #69 was opened for. It was introduced deliberately in c60d78a to get the pipeline green.

A skip is the right tool for "this environment can't run this test". It is the wrong tool for "the product did the wrong thing" — that is a failure, and it should be loud. These should become failures again, with the underlying bugs tracked.

4. The map/routing subsystem has effectively no CI E2E coverage — ❌ OPEN

tests/e2e/map.spec.ts has 8 hard-disabled tests: test.skip('...') at lines 114, 150, 182, 212, 221, 389, 572, 591 — geolocation consent, permission denial, consent memory, markers, popups, offline tiles, accuracy circle, rapid location updates.

That is essentially the whole map interaction surface, permanently off. Additionally, whole-file !!process.env.CI gates gate out tests/e2e/routes/* (route-features, route-sidebar-ux, route-home-connection, active-route-visual, route-visual-verification, start-end-markers), so they run locally and never in CI.

MapLibre + OSRM routing is the product's differentiator. It is the least-tested thing in the pipeline.

5. Pa11y is referenced everywhere and installed nowhere — ❌ OPEN

grep -c pa11y package.json → 0. There is no pa11y or pa11y-ci dependency and no script that invokes one.

Still committed and still cited:

• config/pa11yci.json and config/pa11yci-auth.json — orphaned config, read by nothing
• README.md claimed "Pa11y and axe-core for accessibility" (corrected in the companion commit)
• Accessibility suite failures + mobile orientation bug #70's brief is written around running "the accessibility tests" as if Pa11y were the runner

Decide: adopt pa11y-ci properly, or delete the two config files. Right now the repo describes a tool it does not have.

6. The orientation gate tolerates 100px, and CSS hides what it measures — ❌ OPEN

tests/e2e/tests/mobile-orientation.spec.ts:142-151 — on overflow it console.warns, then asserts:

expect(scrollWidth).toBeLessThanOrEqual(844 + 100);

Up to 100px of horizontal overflow passes silently. And src/app/globals.css:9 sets html, body { overflow-x: hidden; max-width: 100vw; }, which suppresses the very overflow the test is trying to detect. The gate cannot see the bug and would tolerate it anyway.

Related: #70's "orientation bug" is real, but it was already loosened into passing, which is why it never showed up.

---

Why this matters more than any single bug here

Every remediation decision made while this repo was dormant — "the pipeline is green", "18/18", #66's reconciliation, closing out the E2E stabilisation arc — rested on a signal that was partly measuring nothing. Before trusting a green run again, items 3-6 need to be either fixed or explicitly acknowledged as uncovered.

A useful rule for this repo going forward: a test that has never failed is not evidence. When adding a gate, prove it can fail (mutate the thing it checks and watch it go red) before believing it.

Checklist

• Add msg- 3/4 and msg- 4/4 to all three browser matrices
• Repair the three no-op assertions in accessibility.spec.ts
• Convert the two test.skip(true, ...) product-failure skips back into failures; file the underlying bugs
• Decide the fate of the 8 disabled map.spec.ts tests and the CI-gated routes/* specs
• Adopt pa11y-ci or delete config/pa11yci*.json; correct Accessibility suite failures + mobile orientation bug #70's premise
• Tighten the orientation assertion and remove the overflow-x: hidden that masks it
• Audit the remaining ~122 test.skip/fixme call sites in tests/ for others of the same shape