test(e2e): retry a crashed Windows spec in a fresh VS Code session

[rugpanov]

Why

On the slow Windows e2e shards, a VS Code window reload — e.g. opening a freshly initialized bundle in bundle_init.e2e.ts — can drop the wdio↔VS Code websocket with Connection closed. Code: 1006. That kills the whole session, so every remaining test in the spec cascades into target window already closed / web view not found. It was the failure on PR #2130's IT run and recurs across nightlies as a bystander flake.

This whole-session crash is unrecoverable from inside the test: no added wait or try/catch helps once the session is dead (the earlier per-waitUntil hardening in #2056/#2123 addresses a different class — transient element races within a live session). The only recovery is a fresh session, which is exactly what a spec-file retry starts.

wdio's specFileRetries was 0, so this class of infra flake was an instant hard failure with no recovery.

What

• Set specFileRetries to re-run a whole spec once, Windows-only, via a small specFileRetriesForPlatform(platform) helper in src/test/retry.ts.
• Linux/macOS stay at 0 so a real regression there fails fast instead of being masked, and CI time isn't doubled on the stable platforms.
• Each CI shard already runs a single spec file, so a retry re-runs that spec in a fresh VS Code session within the same job and reports green if it recovers.

Tradeoff

Blanket spec retry can mask a reproducible-once real failure. Scoping to Windows + a single retry keeps that risk small while covering the observed crash class.

Verification

• yarn test:unit — 845 passing, 0 failing (includes the new helper tests).
• eslint + prettier -c clean on the changed files.

This pull request and its description were written by Isaac.

[rugpanov]

🤖 Integration tests ✅ all 41 test jobs passed for 2678f61d.
View run

[rugpanov]

🤖 Integration tests ✅ all 41 test jobs passed for 728c397e.
View run

[rugpanov]

🤖 Integration tests triggered for a86ad79d — ⏳ running.
View run

[rugpanov]

🤖 Integration tests triggered for 38b194f1 — ⏳ running.
View run

[anton-107]

Approving — the diagnosis holds up. A WS-1006 session death really is unrecoverable in-test, so this is a different class from the per-waitUntil hardening in #2056/#2123, and Windows-only + a single retry + surfacing the recovery instead of swallowing it is the right shape. SpecRetryTracker keying on the spec string is correct: _endHandler re-pushes the same specs array on requeue, so the key is stable across attempts.

One thing worth a follow-up, because it partly defeats the tracker's own purpose.

The retry truncates the crashed attempt's logs

wdio deliberately reuses the failing runner's cid for a retry — @wdio/cli/build/launcher.js:329-331, "Retried tests receive the cid of the failing test as rid". Every per-worker artifact path is therefore byte-identical across attempts, and all of them are opened truncating:

• WDIO_LOG_PATH = logs/wdio-<cid>.log (@wdio/local-runner/build/worker.js:91), opened as fs.createWriteStream(path) with default flags: 'w' (@wdio/logger/build/node.js:63)
• logs/wdio-<worker-id>-chromedriver.log — explicit {flags: 'w'} (@wdio/utils/build/node/startWebDriver.js:126-129)
• afterSession (wdio.conf.ts:606) copies extension logs into logs/vscode-logs-<spec-basename>/<basename>.json; both name parts are deterministic (LoggerManager.getLogFile → ${prefix}-logs.json), so cpSync overwrites

So on exactly the outcome this PR exists to produce — attempt 1 crashes, attempt 2 goes green — the wdio log holding the Connection closed. Code: 1006 and the extension logs from the crashed session are gone by the time the run finishes. The ::warning:: says bundle_init.e2e.ts flaked, but the artifact no longer contains the evidence to act on it. Videos survive (timestamped filenames, and saveAllVideos: false keeps the failed one), so it isn't a total loss — but the logs are the part you'd actually want.

Quick fix

Cheap, and onWorkerEnd is already the right place — note the hook currently drops the 4th param, which is precisely the signal needed (retries > 0 means wdio is about to requeue this spec and reopen those paths):

onWorkerEnd: function (cid, exitCode, specs, retries) {
    for (const spec of specs) {
        specRetryTracker.record(spec, exitCode === 0);
    }
    // A retry reuses this cid, so wdio reopens logs/wdio-<cid>.log with
    // flags:'w' and truncates the very crash we retried for. Park it.
    if (exitCode !== 0 && retries > 0) {
        const from = path.join("logs", `wdio-${cid}.log`);
        const to = path.join("logs", `wdio-${cid}-failed-attempt.log`);
        try {
            renameSync(from, to);
        } catch (e) {
            console.error(`Could not preserve ${from}:`, e);
        }
    }
},

Same treatment for wdio-<cid>-chromedriver.log. For the extension logs, folding an attempt marker into the vscode-logs-* directory name in afterSession covers it — a module-level Map<spec, attempt> incremented in onWorkerEnd won't reach the worker (different process), so the simplest version is a suffix derived from something already per-attempt, or just letting the parked wdio log carry the diagnosis.

Smaller notes, non-blocking

• specFileRetriesDelay: 0 stays at zero on the one platform where that's riskiest. The retry relaunches VS Code immediately after a session died mid-window-reload, while the old Electron and chromedriver are still exiting, and its first act (beforeSession) is code --install-extension --force against the shared EXTENSIONS_DIR — the same Windows file-lock territory isTransientFileLockError in this very file exists to paper over. ~5–10s via the same per-platform helper costs nothing against a multi-minute spec.
• The ::warning:: probably doesn't reach the PR author. integration-tests.yml dispatches the isolated workflow in a separate repo and reports back only through an API-created check run, so the annotation attaches to that run rather than the "Integration Tests" check on the PR. GITHUB_STEP_SUMMARY, or feeding the check-run output, would land where people look — or at minimum a comment noting where the annotation actually surfaces.
• Shard timeout headroom. A retry roughly doubles the shard's wall clock, and afterSuite compounds it: it runs bundle destroy over every subfolder of WORKSPACE_PATH, and onWorkerStart mints a fresh testRoot per attempt, so the retry's cleanup also destroys the orphaned one. Worth confirming the Windows shard has ~2x headroom — otherwise this turns a clean failure into a timeout with no report, which is strictly worse.
• Convention nit: SpecRetryTracker is a class in a camelCase file. CODE_CONVENTIONS §1 wants PascalCase.ts named for the class export, and the retry.ts header rationale (live outside e2e/ so the tsconfig exclude doesn't hide the unit test) applies just as well to a sibling src/test/SpecRetryTracker.ts.
• formatRecoveredSpecsReport prints whatever onWorkerEnd passes, which is a file:///… absolute URL (FileSystemPathService.ensureAbsolutePath returns pathToFileURL(p).href). Basenames read better — afterSession already does that split.

[rugpanov]

🤖 Integration tests triggered for 370488e1 — ⏳ running.
View run

[rugpanov]

🤖 Integration tests ✅ all 41 test jobs passed for 86209f63.
View run

[rugpanov]

Thanks @anton-107 for the detailed review — every point verified against the wdio 9.29 source. Here's what was addressed:

Addressed

• Retry truncates the crashed attempt's logs (main follow-up) — onWorkerEnd now parks the worker's logs to -failed-attempt names before the retry (which reuses the cid) reopens them with flags:"w", gated on exitCode !== 0 && retries > 0. Verified retries is the pre-decrement value, so this fires on exactly the attempt that will be requeued and never on the final failure. (370488e1)
  • A follow-up review pass caught that the runner log is named ${specBaseName}-<cid>.log, not wdio-<cid>.log (@wdio/local-runner build/index.js:261-266) — the first version would have ENOENT'd and preserved nothing. Fixed to derive the name from the spec basename (handles file:// URLs too). (86209f63)
• specFileRetriesDelay: 0 on Windows — added a Windows-only 5s delay (specFileRetriesDelayForPlatform) so the retry doesn't relaunch VS Code / --install-extension into the still-exiting Electron + chromedriver (the file-lock territory isTransientFileLockError guards). Applies only to retries, never the happy path. (370488e1)
• SpecRetryTracker class in a camelCase file (CODE_CONVENTIONS §1) — moved to its own src/test/SpecRetryTracker.ts with a co-located test. (370488e1)
• Report printed file:// URLs — formatRecoveredSpecsReport now shows spec basenames. (370488e1)

Deferred (tracked as a separate follow-up)

• ::warning:: doesn't reach the PR (Note 2) — confirmed: e2e is dispatched into the isolated workflow, which reports back only a check run, so the annotation attaches to that run rather than this PR. Surfacing recovered flakes here means feeding them into the check-run's output.summary, which is a workflow-side change out of scope for this PR. Tracking it separately.

Noted, not changed

• Shard timeout headroom (Note 3) — a retry ~doubles a failed shard's wall clock (plus the 5s delay). Leaving the shard timeout as-is for now; worth watching if retries become common.
• Chromedriver log name — wdio-<cid>-chromedriver.log matches @wdio/utils; if the driver identifier ever differs, that rename simply no-ops (try/caught) and the primary runner log carrying the 1006 is preserved regardless.

Unit tests: 861 passing; eslint/prettier/e2e-typecheck clean.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2132
• Commit SHA: 500e342e939ea3d347e99809604ee58e7adf2c67

Checks will be approved automatically on success.