[Bug]: the Windows removal retry budget is 2.5 seconds and is exceeded under shard load #4789

Client or integration

Other

Area

Platform (Windows / macOS / Linux)

Summary

removeTreeWithRetry absorbs the Windows filesystem-release race that src/config/paths.ts:57 documents: hardenConfigDir() spawns icacls.exe, which holds the config directory open, and "anything that removes or renames that directory after a clean shutdown — a test fixture teardown, an uninstaller, a home move — gets EPERM/EBUSY unless the process that started the child also waits for it."

The retry budget is REMOVE_ATTEMPTS = 50 at REMOVE_RETRY_DELAY_MS = 50, so the helper gives that race 2.5 seconds total (tests/helpers/remove-tree.ts:5-6). Under six concurrent Windows shards that budget is not always enough, and the helper rethrows the EPERM it exists to absorb:

tests\helpers\remove-tree.ts:38
error: EPERM: operation not permitted, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\ocx-management-auth-<id>'
  at removeTreeWithRetry (tests/helpers/remove-tree.ts:38:83)
  at tests/server/server-management-auth.test.ts:206:17

The budget appears to be tuned for a lightly loaded machine. The same concern applies outside tests: an uninstall or a home move issued while the machine is busy can exceed 2.5 seconds of handle retention for the same reason, and that path has no shard to retry it.

This is a pre-existing condition rather than a regression. It surfaced during a stacked-lane CI run and is filed rather than fixed, because whether the right answer is a larger budget, exponential backoff, or waiting on the icacls child directly is a maintainer decision.

Reproduction

1. Run the Windows leg with six concurrent shards (lane=all dispatch of ci.yml).
2. Observe a shard failing in removeTreeWithRetry with EPERM on a mkdtemp directory that a server fixture used.
3. Re-run the failed shard; it passes, because the race is timing-dependent rather than deterministic.

Evidence that this is load-dependent and not caused by the code under test: two dispatch runs of the same lane, with byte-identical tests/helpers/remove-tree.ts, disagreed. Run 35050601423 completed tests/server/server-management-auth.test.ts successfully on Windows; run 35053542913 failed it with the EPERM above. The only difference between the two runs was the absorbed upstream dev commits and the resulting shard distribution.

Suggested directions, in the order they seem worth considering:

• Wait on the icacls child directly where the caller owns it, which is the mechanism flushConfigDirHardening already provides for server.stop, rather than polling blind.
• Replace the flat 50 x 50ms with a bounded exponential backoff, so a busy machine gets a longer tail without adding 2.5 seconds of latency to the common case.
• Keep the flat schedule but raise the ceiling on Windows only, where this race is the documented one.

Version

2.57.0

Operating system

Windows (GitHub Actions windows-latest shards)

Provider and model

Not provider-specific.

Logs or error output

Windows shard 5/6 failed on attempt 1 (exit 1); assertion failures are not retried.
tests\helpers\remove-tree.ts:38
error: EPERM: operation not permitted, rm 'C:\Users\RUNNER~1\AppData\Local\Temp\ocx-management-auth-<id>'
  at removeTreeWithRetry (tests/helpers/remove-tree.ts:38:83)
  at <anonymous> (tests/server/server-management-auth.test.ts:206:17)

Screenshots and supporting files

Not applicable; the annotation above is the full evidence.

Redacted configuration

Not applicable.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.