Skip to content

fix: stabilize Windows CI integration tests - #10940

Open
joehan wants to merge 25 commits into
mainfrom
fix-windows-ci-flakiness
Open

fix: stabilize Windows CI integration tests#10940
joehan wants to merge 25 commits into
mainfrom
fix-windows-ci-flakiness

Conversation

@joehan

@joehan joehan commented Aug 14, 2026

Copy link
Copy Markdown
Member

Description

Stabilizes Windows CI integration tests, addresses review feedback, and improves runner reliability.

Key Changes

  • Zip Slip Path Normalization & Protection (src/unzip.ts, src/unzip.spec.ts):
    • Enforce parentWithSep trailing separator check in isChildDir to prevent prefix-matching sibling directories (e.g. /tmp/app vs /tmp/app-secret) from bypassing directory traversal protection.
    • Case-normalize paths on win32 to prevent casing discrepancies between process.cwd() and archive entries from triggering false-positive Zip Slip errors.
    • Added comprehensive unit tests for isChildDir.
  • Process Lifecycle & Timeout Management (scripts/integration-helpers/cli.ts):
    • In CLIProcess.stop() on Windows, execute taskkill /pid ${p.pid} /T /F and await the process exit event with a 2-second timeout race fallback.
    • Capture and clear the timeout timer with clearTimeout when process exit resolves first.
  • Port Cleanup Safety (scripts/hosting-tests/run.sh, scripts/triggers-end-to-end-tests/run.sh):
    • Restrict socket matching to LISTENING state (lsof -sTCP:LISTEN on POSIX and $2 ~ port"$" && $4 == "LISTENING" on Windows) to prevent killing client test runner processes with active connections to emulator ports.
    • Scope preview channel IDs and target filenames to runner OS and random run ID (${GITHUB_RUN_NUMBER}-${RUNNER_OS}-${RANDOM}) to prevent cross-OS test collisions in matrix CI runs.
  • Disable MOTD in CI (src/fetchMOTD.ts):
    • Skip fetchMOTD() early when process.env.CI is set to avoid unnecessary network calls and potential timeouts during CI test execution.
  • CLI Commands in Integration Tests (scripts/hosting-tests/run.sh, scripts/storage-deploy-tests/run.sh):
    • Use standard firebase target:apply <service> <target-name> <resource> --project <project> commands instead of writing .firebaserc directly.
  • Runner Stability & Windows Defender (.github/workflows/node-test.yml):
    • Exclude workspace and npm directories from Windows Defender (Set-MpPreference -ExclusionPath @("${{ github.workspace }}", "$env:LocalAppData\npm-cache", "$env:AppData\npm")) with explanatory documentation.
    • Removed npm@9.5 downgrade step on Node 24 Windows runners to eliminate Arborist deadlock during npm ci.
    • Added --prefer-offline --no-audit across fixture dependency installations in test runners.
  • Client Integration Tests (scripts/client-integration-tests/tests.ts, run.sh):
    • Add explicit this: Mocha.Suite annotations.
    • Add this.timeout(15 * 1000) and this.retries(2) to apps:list and apps:sdkconfig suites.
    • Run apps.list variants in parallel using Promise.all.
    • Wrap .firebaserc deletion in deployHosting after hook with try/catch.
    • Increase Mocha timeout to 30000ms in scripts/client-integration-tests/run.sh.

CI Validation Runs

All 6 Windows integration test suites (client-integration, emulator, hosting, triggers-end-to-end:inspect, functions-discover, storage-deploy) and all 12 Linux suites have been validated with 100% GREEN (53/53 passing checks):

Run Commit Status Windows Integration Suites Duration
Latest Validation Run (Run 32197607736) b667bc42d 53/53 PASSED (100%) client-integration (2m43s), emulator (4m29s), hosting (7m11s), triggers:inspect (8m26s), functions-discover (6m45s), storage-deploy (6m38s)
Validation Run (Run 32084241549) fa6cdcd78 50/50 PASSED (100%) client-integration (3m50s), emulator (4m26s), hosting (7m9s), triggers:inspect (8m6s), functions-discover (8m21s), storage-deploy (11m54s)
Validation Run (Run 31884410611) b10a8181d 50/50 PASSED (100%) client-integration (2m39s), emulator (4m41s), hosting (6m55s), triggers:inspect (9m8s), functions-discover (7m24s), storage-deploy (6m53s)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several test stability improvements, including Windows-specific process teardown using taskkill, increased timeouts, parallelized test execution, and skipping MOTD fetching in CI environments. Feedback on these changes highlights a potential race condition on Windows where the process teardown promise is resolved too early, implicit any type violations on the this parameter in test suites, and a redundant process.env.CI check before calling fetchMOTD().

Comment thread scripts/integration-helpers/cli.ts
});

describe("apps:list", () => {
describe("apps:list", function (this) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In TypeScript, the special this parameter must have an explicit type annotation (e.g., this: Mocha.Suite). Leaving it as function (this) results in an implicit any type, which violates the repository style guide's rule against using any as an escape hatch.

describe("apps:list", function (this: Mocha.Suite) {
References
  1. Never use any or unknown as an escape hatch. Define proper interfaces/types or use type guards. (link)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Added explicit type annotation.

});

describe("apps:sdkconfig", () => {
describe("apps:sdkconfig", function (this) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In TypeScript, the special this parameter must have an explicit type annotation (e.g., this: Mocha.Suite). Leaving it as function (this) results in an implicit any type, which violates the repository style guide's rule against using any as an escape hatch.

describe("apps:sdkconfig", function (this: Mocha.Suite) {
References
  1. Never use any or unknown as an escape hatch. Define proper interfaces/types or use type guards. (link)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Added explicit type annotation.

Comment thread src/bin/cli.ts Outdated
Comment on lines +97 to +99
if (!process.env.CI) {
fetchMOTD();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This process.env.CI check is redundant because fetchMOTD() itself already performs the exact same check and returns early if process.env.CI is set. We can simplify this by calling fetchMOTD() directly.

  fetchMOTD();

@joehan

joehan commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

/joe-review

Comment thread src/bin/cli.ts Outdated
@joehan
joehan requested a review from ajperel August 14, 2026 23:29
joehan added 18 commits August 14, 2026 23:32
Comment thread scripts/hosting-tests/run.sh Outdated
Comment thread src/unzip.ts
Comment thread src/unzip.ts Outdated
Comment thread src/unzip.ts Outdated
Comment thread scripts/integration-helpers/cli.ts Outdated
p.once("exit", () => resolve());
});

const timeoutPromise = new Promise<void>((resolve) => setTimeout(resolve, 2000));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Uncleared timeout

Rationale: The setTimeout is not cleared if the process exits before the 2-second timeout, which can keep the event loop active.

Suggested Fix:

      let timeoutId: NodeJS.Timeout;
      const timeoutPromise = new Promise<void>((resolve) => {
        timeoutId = setTimeout(resolve, 2000);
      });

      try {
        execSync(`taskkill /pid ${p.pid} /T /F`);
      } catch {
        // ignore if process already exited
      }

      return Promise.race([exitPromise, timeoutPromise]).then(() => {
        clearTimeout(timeoutId);
        this.process = undefined;
      });

Comment thread .github/workflows/node-test.yml
Comment thread scripts/hosting-tests/run.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants