CI: Sporadic failures in example-firefox

[MikeMcC399]

Describe the bug

.github/workflows/example-firefox.yml often fails.

Typically the failure logs show:

Still waiting to connect to Firefox, retrying in 1 second
Warning: We failed processing this video.
Cypress failed to make a connection to Firefox.
Could not find Cypress test run results
Error: Could not find Cypress test run results

Possibly also

Error: connect ECONNREFUSED 127.0.0.1:38171
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)

Bug history

View workflow logs via https://github.com/cypress-io/github-action/actions/workflows/example-firefox.yml?query=branch%3Amaster

Job Id	Failure job	Firefox version
firefox	6185401336	107
firefox-v10	6056300428	105

The workflow has been failing on merges into the master branch since November 22, 2022. Problems with merges into other branches have been going on for much longer. Due to log retention policies only recent detailed logs going back 90 days are available. Job 5079650276 from September 26, 2022 is the earliest concrete evidence of the exact same failure with Firefox 104.

Steps to reproduce the issue

Open a pull request to trigger a run of .github/workflows/example-firefox.yml

View the results on https://github.com/cypress-io/github-action/actions/workflows/example-firefox.yml

Expected behavior

All sub-tests of .github/workflows/example-firefox.yml should run reliably and succeed.

Comments

I could not find a reason for the unreliable results of .github/workflows/example-firefox.yml so far.

It would be possible to stop the following errors being reported:

• "Warning: We failed processing this video." (disable videos)
• "Warning: No files were found with the provided path: examples/firefox/cypress/screenshots. No artifacts will be uploaded" in the firefox job for v9 (correct the directory)

however this does not make the action more reliable.

[MikeMcC399]

Related issues

Cypress v9

v9 (9.6.1) errors sometimes with the following. See 6185401336 for log.

Cypress failed to make a connection to Firefox.

This usually indicates there was a problem opening the Firefox browser.

Error: could not find CRI target
    at lazyAssLogic (/home/runner/.cache/Cypress/9.6.1/Cypress/resources/app/node_modules/lazy-ass/index.js:110:14)
    at lazyAss (/home/runner/.cache/Cypress/9.6.1/Cypress/resources/app/node_modules/lazy-ass/index.js:115:28)
    at findStartPage (/home/runner/.cache/Cypress/9.6.1/Cypress/resources/app/packages/server/lib/browsers/protocol.js:56:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

• This corresponds to issue "Still waiting to connect to Firefox" Firefox103 | Ubuntu 20.04 | Cypress 9.50.0 cypress#23215 which was closed due to inactivity.

The issue is reproducible locally with Cypress 9.6.1 under Ubuntu 20.04 with
npx cypress run --browser firefox
executed in examples/v9/firefox with Firefox 107.0.1 using Node.js 16.13.0.

The problem is still reproducible with Cypress 9.7.0 under Ubuntu 20.04
with Firefox 107.0.1 using Node.js 18.12.1. Moving to Ubuntu 22.04 does not help either: the problem is again still reproducible.

Edit: @nagash77 wrote in cypress-io/cypress#23215 (comment) "we currently do not backport fixes to older versions of Cypress." so there is no basis to believe that this issue would be reopened and resolved. Therefore the assumption has to be that the test will never work reliably.

Cypress v10

v10 (10.0.0) errors sometimes with the following. See 6056300428 for log.

Error: connect ECONNREFUSED 127.0.0.1:38171
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)

• This is very similar to the error message noted in the Cypress issue Cypress failed to make a connection to Firefox - ECONNREFUSED at TCPConnectWrap.afterConnect cypress#14272

I could not reproduce this issue locally with Cypress 10.0.0 under Ubuntu 20.04 with
npx cypress run --browser firefox
executed in examples/v10/firefox with Firefox 107.0.1 using Node.js 16.13.0.

[MikeMcC399]

Suggestion

Summary

Disable Firefox v9 test. Leave Firefox v10 test enabled.

Details

Since the Firefox issue cypress-io/cypress#23215 on Cypress v9 is reproducible on a local system, even with the latest Cypress 9.7.0 of the 9.x series together with the latest Ubuntu LTS (22.04) and Node.js 18 LTS, there is little hope that it can run reliably under GHA. I assume also that there will not be a Cypress 9.x fix found and released. So I suggest to disable the v9 example-firefox test, for instance by adding if: false into .github/workflows/example-firefox.yml to cause a skip.

I also looked at the possibility of using windows-2022 with v9 as a workaround, instead of running under Ubuntu. This performed even worse and just timed out after 3.5 minutes ("Error: The action has timed out.").

The Firefox test using Cypress 10.0.0 runs locally with no issue.

It seems that connection problems in the Firefox test under v9 may have been affecting the following Firefox test under v10. I was not able to get the v10 Firefox test to fail running under GitHub actions if the v9 test is disabled. It does however sometimes output "Still waiting to connect to Firefox, retrying in 1 second" up to 5 times. This is not improved by using Cypress 12.0.2 instead of Cypress 10.0.0.

[MikeMcC399]

• I submitted PR Skip Firefox test using Cypress v9 to resolve flaky tests #660 to disable the Firefox v9 test, which should resolve this issue.
• I will also submit a separate issue and follow up with a PR to fix the warning messages noticed when troubleshooting the failures with the Firefox test i.e. video and artifacts warnings. Edit: Done. This is now PR CI: Artifact and video warnings in example-firefox #661.

[MikeMcC399]

I got today's new release of Firefox 108 and tried on Windows 11, with Cypress 9.7.0 (without GHA).

If I start a separate instance of Firefox then execute
npx cypress run --browser firefox --headed
I get no problem, however if I then close all Firefox sessions and execute the above command again, I get
Error: could not find CRI target

This is an interoperability problem between Cypress 9.7.0 and Firefox 108. I can't say which side of the two connection partners is at fault, but it certainly isn't github-action, since I'm able to reproduce the issue without it.

The test script is:

/// <reference types="Cypress" />

it('works', () => {
  cy.visit('https://example.cypress.io')
})

[MikeMcC399]

• PR Skip Firefox test using Cypress v9 to resolve flaky tests #660 has resolved this issue through the workaround of skipping the problematic Firefox test with Cypress v9. Issue "Still waiting to connect to Firefox" Firefox103 | Ubuntu 20.04 | Cypress 9.50.0 cypress#23215 provides background information about why no other resolution is being pursued.

• The action log 3705934530 shows the successful result of skipping the Firefox v9 test, which allows the Firefox v10 test to be successful.

Thanks to @jaffrepaul and @nagash77 for supporting the resolution of this issue!