Inconsistent failures using Cypress Run

[sjaugmented]

I have a series of tests, all working and passing with cypress open and the UI. The problem is when I run the test suite using cypress run and running headless from the command line. I get failures on tests that consistently pass in the UI.

Extra aggravation is when I single out a failed test with npx cypress run --spec "failedTest.spec". Running solo tests, they all pass. It's only when I run the entire batch from cypress run that I get failures.

There doesn't seem to be any consistency to the failures either. With the exception of one test, most of the failures seem to be random, sometimes even happening in the beforeEach block, which is identical across the tests. Other times it says there are missing testid's, which - again - most of the tests share. I've also tried specifying a browser as well, as I thought Electron might be an issue, but it doesn't seem to make a difference.

So two questions, if anyone has any thoughts: why are tests passing in the UI but failing from the command line? And why are tests passing when run individually from the command line, but failing when running the whole suite?

[jennifer-shehane]

I would suggest following our recommendations on troubleshooting why tests are fail in CI but pass locally: https://on.cypress.io/using-cypress-faq#Why-do-my-Cypress-tests-pass-locally-but-not-in-CI

[danielSoellerAtBearP]

@jennifer-shehane What you are suggesting is really tedious and almost defeats all the pros for using cypress. It will come done to "ensure that every network request has finished". This is not feasible for big tests suites.
Run and Open should be as close to each other as Possible, or at least run should be more robust than open.
I don't care if run takes a bit longer if it is more robust out of the box, in exchange
It is really killing the productivity gains from using cypress.

[danielSoellerAtBearP]

For now, I assume run falling inconsistent is related to undiscovered memory leaks like ##4164