Tests hang when afterEach promise does not resolve with Retires enabled

[taylorjellsworth]

Current behavior

I recently noticed that our e2e tests periodically fail the first run and then randomly hang on the retry run. After outputting cypress logs via DEBUG='cypress:*' I found the following:

• The test:after:run event never gets fired when the test hang on the retry run, The test:after:run event is supposed to happen after the afterEach and after Mocha hooks run
• We get an onMocha hook output, which means that the afterEach hook has started
• We never receive an onMocha hook end output. This signifies the afterEach hook has never finished. So the afterEach hook is hanging for some reason
  

It appears to me that the tests are hanging when retries are enabled and the afterEach promise does not resolve

Desired behavior

During the first retry, Cypress should reach timeout and fail. The test should not hang.

Test code to reproduce

I am unable to share test code/logs due to proprietary rights. However, I was able to reproduce a similar error where the test hangs on first retry when the done() callback is never invoked.

In cypress.json

"retries": {
    "runMode": 1,
    "openMode": 1
 }

Test Case

describe('Cypress Tests Hang', () => {
  it('Cypress Tests Hang', () => {
    cy.visit('https://www.google.com');
  });

  afterEach(run => {
    cy.log('In afterEach');
  });
});

You'll see that the first run fails, and the retry run will end up hanging

Cypress Version

9.7.0

Other

No response

[ZachJW34]

I was able to reproduce this with latest Cypress (9.7.0). Like you are suggesting @taylorjellsworth, there seems to be something going wrong with how we are processing the fail event for hooks that timeout. I think a timeout of the hook should act the same as an error occurring within the hook.

Here is a snapshot of the fail function executing when throwing an error in an afterEach

With the timeout, the breakpoint is never reached

Thanks for posting the issue, we'll add this bug to our backlog.

[taylorjellsworth]

@ZachJW34 Thanks for investigating! I agree, I think a timeout of the hook should act the same as an error occurring within the hook.

If it's possible, can this bug be prioritized? This is currently impacting our test suites, and if it gets stuck in the backlog we'd be forced to move off of Cypress (even though we really don't want to!)

[jennifer-shehane]

Maybe related to code-coverage plugin? Maybe there's an afterEach promise in there that doesn't resolve sometimes.

[taylorjellsworth]

I was wondering if there are any updates on this?

[taylorjellsworth]

Hi @jennifer-shehane ! This is still causing us issues, I was wondering if anyone has started looking into it yet?

[ivanspasov99]

Facing the same issue. When using cy.request().then() on before or when there are more than 2 it cypress stucks.

It strange because if I put the logic inside one it it works if that helps somehow.

[flotwig]

Similar issue here: #18502

[taylorjellsworth]

@flotwig I was wondering if there's any estimate on when this will get looked at? We are still running into this issue.

[taylorjellsworth]

@flotwig @mjhenkes @jennifer-shehane @mschile bump. We are close to moving off of Cypress if this is not addressed

[Regaddi]

I'd like to add that not being able to use retries also increases costs.
In our app there is always some network flakiness to expect, so tests can sporadically fail without a corresponding change to the application.
We currently have a test suite of 120 Cypress tests.
Every time a test fails due to some flakiness that could otherwise be covered with the retries option, we'd have to re-run the tests, which typically adds $0,72 to our bill for Cypress plus CI costs. (I know this isn't a super fair comparison, but I'd prefer knowing that a test passed in a second run than not re-running the test at all).
While I understand that fixing this issue is not necessarily in the interest of the Cypress team (because it drives revenue) it's certainly annoying enough to start considering alternatives (echo-ing @taylorjellsworth).

[tgdevereux]

Just to add to this, we appear to be hitting a similar issue, however it appears to happen regardless of retries being set.
And we’re repro-ing it when we hit a separate file download issue, possibly related to #1551

Basically, we have a test that downloads a file, and then hits a pageLoadTimeout, perhaps due to #1551
But if I have an afterEach block after this test (which just clicks a back button on the page), the test doesn’t throw the pageLoadTimeout error, and the test just hangs indefinitely. The tests are written in Typescript, running against a React UI.

Repro’d with:
Cypress version: 10.11.0
OS: macOS Big Sur (v11.7)
Browser version: Chrome 105.0.5195.125

[taylorjellsworth]

fyi - we have moved off of Cypress completely and onto Playwright due to this issue. We would have loved to continue to use Cypress, but the lack of communication (even after asking for updates many times) on this issue was the driving force. We could not wait around forever. Hopefully this issue gets resolved at some point for those who are still dealing with it.

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.

[mikhail-g]

I faced the same issue, it would be really nice having it fixed. Now we cannot use afterEach when having retries, it's very inconvenient

[ptrcortes]

I'm also experiencing this issue with Cypress 12. Would love a fix.

[sbfrancies-onyx]

I am experiencing this in cypress 13.11.0, my tests can hang indefinitely both locally and in an Azure Devops Pipeline when there are failures with retries enabled (we are using the cypress-cucumber-preprocessor which uses afterEach). The same behaviour does not appear to occur when retries are not enabled.