Exception in root level after() doesn't fail test run in interactive mode

[amitzur]

Current behavior:

If there's an after() outside of any describe, it or context ("rool level"), which throws an exception, that doesn't fail the test run. This is different from how it works in mocha, where you get a "after all" hook failure.

Also, if it's a synchronous failure, the exception doesn't show up anywhere.
If it's async, it shows up like this:

Desired behavior:

The whole test run should fail (sometimes you would want to place an assertion there that is possible only after all tests have run).

Steps to reproduce:

Add this to your tests:

after(() => {
  cy.then(() => {
    throw new Error('bla bla');
  });
});

Versions

Cypress: 3.0.3
OS: Mac OS X 10.12.6 (Sierra)
Browser: both Electron (59) and Chrome (68)

[amitzur]

I just found out that running cypress run or through the module API does report a failed test. It's just cypress open that shows the test as green.

[jennifer-shehane]

Yes, this is likely a UI level issue if that is the case.

[amitzur]

Hi there, is there progress in investigation of this issue?

[kuceb]

@amitzur considering this is only a UI bug, it's a pretty low priority at the moment, but PRs are welcome!

[amitzur]

glad to take the challenge. Any chance to get a head start on code? I already got familiar with it, but where are test results communicated to the client?

[jennifer-shehane]

@amitzur The code for this is in our reporter package which is a React app. I'm not totally sure where this discrepancy is at - but here's a link to the runnable area of the app (may not be where the error is), but hopefully gets you started. https://github.com/cypress-io/cypress/blob/develop/packages/reporter/src/runnables/runnable-and-suite.jsx#L8

[amitzur]

great, thanks!
2 things:

1. The URL you listed is from the whitelist branch, it needs to be fixed to master or develop.
2. How do you setup React devtools? Via a browser extension or a standalone app? I couldn't easily find how to inject <script src="http://localhost:8097"></script> if it's a standalone app, or how to provide a path to the extension via a plugin.

[jennifer-shehane]

1. Oops, yeah, should be PR from develop
2. @chrisbreiding

[chrisbreiding]

If you run in Chrome, you should be able to install the extension the standard way by visiting https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi

[jennifer-shehane]

Example repo of failure: https://github.com/danielschwartz85/after-error-not-shown provided in #3348

[jennifer-shehane]

Trying to find the root of this issue:

It looks like the test:after:run event is never fired after the after runs and throws, although this test:after:run event is fired when there is an error thrown in the other hooks (before, etc).

Because of this, my suspicion is that the runnables object is not being built properly, so the Test Runner is not displaying the failure properly.

I tested that if the correct runnables object is sent to the UI, that it's rendered properly, so I think the object must be malformed somewhere.

afterEach failure runnable

{
  "id": "r3",
  "title": "test here",
  "hookName": "after each",
  "err": {
    "message": "bla\n\nBecause this error occurred during a 'after each' hook we are skipping all of the remaining tests.",
    "name": "Error",
    "stack": "Error: bla\n\nBecause this error occurred during a 'after each' hook we are skipping all of the remaining tests.\n    at Context.<anonymous> (http://localhost:56948/__cypress/tests?p=cypress/integration/issue_674_spec.coffee-840:4:11)"
  },
  "state": "failed",
  "failedFromHookId": "h1",
  "body": "function() {\n    return cy.window();\n  }",
  "type": "test",
  "duration": 328,
  "wallClockStartedAt": "2019-05-02T10:13:47.366Z",
  "wallClockDuration": 409,
  "timings": {
    "lifecycle": 47,
    "test": {
      "fnDuration": 22,
      "afterFnDuration": 1
    },
    "after each": [
      {
        "hookId": "h1",
        "fnDuration": 22,
        "afterFnDuration": 306
      }
    ]
  }
}

after failure runnable

{
  "id": "r3",
  "title": "test here",
  "state": "passed",
  "body": "function() {\n    return cy.window();\n  }",
  "type": "test",
  "duration": 75,
  "wallClockStartedAt": "2019-05-02T10:15:46.442Z",
  "wallClockDuration": 82,
  "timings": {
    "lifecycle": 45,
    "test": {
      "fnDuration": 28,
      "afterFnDuration": 2
    }
  }
}

[danielschwartz85]

Hi, any updates / plans regarding when this would be fixed ?
Thanks.

[cypress-bot]

The code for this is done in cypress-io/cypress#7035, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[jennifer-shehane]

Sorry, we accidentally merged the PR above prematurely. This is still in the review stage. We will close this issue and comment when this issue is merged for the next upcoming release.

[cypress-bot]

The code for this is done in cypress-io/cypress#7154, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[cypress-bot]

Released in 4.8.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v4.8.0, please open a new issue.