Screenshot (failed) captured during skipped tests

[jennifer-shehane]

Current behavior

In some circumstances invovling skipped tests and the Cypress 'fail' event listener a failure screenshot will be taken when it shouldn't be.

Desired behavior

1. A failure screenshot should never be taken when the test is skipped.
2. The fail event handler shouldn't be triggered during skipped tests.

Test code to reproduce

spec.js

Cypress.on('fail', () => {
  throw new Error('error')
})

describe('tests', () => {
  before(function () {
    this.skip()
  })
  it('failing test', () => {
    expect(true).to.be.false
  })
})

Cypress Version

8.2.0

Other

No response

[jmoses]

I'm seeing this with 8.4.1 using the https://github.com/cypress-io/cypress-skip-test plugin, as well.

[cesine]

I have this issue as well, i am not able to use this.skip() in an it like this:

describe('tests', () => {

  it('failing test', function () {
    if (process.env.FOO_ENABLED === 'false') {
      this.skip();
    }
    console.assert(false);
  });
});

I get the feeling that the docs are over simplifying, maybe there is not really Mocha under Cypress. https://docs.cypress.io/faq/questions/general-questions-faq#Mocha

Cypress isn't different from Mocha, it actually uses Mocha under the hood. All of your tests will be written on top of Mocha's bdd interface.

I can see references to mocha https://github.com/cypress-io/cypress/search?q=mocha

I made a sample repo which demos how Mocha skips work, in case you want to use it to find out why Cypress isn't able to support this.skip() https://github.com/cesine/mocha-skip-demo

[rkrisztian]

Same bug when using cypress-cucumber-preprocessor, which I use to filter tests by tags, thus I'm having plenty of failed screenshots when none of my tests fail.

[petetnt]

I think I am running into this, but even in a weirder form on 8.7.0.

In my case, is have two test specs, called Foo and Bar.

In Foo.spec.ts, I have a failing case:

describe('Foo', () => {
  it('foo fails', () => {
    throw new Error('Fail');
  });
});

I have bar.spec.ts

describe('Bar', () => {
  it.skip('skip', () => {
    // noop 
  });
});

When running the tests, I had the following screenshot saved:

screenshots/bar.spec.ts/Foo -- foo fails (failed).png

The test output shows that Bar - skip failed and the Foo - foo fails succeeded.

I have no idea what's going on with that.