Allow to skip tests programmatically

[medikoo]

🚀 Feature Proposal

Mocha supports skipping tests programmatically (in both before and it) as:

describe("Some test" () => {
  it("Should skip", function () {
    if (true) {
      this.skip(); // test marked as skipped, no further part run
    }
    notInvoked();
  });
}):

Motivation

It's very useful for cases where during tests setup we find out whether test can be pursued or not e.g. we need some external data, but due to some unavailability we can't so we decide to skip tests.

Is this somewhere on a roadmap?

[palmerj3]

This seems like a bad idea to me. Currently you can do it.skip() to explicitly skip a particular test and it's not even executed.

Skipping programmatically, and only running a portion of your test suite as a result, doesn't seem like it's serving the purpose of tests. A test failure at that point would be beneficial so the problem(s) could be fixed. And if they can't be fixed then marking a test as skipped explicitly like I've showed above is an appropriate reaction.

[medikoo]

Skipping programmatically, and only running a portion of your test suite as a result, doesn't seem like it's serving the purpose of tests. A test failure at that point would be beneficial so the problem(s) could be fixed

It serves integration tests. Where tests depend on some external factors, and unavailability of some external resource, shouldn't indicate a problem with a project (reported with fail) but fact that test cannot be pursued (hence skipped)

[palmerj3]

I'll leave this here for others to discuss. But personally I don't think this is a great idea.

Your hypothetical example would not give confidence that any given change in a code base caused problems, thus why I said a test failure is beneficial.

[medikoo]

Your hypothetical example would not give confidence that any given change in a code base caused problems, thus why I said a test failure is beneficial.

Yes, it's not for a case, where we want to confirm our project is free of bugs on its own (that should be solved with unit tests or mocked integration tests).

It's about the case, where we test integration with external resource (such tests might be run on different schedule). Having fails for both resource unavailability and an errors in handling of it, makes tests unreliable, as it produces false positives and in result increases the risk of ignoring the latter type of the issue.

[palmerj3]

For that case you could consider using jest.retryTimes

[medikoo]

For that case you could consider using jest.retryTimes

I don't want to retry, I want to abort without fail (and also signal that test was not really run)

[mattphillips]

@medikoo I agree with @palmerj3

I think being able to dynamically disable tests kind of misses the point of testing.

Instead of disabling because of unavailability of some resource I would argue that you probably want to be alerted to this with the failure and then resolve the real problem of the resource not being available.

jest.retryTimes should help with this if it's just a case of the resource being flakey but if it is completely down then you have a bigger problem IMO and want to know about it 😄

[medikoo]

then resolve the real problem of the resource not being available.

When speaking of external resource I mean something I do not own or control, so it's not an issue I can resolve.

And uptime monitoring of external resources which alarms/informs me whether given service is accessible or not, is a different thing which I don't see as a part of integration tests.

[SimenB]

This is part of jasmine (pending global function), but I think it was an explicit choice not to port it to Circus.

@aaronabramov might have thoughts on that?

[boujeepossum]

what if the external source starts failing all the time? then you'll just have a skipped test that will never run.

i think for very specific scenarios you can just use:

test('stuff', () => {
  if (serviceUnavailable) {
    logSomething();
    return;
  }

  // else test something
});

but i agree with @palmerj3 that having this in the core doesn't look like a great idea

[medikoo]

@aaronabramov it's what we do now (return and log), still as we have large number of tests, those logs usually come unnoticed.

If they would be skipped, then in the final summary any skip if happened will be noticed.

[ryanmark]

This is a pretty common use case. Sometimes writing tests is hard and takes a long time to do it correctly. Sometimes writing a test to work only in certain circumstances is achievable in far less time and better than writing no tests at all or permanently skipping tests.

So for all the devs with deadlines, here is a hacky workaround:

let someTestName = 'some test';
let someTestCB = () => {
  it("Should skip", function () {
    notInvoked();
  });
};
if ( process.env['RUN_ALL_TESTS'] == 'yes' ) describe(someTestName, someTestCB);
else describe.skip(someTestName, someTestCB);

[SimenB]

Another possibility

if (thing) test.only('skipping all other things', () => {
  console.warn('skipping tests');
});

// ...all other things

Jest itself uses this to skip certain tests on windows