Allow asynchronously skipping tests

[wolever]

🚀 Feature Proposal

Other testing frameworks allow tests to asynchronously decide whether they should skip themselves.

For example, in Mocha:

it('tests a remote service', async function() {
  if (!(await remoteService.isActive())) {
    return this.skip()
  }
  … test the remote service …
})

Currently, however, it is impossible to asynchronously decide whether a test should be skipped in Jest.

See also: discussion here: #7245

Motivation

Some tests either depend on - or are explicitly testing - remote services which may or may not be available.

Without being able to programatically and asynchronously decide whether tests can be skipped, there are only three options for writing these sorts of tests:

1. Decide that they will either always pass or always fail if the service is unavailable. In either case the result can be misleading (ie, because in many cases "failure" indicates "the service is wrong", not merely "the service is unavailable", and "passing" suggests that everything is okay, which is also not necessarily true).

2. Keep them in a separate suite, one per remote service, which can be run with (for example), npm run test:service-a).

3. Use a regular expression (or similar) to include / exclude these tests from a test run.

Example

A complete, real-world (but anonymized) example from a Mocha-based test suite:

describe('example.com', () => {
  beforeAll(async function() {
    try {
      await fetch('https://example.com/api/whoami')
    } catch (e) {
      return this.skip(`Skipping tests for example.com: ${e}`)
    }
  })

  it('returns the current user', async () => { … })
  it('does the other thing', async () => { … })
})

Pitch

This belongs in Jest core because:

1. It's currently supported by Mocha: https://mochajs.org/ (search for this.skip)
2. It's impossible to implement outside of core (see elaboration below)
3. The related discussion on Allow to skip tests programmatically #7245 suggests that it's important to a number of people (see, ex, this comment, which as of this writing has 16 👍 : Allow to skip tests programmatically #7245 (comment))

FAQ

Why can't you use an if-statement?

A common suggestion in #7245 is to use an if-statement (or similar) to skip tests:

describe('example.com', () => {
  const isActive = remoteService.isActive()
  if (isActive) it('returns the current user', async () => { … })
})

However, this will not work for asynchronous tests, as tests must be declared synchronously, but the "is a remote service active?" check is necessarily asynchronous.

Wouldn't it be better if the tests failed/succeeded/retried/did something else?

There are situations when this is true, but (as evidenced by discussion on #7245) there are also situations where "skip tests when a remote service is not available" is a reasonable business decision (ex: #7245 (comment))

[mattphillips]

I struggle to see the value of this feature, you essentially want to be have support for a sort of pseudo false-positive state. When service A is unavailable then skip test B, what do you want to do with that information?

By skipping in a CI environment you will receive no notification of this state without explicitly going in and checking, what is the difference between just leaving the test as a pass when the service is unavailable?

[lucasfcosta]

My 2cents on this matter (quite an interesting thing to think about btw):

@mattphillips I guess there might be no practical difference but the semantic difference is huge. This matters if you want to use the information produced by Jest to determine what happened during the build.

A practical example: consider a test which depends on an external service. I push new code which would make that test fail and introduce a regression. When CI runs the checks on the code I’ve pushed, the service was not available but the test is marked as passed, even though it didn’t run. My code then fails in production but I have no way of determining whether it was because the test was flaky or if it was because it didn’t run.

The false positive state, as you have described, would be to have a test which is actually not running but being marked as passed.

Now, I can understand the appeal for this feature as it has a real world correlation, but one might argue that introducing it in would encourage — or at least acknowledge that it is common to write — tests that aren’t deterministic (I.e. tests which rely on something you do not have control over). On the other hand, there doesn’t seem to be any reasonable manner to implement this outside of the core and it might be a decision from Jest that it doesn’t want to be prescriptive over what it’s users should do. In the case described by @wolever, for example, it seems reasonable to have tests being asynchronously skipped.

To sum it up: I’d say this is a valid feature, even though it’s encouraged only for a minority of cases. Since in those cases there doesn’t seem to be a workaround which is “good enough” (or precise enough), I’d say it would be ok to have it in.

[langpavel]

What about some cascading?
Like if precondition test will fail
only one error will be reported with count of skipped tests?

If i wrote test for 3rd party service availability
only this one fails and I can see real reason why other tests not run?

Feels much better than seeing hundreds of failing tests.

[wolever]

Thanks @lucasfcosta - that's a great summary of the problem.

I would also add that the "conditional skipping" can also be environmentally aware. For example, in my environment, there are certain tests which are skipped only if they are running in a development environment (ex, tests which require a certain amount of local setup), but required in a CI environment:

describe('some complicated service', () => {
  beforeAll(async () => {
    const shouldSkip = (
      process.env.NODE_ENV == 'development' &&
      await checkServiceAvailability()
    )
    if (shouldSkip)
      return this.skip(`Skipping tests because ComplicatedService is not running (hint: run it with ./complicated-service)`)
  })

  … rest of the tests …
})

[ClaytonSmith]

@mattphillips What would your solution be for platform specific tests?

It would be wildly inappropriate to falsely mark tests as passed or failed when it doesn't even make sense to run the test.

The questions we want our test frameworks to answer are: Can I run this test? Did the test pass? A boolean Pass/Fail cannot answer both of those question.

[langpavel]

jest can mark tests as skipped, I'm using this hack, but it only works synchronously:

const haveDb = !!process.env.DB_CONNECTION_STRING;
const testDb = haveDb ? test : test.skip;

[GoMino]

Any update on this ?

[jfairley]

To add to @langpavel's idea, you could have an asynchronous check in a custom testEnvironment class.

const { connect } = require("amqplib");
const { EventEmitter } = require("events");
const NodeEnvironment = require("jest-environment-node");
const { logger } = require("../src/logging");

class TestEnvironment extends NodeEnvironment {
  constructor(config) {
    super(config);
  }

  async setup() {
    await super.setup();

    // rabbit
    try {
      this.global.amqpConnection = await connect({
        username: "admin",
        password: "admin",
      });
      this.global.channel = await this.global.amqpConnection.createChannel();
      this.global.hasAmqp = true;
    } catch (err) {
      logger.warn("AMQP is not available. Skipping relevant tests.");
      this.global.hasAmqp = false;
    }
  }

  async teardown() {
    if (this.global.amqpConnection) {
      await this.global.amqpConnection.close();
    }
    await super.teardown();
  }
}

module.exports = TestEnvironment;

describe("example", () => {
  const hasAmqp: boolean = (global as any).hasAmqp;
  const channel: Channel = (global as any).channel;

  it = hasAmqp ? test : test.skip;

  it("should send/receive", async () => {
    // ...
  });
});

[nstaromana]

I want to add a use case for programmatically skipping tests:
Our product runs on multiple platforms & uses the same Jest tests for each platform. Sometimes a platform implementation is not yet complete, the product will report this dynamically via API (but only after setup in beforeAll()).
Our tests also use snapshot matching. What happens is, if a platform doesn't support a feature and the test bails out, Jest will complain about the unused snapshot, resulting in a test failure.
Our current workaround is to use 'inline snapshots', but this is exceedingly cumbersome.
A programmatic skip() API would allow our tests to skip when a feature is not available on a certain platform, and also avoid having the Jest engine complain about unused snapshots.

[MrLoh]

I created a proof of concept PR (#9944) that demonstrates how easy this is to implement in jest-circus. I think it is well worth adding this functionality to jest-circus as it allows to implement custom bailing behaviors (#6527) or stepped tests (#8387) in user land easily.