Ability to skip rest test cases if one fails

[vilicvane]

🚀 Feature Proposal

Provide ability to skip rest test cases in a file or describe block if one test fails.

Motivation

For E2E testing, the test cases are usually written to be run sequentially, and a failed test should stop the rest from running.

Example

test('should login', async () => {});

test('should edit profile', async () => {});

test('should render profile', async () => {});

Or with describes:

describe('#1', () => {
  test('should login', async () => {});

  describe('#1.1', () => {
    test('should edit profile', async () => {});

    test('should render profile', async () => {});
  });
});

[Mouvedia]

keyword: step
prior art: https://github.com/rprieto/mocha-steps

[jeysal]

Duplicate of #6527 (for hooks and tests)

[Mouvedia]

No it's not a duplicate. We are requesting mocha-steps not jasmine-fail-fast.

TL;DR: skip (and auto fail) the following steps after the first failing step within the current describe (not exit on first failure)

@jeysal please reopen.

[jeysal]

Okay then, we can keep this open to track fail-fast within a describe block (although the OP does not explicitly request this).
I think it's unlikely that this goes into Jest core though, while #6527 might as it's a logical extension of bail.

[G-Rath]

Personally I think this makes sense, and would love to see it in the jest core under step, as it lets you break tests into smaller parts (allowing you to describe them better, since they each have an it/step message), and increase performance for when you're testing something that is dependent on prior tests.

Take for example this test:

describe('.root', () => {
  it('contains the root package.json', () => {
    // eslint-disable-next-line @typescript-eslint/no-require-imports
    const requirePackageJson = () => require(path.join(pathTo.root, 'package.json'));

    expect(() => requirePackageJson()).not.toThrow();
    expect(requirePackageJson()).toStrictEqual(packageJson);
  });
});

It's checking that pathTo.root points to a directory with a package.json, and that the package.json is the actual root package.json (by comparing the object returned by the require to one imported via the normal module loader).

Since require throws an error if it can't find the file (the first expect), it makes no sense to test contents of the file (the second expect) if this happens.

But, since its are independent of each other, I either will have two tests failing (which while technically correct means I've got pointless noise that adds nothing to the situation), or one "big" it that combines two tests.

Meanwhile, if we had step:

describe('.root', () => {
  // eslint-disable-next-line @typescript-eslint/no-require-imports
  const requirePackageJson = () => require(path.join(pathTo.root, 'package.json'));

  step('has the root package.json', () => {
    expect(() => requirePackageJson()).not.toThrow();
  });
  step('contains the root package.json', () => {
    expect(requirePackageJson()).toStrictEqual(packageJson);
  });
});

---

This maybe be a result of me being relatively new to the testing game, but it seems to me that step would empower jest further, as it enables you to do feature-like testing letting you nicely test apps as a whole.

I know that step is pretty self explanatory in its advantages, I felt like it could be valuable to put one to paper - I've got a lot of tests that are similar to that.

I don't mind step not being in the jest core, just so long it's as easy to use as it, describe, #each, etc are today.

I'm just really excited at the idea of having this feature in jest, as I've come across this problem a lot! 😄

[cinderblock]

Smoke tests in front of more substantial tests save people time. Without the ability to short circuit subsequent test, the time saving is mitigated.

Where does this functionality need to be added? Would this live in a test runner or is this a jest issue? (Many people don't even realize the distinction between jest and test runners...)

[k1832]

Here is my temporary solution.

describe("test", () => {
  let skipOther = false
  test("check something", async () => {
    try {
      await checkSomething()
    } catch(e) {
      skipOther = true
      throw new Error("error")
    }
  })

  test("might be skipped", async () => {
    if(skipOther) throw new Error("error occured in previous tests.")
    console.log("display this, if there is no error before this.")
  })
})

[MrLoh]

I would be interested in implementing a test.step function in jest circus, I don't see a way to do this outside of the jest-circus core though, as it requires adding some flag to the state.

[MrLoh]

I created a PR that implements asynchronously canceling tests as described in #8604. With that in place, one could implement this functionality through easy monkey patching in user land like this:

let prevFailed;
const _describe = global.describe;
global.describe = (...args) => {
  prevFailed = false;
  return _describe(...args);
};
global.test.step = (name, fn, ...args) => {
  test(name, async function(...fnArgs) {
    if (prevFailed) {
      return this.skip();
    }
    try {
      return await fn(...fnArgs);
    } catch (e) {
      prevFailed = true;
      throw e;
    }
  }, ...args);
};

[FezVrasta]

@MrLoh where's the PR? I only see the issue