feat: onTestFailed hook

[antfu]

import { expect, it, onTestFailed } from 'vitest'

it('on-failed', () => {
  const square3 = 3 ** 2
  const square4 = 4 ** 2

  onTestFailed(() => {
    console.log('Unexpected error encountered, internal states:', { square3, square4 })
  })

  expect(Math.sqrt(square3 + square4)).toBe(4)
})

Motivation

It's common we use tests to debug our implementation. When debugging, we might want to inspect the intermediate state to see what is going on internally. While they are useful in debugging, it's could be noisy to print them when the test passes as expected.

Thus I propose to introduce an onTestFailed hook, so you can have a chance to execute some code before the test suite gets killed - with access to the internal closure. Like printing the state, doing some cleanup, or dumping something into the file system, etc. This might help you to identify the issue earlier when reading the CI output.

Details

onTestFailed is provided as a global API and also in the test context, with access to local closure. Similar to other hooks, it can be called multiple times to register multiple listeners. When passing an async function, it will be awaited sequentially.

Alternatives

• Expose this hook global from the package import { onTestFailed } from 'vitest' - we might still need it in the context as it might not work in concurrent tests. Implemented
• Expose this hook as an attribute of expect -> expect.onTestFailed() - the semantic feels a bit weird, but we could reuse it for concurrent.

[privatenumber]

I like that it's a simple yet elegant solution to debugging large test suites that have race conditions. I can imagine this coming in handy in unreliable tests that sometimes fail but can't be easily reproduced. I also love that this idea is framework agnostic.

Drawbacks:

• The hook can only be registered before the error throws.

  • Using the hook may be limited to tests that are separated by setup vs assertions, so the hook must be registered in between.
  • This limitation can be confusing for devs that don't realize where the error is coming from, and lead them to wonder why it's not triggering the hook.
  • It might be possible to work around this by statically extracting the hook function, but that may be too expensive and too magical.

• It encourages debugging code to be left inside test files.

  • This may not necessarily be bad if it's kept organized, but I can imagine a lot of just-in-case logs being left in. The debugging code might not be very reusable because it needs access to the closure.
  • It can also make it more acceptable to merge in unstable tests via "we'll debug it later as it runs on CI".

I would design the API to be more targeted for last resorts, and discourage frequent usage by:

• Emitting a warning that a hook was registered but not triggered.
• Have a config option to limit the number of them.
• Naming it something specific & discouraging (e.g. shamefullyDebugOnFail() inspired by pnpm's shamefully-hoist).

[antfu]

Thanks for the feedback!

The hook can only be registered before the error throws.

I think this is necessary due to the nature of JS. Hoisting might not be a good option as it requires accessing the closure and you can't know what variables are available from arbitrary crashing points.

It encourages debugging code to be left inside test files.

As you said, it's not necessarily a bad thing. The reason I call it onTestFailed in a more neutral way instead of debugOnFail is that it's not necessarily related to debugging. It can be a cleanup like releasing some resources or closing the server etc. A bit like a test level try...catch...finally.
If it's really a concern to some teams, we could introduce flags similar to --no-only for CI, or have an eslint rule to warn it. shamefully sounds a bit too much to me.