Skip to content

Commit

Permalink
feat: add reporters
Browse files Browse the repository at this point in the history 
PR-URL: nodejs/node#45712
Fixes: nodejs/node#45648
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
(cherry picked from commit a1b27b25bb01aadd3fd2714e4b136db11b7eb85a)
  • Loading branch information
@MoLow
MoLow committed Feb 7, 2023
1 parent e123c03 commit 61cf2c2
Show file tree
Hide file tree
Showing 33 changed files with 1,730 additions and 222 deletions.
240 changes: 206 additions & 34 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ Differences from the core implementation:
<!-- source_link=lib/test.js -->

The `node:test` module facilitates the creation of JavaScript tests that
report results in [TAP][] format. This package is a port of `node:test`.
The `node:test` module facilitates the creation of JavaScript tests.
To access it:

```mjs
Expand Down Expand Up @@ -99,9 +98,7 @@ test('callback failing test', (t, done) => {
})
```

As a test file executes, TAP is written to the standard output of the Node.js
process. This output can be interpreted by any test harness that understands
the TAP format. If any tests fail, the process exit code is set to `1`.
If any tests fail, the process exit code is set to `1`.

#### Subtests

Expand Down Expand Up @@ -130,8 +127,7 @@ test to fail.
## Skipping tests

Individual tests can be skipped by passing the `skip` option to the test, or by
calling the test context's `skip()` method. Both of these options support
including a message that is displayed in the TAP output as shown in the
calling the test context's `skip()` method as shown in the
following example.

```js
Expand Down Expand Up @@ -265,7 +261,7 @@ Test name patterns do not change the set of files that the test runner executes.

## Extraneous asynchronous activity

Once a test function finishes executing, the TAP results are output as quickly
Once a test function finishes executing, the results are reported as quickly
as possible while maintaining the order of the tests. However, it is possible
for the test function to generate asynchronous activity that outlives the test
itself. The test runner handles this type of activity, but does not delay the
Expand All @@ -274,13 +270,13 @@ reporting of test results in order to accommodate it.
In the following example, a test completes with two `setImmediate()`
operations still outstanding. The first `setImmediate()` attempts to create a
new subtest. Because the parent test has already finished and output its
results, the new subtest is immediately marked as failed, and reported in the
top level of the file's TAP output.
results, the new subtest is immediately marked as failed, and reported later
to the {TestsStream}.

The second `setImmediate()` creates an `uncaughtException` event.
`uncaughtException` and `unhandledRejection` events originating from a completed
test are marked as failed by the `test` module and reported as diagnostic
warnings in the top level of the file's TAP output.
warnings at the top level by the {TestsStream}.

```js
test('a test that creates asynchronous activity', t => {
Expand Down Expand Up @@ -431,6 +427,163 @@ test('spies on an object method', (t) => {
});
```


## Test reporters

<!-- YAML
added: REPLACEME
-->

The `node:test` module supports passing [`--test-reporter`][]
flags for the test runner to use a specific reporter.

The following built-reporters are supported:

* `tap`
The `tap` reporter is the default reporter used by the test runner. It outputs
the test results in the [TAP][] format.

* `spec`
The `spec` reporter outputs the test results in a human-readable format.

* `dot`
The `dot` reporter outputs the test results in a comact format,
where each passing test is represented by a `.`,
and each failing test is represented by a `X`.

### Custom reporters

[`--test-reporter`][] can be used to specify a path to custom reporter.
a custom reporter is a module that exports a value
accepted by [stream.compose][].
Reporters should transform events emitted by a {TestsStream}

Example of a custom reporter using {stream.Transform}:

```mjs
import { Transform } from 'node:stream';
const customReporter = new Transform({
writableObjectMode: true,
transform(event, encoding, callback) {
switch (event.type) {
case 'test:start':
callback(null, `test ${event.data.name} started`);
break;
case 'test:pass':
callback(null, `test ${event.data.name} passed`);
break;
case 'test:fail':
callback(null, `test ${event.data.name} failed`);
break;
case 'test:plan':
callback(null, 'test plan');
break;
case 'test:diagnostic':
callback(null, event.data.message);
break;
}
},
});
export default customReporter;
```

```cjs
const { Transform } = require('node:stream');
const customReporter = new Transform({
writableObjectMode: true,
transform(event, encoding, callback) {
switch (event.type) {
case 'test:start':
callback(null, `test ${event.data.name} started`);
break;
case 'test:pass':
callback(null, `test ${event.data.name} passed`);
break;
case 'test:fail':
callback(null, `test ${event.data.name} failed`);
break;
case 'test:plan':
callback(null, 'test plan');
break;
case 'test:diagnostic':
callback(null, event.data.message);
break;
}
},
});
module.exports = customReporter;
```

Example of a custom reporter using a generator function:

```mjs
export default async function * customReporter(source) {
for await (const event of source) {
switch (event.type) {
case 'test:start':
yield `test ${event.data.name} started\n`;
break;
case 'test:pass':
yield `test ${event.data.name} passed\n`;
break;
case 'test:fail':
yield `test ${event.data.name} failed\n`;
break;
case 'test:plan':
yield 'test plan';
break;
case 'test:diagnostic':
yield `${event.data.message}\n`;
break;
}
}
}
```

```cjs
module.exports = async function * customReporter(source) {
for await (const event of source) {
switch (event.type) {
case 'test:start':
yield `test ${event.data.name} started\n`;
break;
case 'test:pass':
yield `test ${event.data.name} passed\n`;
break;
case 'test:fail':
yield `test ${event.data.name} failed\n`;
break;
case 'test:plan':
yield 'test plan\n';
break;
case 'test:diagnostic':
yield `${event.data.message}\n`;
break;
}
}
};
```

### Multiple reporters

The [`--test-reporter`][] flag can be specified multiple times to report test
results in several formats. In this situation
it is required to specify a destination for each reporter
using [`--test-reporter-destination`][].
Destination can be `stdout`, `stderr`, or a file path.
Reporters and destinations are paired according
to the order they were specified.

In the following example, the `spec` reporter will output to `stdout`,
and the `dot` reporter will output to `file.txt`:

```bash
node --test-reporter=spec --test-reporter=dot --test-reporter-destination=stdout --test-reporter-destination=file.txt
```

When a single reporter is specified, the destination will default to `stdout`,
unless a destination is explicitly provided.

## `run([options])`

<!-- YAML
Expand Down Expand Up @@ -459,7 +612,7 @@ added: REPLACEME
incremented from the primary's `process.debugPort`.
**Default:** `undefined`.

* Returns: {TapStream}
* Returns: {TestsStream}

```js
run({ files: [path.resolve('./tests/test.js')] })
Expand Down Expand Up @@ -501,12 +654,11 @@ run({ files: [path.resolve('./tests/test.js')] })
- Returns: {Promise} Resolved with `undefined` once the test completes.

The `test()` function is the value imported from the `test` module. Each
invocation of this function results in the creation of a test point in the TAP
output.
invocation of this function results in reporting the test to the {TestsStream}.

The `TestContext` object passed to the `fn` argument can be used to perform
actions related to the current test. Examples include skipping the test, adding
additional TAP diagnostic information, or creating subtests.
additional diagnostic information, or creating subtests.

`test()` returns a `Promise` that resolves once the test completes. The return
value can usually be discarded for top level tests. However, the return value
Expand Down Expand Up @@ -545,8 +697,7 @@ thus prevent the scheduled cancellation.
* Returns: `undefined`.

The `describe()` function imported from the `test` module. Each
invocation of this function results in the creation of a Subtest
and a test point in the TAP output.
invocation of this function results in the creation of a Subtest.
After invocation of top level `describe` functions,
all top level tests and suites will execute.

Expand All @@ -572,8 +723,6 @@ Shorthand for marking a suite as `TODO`, same as
* Returns: `undefined`.

The `it()` function is the value imported from the `test` module.
Each invocation of this function results in the creation of a test point in the
TAP output.

## `it.skip([name][, options][, fn])`

Expand Down Expand Up @@ -961,46 +1110,69 @@ added: REPLACEME
This function is syntax sugar for [`MockTracker.method`][] with `options.setter`
set to `true`.

## Class: `TapStream`
## Class: `TestsStream`

<!-- YAML
added: REPLACEME
-->

* Extends {ReadableStream}

A successful call to [`run()`][] method will return a new {TapStream}
object, streaming a [TAP][] output
`TapStream` will emit events, in the order of the tests definition
A successful call to [`run()`][] method will return a new {TestsStream}
object, streaming a series of events representing the execution of the tests.
`TestsStream` will emit events, in the order of the tests definition

### Event: `'test:diagnostic'`

* `message` {string} The diagnostic message.
* `data` {Object}
* `message` {string} The diagnostic message.
* `nesting` {number} The nesting level of the test.

Emitted when [`context.diagnostic`][] is called.

### Event: `'test:fail'`

* `data` {Object}
* `details` {Object} Additional execution metadata.
* `duration` {number} The duration of the test in milliseconds.
* `error` {Error} The error thrown by the test.
* `name` {string} The test name.
* `nesting` {number} The nesting level of the test.
* `testNumber` {number} The ordinal number of the test.
* `todo` {string|undefined} Present if [`context.todo`][] is called
* `skip` {string|undefined} Present if [`context.skip`][] is called
* `todo` {string|boolean|undefined} Present if [`context.todo`][] is called
* `skip` {string|boolean|undefined} Present if [`context.skip`][] is called

Emitted when a test fails.

### Event: `'test:pass'`

* `data` {Object}
* `details` {Object} Additional execution metadata.
* `duration` {number} The duration of the test in milliseconds.
* `name` {string} The test name.
* `nesting` {number} The nesting level of the test.
* `testNumber` {number} The ordinal number of the test.
* `todo` {string|undefined} Present if [`context.todo`][] is called
* `skip` {string|undefined} Present if [`context.skip`][] is called
* `todo` {string|boolean|undefined} Present if [`context.todo`][] is called
* `skip` {string|boolean|undefined} Present if [`context.skip`][] is called

Emitted when a test passes.

### Event: `'test:plan'`

* `data` {Object}
* `nesting` {number} The nesting level of the test.
* `count` {number} The number of subtests that have ran.

Emitted when all subtests have completed for a given test.

### Event: `'test:start'`

* `data` {Object}
* `name` {string} The test name.
* `nesting` {number} The nesting level of the test.

Emitted when a test starts.

## Class: `TestContext`

An instance of `TestContext` is passed to each test function in order to
Expand Down Expand Up @@ -1092,9 +1264,9 @@ test('top level test', async (t) => {

### `context.diagnostic(message)`

- `message` {string} Message to be displayed as a TAP diagnostic.
- `message` {string}Message to be reported.

This function is used to write TAP diagnostics to the output. Any diagnostic
This function is used to write diagnostics to the output. Any diagnostic
information is included at the end of the test's results. This function does
not return a value.

Expand Down Expand Up @@ -1128,19 +1300,19 @@ test('top level test', async (t) => {

### `context.skip([message])`

- `message` {string} Optional skip message to be displayed in TAP output.
* `message` {string} Optional skip message.

This function causes the test's output to indicate the test as skipped. If
`message` is provided, it is included in the TAP output. Calling `skip()` does
`message` is provided, it is included in the output. Calling `skip()` does
not terminate execution of the test function. This function does not return a
value.

### `context.todo([message])`

- `message` {string} Optional `TODO` message to be displayed in TAP output.
* `message` {string} Optional `TODO` message.

This function adds a `TODO` directive to the test's output. If `message` is
provided, it is included in the TAP output. Calling `todo()` does not terminate
provided, it is included in the output. Calling `todo()` does not terminate
execution of the test function. This function does not return a value.

### `context.test([name][, options][, fn])`
Expand Down
Loading

0 comments on commit 61cf2c2

Please sign in to comment.