Skip to content

Commit

Permalink
Graduate shared workers to be non-experimental
Browse files Browse the repository at this point in the history
  • Loading branch information
novemberborn committed Nov 1, 2021
1 parent 0edfd00 commit ad521af
Show file tree
Hide file tree
Showing 36 changed files with 122 additions and 193 deletions.
82 changes: 41 additions & 41 deletions docs/recipes/shared-workers.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Extending AVA using shared workers

Shared workers are a new, powerful (and experimental) AVA feature. A program can be loaded in a [worker thread](https://nodejs.org/docs/latest/api/worker_threads.html) in AVA's main process and then communicate with code running in the test workers. This enables your tests to better utilize shared resources during a test run, as well as providing opportunities to set up these resources before tests start (or clean them up after).
Shared workers are a powerful AVA feature. A program can be loaded in a [worker thread](https://nodejs.org/docs/latest/api/worker_threads.html) in AVA's main process and then communicate with code running in the test workers. This enables your tests to better utilize shared resources during a test run, as well as providing opportunities to set up these resources before tests start (or clean them up after).

When you use watch mode, shared workers remain loaded across runs.

## Enabling the experiment
## Enabling the experiment (only needed with AVA 3)

Shared workers are available when you use AVA with Node.js 12.17.0 or newer. AVA 3.13.0 or newer is required. It is an experimental feature so you need to enable it in your AVA configuration:

Expand All @@ -31,20 +31,20 @@ Here we'll discuss building low-level plugins.

### Registering a shared worker

Plugins are registered inside test workers. They'll provide the path for the main program, which AVA will load in a [worker thread](https://nodejs.org/docs/latest/api/worker_threads.html) in its main process. For each unique path one worker thread is started.
Plugins are registered inside test workers. They'll provide the path for the shared worker, which AVA will load in a [worker thread](https://nodejs.org/docs/latest/api/worker_threads.html) in its main process. For each unique path one worker thread is started.

Plugins communicate with their main program using a *protocol*. Protocols are versioned independently from AVA itself. This allows us to make improvements without breaking existing plugins. Protocols are only removed in major AVA releases.
Plugins communicate with their shared worker using a *protocol*. Protocols are versioned independently from AVA itself. This allows us to make improvements without breaking existing plugins. Protocols are only removed in major AVA releases.

Plugins can be compatible with multiple protocols. AVA will select the best protocol it supports. If AVA does not support any of the specified protocols it'll throw an error. The selected protocol is available on the returned worker object.

**While shared workers are experimental, there is only an unversioned *experimental* protocol. Breaking changes may occur with any AVA release.**
**For AVA 3, substitute `'ava4'` with `'experimental'`.**

```js
const {registerSharedWorker} = require('ava/plugin');
import {registerSharedWorker} from 'ava/plugin';

const shared = registerSharedWorker({
filename: path.resolve(__dirname, 'worker.js'),
supportedProtocols: ['experimental']
supportedProtocols: ['ava4']
});
```

Expand All @@ -55,61 +55,61 @@ You can supply a `teardown()` function which will be called after all tests have
```js
const worker = registerSharedWorker({
filename: path.resolve(__dirname, 'worker.js'),
supportedProtocols: ['experimental'],
supportedProtocols: ['ava4'],
teardown () {
// Perform any clean-up within the test process itself.
}
});
```

You can also provide some data passed to the main program when it is loaded. Of course, it is only loaded once, so this is only useful in limited circumstances:
You can also provide some data passed to the shared worker when it is loaded. Of course, it is only loaded once, so this is only useful in limited circumstances:

```js
const shared = registerSharedWorker({
filename: path.resolve(__dirname, 'worker.js'),
initialData: {hello: 'world'},
supportedProtocols: ['experimental']
supportedProtocols: ['ava4']
});
```

On this `shared` object, `protocol` is set to the selected protocol. Since the main program is loaded asynchronously, `available` provides a promise that fulfils when the main program first becomes available. `currentlyAvailable` reflects whether the worker is, well, currently available.
On this `shared` object, `protocol` is set to the selected protocol. Since the shared worker is loaded asynchronously, `available` provides a promise that fulfils when the shared worker first becomes available. `currentlyAvailable` reflects whether the worker is, well, currently available.

There are two more methods available on the `shared` object, which we'll get to soon.

#### Initializing the main program
#### Initializing the shared worker

AVA loads the main program (as identified through the `filename` option) in a worker thread. The program must export a factory method. For CJS programs this can be done by assigning `module.exports` or `exports.default`. For ESM programs you must use `export default`. If the `filename` to an ESM program is an absolute path it must be specified using the `file:` protocol.
AVA loads the shared worker (as identified through the `filename` option) in a worker thread. This must be an ES module file with a default export. The filename must be an absolute path using the `file:` protocol or a `URL` instance.

Like when calling `registerSharedWorker()`, the factory method must negotiate a protocol:
The default export must be a factory method. Like when calling `registerSharedWorker()`, it must negotiate a protocol:

```js
exports.default = ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']);
};
export default ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']);
}
```

On this `main` object, `protocol` is set to the selected protocol. `initialData` holds the data provided when the worker was first registered.

When you're done initializing the main program you must call `main.ready()`. This makes the worker available in test workers. You can call `main.ready()` asynchronously.
When you're done initializing the shared worker you must call `main.ready()`. This makes the worker available in test workers. You can call `main.ready()` asynchronously.

Any errors thrown by the factory method will crash the worker thread and make the worker unavailable in test workers. The same goes for unhandled rejections. The factory method may return a promise.

### Communicating between test workers and the worker thread
### Communicating between test workers and the shared worker

AVA's low-level shared worker infrastructure is primarily about communication. You can send messages from test workers to the shared worker thread, and the other way around. Higher-level logic can be implemented on top of this message passing infrastructure.
AVA's low-level shared worker infrastructure is primarily about communication. You can send messages from test workers to the shared worker, and the other way around. Higher-level logic can be implemented on top of this message passing infrastructure.

Message data is serialized using the [V8 Serialization API](https://nodejs.org/docs/latest-v12.x/api/v8.html#v8_serialization_api). Please read up on some [important limitations](https://nodejs.org/docs/latest-v12.x/api/worker_threads.html#worker_threads_port_postmessage_value_transferlist).

In the main program you can subscribe to messages from test workers:
In the shared worker you can subscribe to messages from test workers:

```js
exports.default = async ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default async ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

for await (const message of main.subscribe()) {
//
}
};
}
```

Messages have IDs that are unique for the main AVA process. Across AVA runs you may see the same ID. Access the ID using the `id` property.
Expand All @@ -121,8 +121,8 @@ You can reply to a received message by calling `reply()`. This publishes a messa
To illustrate this here's a "game" of Marco Polo:

```js
exports.default = ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

play(main.subscribe());
};
Expand All @@ -134,23 +134,23 @@ const play = async (messages) => {
play(response.replies());
}
}
};
}
```

(Of course this sets up many reply listeners which is rather inefficient.)

You can also broadcast messages to all connected test workers:

```js
exports.default = async ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default async ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

for await (const message of main.subscribe()) {
if (message.data === 'Bingo!') {
main.broadcast('Bingo!');
}
}
};
}
```

Like with `reply()`, `broadcast()` returns a published message which can receive replies. Call `replies()` to get an asynchronous iterator for reply messages.
Expand All @@ -162,13 +162,13 @@ These test workers have a unique ID (which, like message IDs, is unique for the
Of course you don't need to wait for a message *from* a test worker to access this object. Use `main.testWorkers()` to get an asynchronous iterator which produces each newly connected test worker:

```js
exports.default = async ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default async ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

for await (const testWorker of main.testWorkers()) {
main.broadcast(`New test file: ${testWorker.file}`);
}
};
}
```

Within test workers, once the shared worker is available, you can publish messages:
Expand Down Expand Up @@ -197,31 +197,31 @@ Messages are always produced in their own turn of the event loop. This means you

### Cleaning up resources

Test workers come and go while the worker thread remains. It's therefore important to clean up resources.
Test workers come and go while the shared worker remains. It's therefore important to clean up resources.

Messages are subscribed to using async iterators. These return when the test worker exits.

You can register teardown functions to be run when the test worker exits:

```js
exports.default = async ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default async ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

for await (const testWorker of main.testWorkers()) {
testWorker.teardown(() => {
// Bye bye…
});
}
};
}
```

The most recently registered function is called first, and so forth. Functions execute sequentially.

More interestingly, a wrapped teardown function is returned so that you can call it manually. AVA still ensures the function only runs once.

```js
exports.default = ({negotiateProtocol}) => {
const main = negotiateProtocol(['experimental']).ready();
export default ({negotiateProtocol}) => {
const main = negotiateProtocol(['ava4']).ready();

for await (const worker of testWorkers) {
counters.set(worker, 0);
Expand All @@ -231,7 +231,7 @@ exports.default = ({negotiateProtocol}) => {

waitForTen(worker.subscribe(), teardown);
}
};
}

const counters = new WeakMap();

Expand All @@ -255,4 +255,4 @@ Not sure what to build? Previously folks have expressed a desire for mutexes, ma

We could also extend the shared worker implementation in AVA itself. Perhaps so you can run code before a new test run, even with watch mode. Or so you can initialize a shared worker based on the AVA configuration, not when a test file runs.

Please [comment here](https://github.com/avajs/ava/issues/2605) with ideas, questions and feedback.
Please [comment here](https://github.com/avajs/ava/discussions/2703) with ideas, questions and feedback.
4 changes: 1 addition & 3 deletions lib/load-config.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ import {packageConfig, packageJsonPath} from 'pkg-conf';

const NO_SUCH_FILE = Symbol('no ava.config.js file');
const MISSING_DEFAULT_EXPORT = Symbol('missing default export');
const EXPERIMENTS = new Set([
'sharedWorkers',
]);
const EXPERIMENTS = new Set();

const importConfig = async ({configFile, fileForErrorMessage}) => {
const {default: config = MISSING_DEFAULT_EXPORT} = await import(url.pathToFileURL(configFile)); // eslint-disable-line node/no-unsupported-features/es-syntax
Expand Down
4 changes: 2 additions & 2 deletions lib/plugin-support/shared-worker-loader.js
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ loadFactory(workerData.filename).then(factory => {

factory({
negotiateProtocol(supported) {
if (!supported.includes('experimental')) {
if (!supported.includes('ava4')) {
fatal = new Error(`This version of AVA (${pkg.version}) is not compatible with shared worker plugin at ${workerData.filename}`);
throw fatal;
}
Expand Down Expand Up @@ -213,7 +213,7 @@ loadFactory(workerData.filename).then(factory => {

return {
initialData: workerData.initialData,
protocol: 'experimental',
protocol: 'ava4',

ready() {
signalAvailable();
Expand Down
7 changes: 2 additions & 5 deletions lib/worker/plugin.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ function createSharedWorker(filename, initialData, teardown) {

return {
available: channel.available,
protocol: 'experimental',
protocol: 'ava4',

get currentlyAvailable() {
return channel.currentlyAvailable;
Expand All @@ -90,15 +90,12 @@ function registerSharedWorker({
teardown,
}) {
const options_ = options.get();
if (!options_.experiments.sharedWorkers) {
throw new Error('Shared workers are experimental. Opt in to them in your AVA configuration');
}

if (!options_.workerThreads) {
throw new Error('Shared workers can be used only when worker threads are enabled');
}

if (!supportedProtocols.includes('experimental')) {
if (!supportedProtocols.includes('ava4')) {
throw new Error(`This version of AVA (${pkg.version}) does not support any of the desired shared worker protocols: ${supportedProtocols.join(',')}`);
}

Expand Down
Loading

0 comments on commit ad521af

Please sign in to comment.