(docs,snippets): redo formatter docs, new loading strategy for snippet syntaxes

.gitattributes

@@ -1 +1,3 @@
 * text eol=lf
+*.gif binary
+*.png binary

CHANGELOG.md

@@ -15,6 +15,7 @@ Please see [CONTRIBUTING.md](https://github.com/cucumber/cucumber/blob/master/CO
 * Remove deprecated `--retryTagFilter` option (the correct option is `--retry-tag-filter`)
 * Remove validation that step definition functions are not generators
 * Remove `--predictable-ids` option (was only used for internal testing)
+* Relative paths for custom snippet syntaxes must begin with `.` ([#1640](https://github.com/cucumber/cucumber-js/issues/1640))
 
 ### Added
 

README.md

@@ -55,8 +55,6 @@ $ npm install cucumber@6
 The following documentation is for master. See below the documentation for older versions.
 
 * [CLI](/docs/cli.md)
-* [Custom Formatters](/docs/custom_formatters.md)
-* [Custom Snippet Syntaxes](/docs/custom_snippet_syntaxes.md)
 * [NodeJs Example](/docs/nodejs_example.md)
 * Support Files
   * [World](/docs/support_files/world.md)
@@ -68,9 +66,12 @@ The following documentation is for master. See below the documentation for older
   * [API Reference](/docs/support_files/api_reference.md)
 * Guides
   * [ES Modules](./docs/esm.md)
+  * [Formatters](./docs/formatters.md)
   * [Running in parallel](./docs/parallel.md)
-  * [Retrying failing scenarios](./docs/retry.md)
   * [Profiles](./docs/profiles.md)
+  * [Rerunning just failures](./docs/rerun.md)
+  * [Retrying flaky scenarios](./docs/retry.md)
+  * [Snippets for undefined steps](./docs/snippets.md)
 * [FAQ](/docs/faq.md)
 
 #### Documentation for older versions

docs/cli.md

@@ -44,46 +44,17 @@ _Note that once you specify any `--require` options, the defaults described abov
 
 Use `--format <TYPE[:PATH]>` to specify the format of the output.
 
-The `TYPE` can be one of:
-* The name of one of the built-in formatters (below) e.g. `progress`
-* A module/package name e.g. `@cucumber/pretty-formatter`
-* A relative path to a local formatter implementation e.g. `./my-customer-formatter.js`
-
-If `PATH` is not supplied, the formatter prints to `stdout`.
-If `PATH` is supplied, it prints to the given file.
-
-This option may be used multiple times in order to output different formats to different files.
-If multiple formats are specified with the same output, only the last is used.
-
-### Built-in formatters
-
-* **message** - prints each [message](https://github.com/cucumber/cucumber/tree/master/cucumber-messages) in NDJSON form, which can then be consumed by other tools.
-* **html** - prints a rich HTML report to a standalone page
-* **json** - prints the feature as JSON. *Note: this formatter is in maintenance mode and won't have new features added to it. Where you need a structured data representation of your test run, it's best to use the `message` formatter. Tools that rely on this formatter will continue to work, but are encouraged to migrate to consume the `message` output instead.*
-* **progress** - prints one character per scenario (default).
-* **progress-bar** - prints a progress bar and outputs errors/warnings along the way.
-* **rerun** - prints the paths of any non-passing scenarios ([example](/features/rerun_formatter.feature))
-  * suggested use: add the rerun formatter to your default profile and the output file to your `.gitignore`.
-  * After a failed run, remove any arguments used for selecting feature files and add the rerun file in order to rerun just failed scenarios. The rerun file must start with an `@` sign in order for cucumber to parse it as a rerun file instead of a feature file.
-  * Use with `--fail-fast` to rerun the failure and the remaining features.
-* **snippets** - prints just the code snippets for undefined steps.
-* **summary** - prints a summary only, after all scenarios were executed.
-* **usage** - prints a table with data about step definitions usage.
-* **usage-json** - prints the step definitions usage data as JSON.
+See [Formatters](./formatters.md).
 
 ### Officially-supported standalone formatters
 
 * [@cucumber/pretty-formatter](https://www.npmjs.com/package/@cucumber/pretty-formatter) - prints the feature with inline results,  colours and custom themes.
 
 ### Format Options
 
-You can pass in format options with `--format-options <JSON>`. The JSON string must define an object. This option is repeatable and the objects will be merged with the last instance taking precedence.
+You can pass in format options with `--format-options <JSON>`.
 
-* Suggested use: add with profiles so you can define an object and use `JSON.stringify` instead of writing `JSON` manually.
-
-## Colors
-
-Colors can be disabled with `--format-options '{"colorsEnabled": false}'`
+See [Formatters](./formatters.md).
 
 ## Exiting
 
@@ -99,27 +70,6 @@ disable _strict_ mode.
 
 By default, cucumber works in _strict_ mode, meaning it will fail if there are pending steps.
 
-## Undefined Step Snippets
-
-Undefined steps snippets are printed in JavaScript using the callback interface by default.
-
-### Interface
-
-Override the snippet interface with `--format-options '{"snippetInterface": "<interface>"}'`.
-Valid interfaces are 'async-await', 'callback', 'generator', 'promise', or 'synchronous'.
-
-### Syntax
-
-Override the snippet syntaxes with `--format-options '{"snippetSyntax": "<FILE>"}'`.
-See [here](/docs/custom_snippet_syntaxes.md) for documentation about building a custom snippet syntax.
-
-## Rerun separator
-
-The separator used by the rerun formatter can be overwritten by specifying `--format-options '{"rerun": {"separator": "<separator>"}}'`.
-This is useful when one needs to rerun failed tests locally by copying a line from a CI log while using a space character as a separator.
-The default separator is a newline character.
-Note that the rerun file parser can only work with the default separator for now.
-
 ## Parallel
 
 See [Parallel](./parallel.md).

docs/custom_snippet_syntaxes.md

@@ -1,4 +1,4 @@
-#### Custom snippet syntaxes
+# Custom Snippet Syntaxes
 
 * See the [JavaScript syntax](/src/formatter/step_definition_snippet_builder/javascript_snippet_syntax.ts) and the [custom snippet syntax](/features/step_definition_snippets_custom_syntax.feature) for examples.
   * Arguments passed to the constructor:

docs/formatters.md

@@ -0,0 +1,127 @@
+# Formatters
+
+In cucumber-js, Formatters ingest data about your test run in real time and then output content, either to the console or a file, in a useful format. (Some frameworks refer to this kind of thing as "reporters".)
+
+cucumber-js provides many built-in Formatters, plus building blocks with which you can [write your own](./custom_formatters.md).
+
+You can specify one or more formats via the `--format <TYPE[:PATH]>` CLI option, where `TYPE` is one of:
+
+* The name of one of the built-in formatters (below) e.g. `progress`
+* A module/package name e.g. `@cucumber/pretty-formatter`
+* A relative path to a local formatter implementation e.g. `./my-customer-formatter.js`
+
+If `PATH` is supplied, the formatter prints to the given file, otherwise it prints to `stdout`.
+
+Some notes on specifying Formatters:
+
+* If multiple formatters are specified with the same output, only the last is used.
+* If no formatter for `stdout` is specified, we default to the `progress` formatter.
+
+## Options
+
+Many formatters, including the built-in ones, support some configurability via options. You can provide this data as a JSON literal via the `--format-options` CLI option, like this:
+
+```shell
+$ cucumber-js --format-options '{"someOption":true}'
+```
+
+This option is repeatable, so you can use it multiple times and the objects will be merged with the later ones taking precedence.
+
+Some options offered by built-in formatters:
+
+- `colorsEnabled` - if set to `false`, colors in terminal output are disabled.
+
+## Built-in formatters
+
+### `summary`
+
+The Summary Formatter outputs a summary of the test run's results.
+
+If everything passed, this will be short and sweet:
+
+![](./images/summary_green.gif)
+
+If there were issues, you'll see useful details including:
+
+- Failed hooks or steps including error messages and stack traces
+- Locations of any pending steps
+- [Snippets](./snippets.md) to implement any undefined steps
+
+### `progress`
+
+The Progress Formatter has the same output as the Summary Formatter at the end of the test run, but also provides concise real-time feedback each time a step or hook completes:
+
+![](./images/progress.gif)
+
+### `progress-bar`
+
+Similar to the Progress Formatter, but provides a real-time updating progress bar based on the total number of steps to be executed in the test run:
+
+![](./images/progress_bar_green.gif)
+
+*Note: the Progress Bar Formatter will only work with a TTY terminal (and not, for example, a file stream).*
+
+### `html`
+
+The HTML Formatter produces a rich interactive report bundled as a standalone HTML page:
+
+![](./images/html_formatter.png)
+
+You can:
+
+- See detailed results including error messages and stack traces
+- See attachments rendered in-place
+- Filter to specific statuses
+- Search by keywords or tag expressions
+
+### `message`
+
+Outputs all the [Cucumber Messages](https://github.com/cucumber/common/tree/main/messages) for the test run as newline-delimited JSON, which can then be consumed by other tools.
+
+### `json`
+
+Outputs details of the test run in the legacy JSON format.
+
+*Note: this formatter is in maintenance mode and won't have new features added to it. Where you need a structured data representation of your test run, it's best to use the `message` formatter. Tools that rely on this formatter will continue to work, but are encouraged to migrate to consume the `message` output instead.*
+
+### `snippets`
+
+The Snippets Formatter doesn't output anything regarding the test run; it just prints [Snippets to implement any undefined steps](./snippets.md). This is useful when you want to quickly zero in on the steps you have to implement and grab the snippet code for them in one go.
+
+### `usage`
+
+The Usage Formatter lists your step definitions and tells you about usages in your scenarios, including the duration of each usage, and any unused steps. Here's an example of the output:
+
+```
+┌───────────────────────────────────────┬──────────┬─────────────────────────────────┐
+│ Pattern / Text                        │ Duration │ Location                        │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ an empty todo list                    │ 760.33ms │ support/steps/steps.ts:6        │
+│   an empty todo list                  │ 820ms    │ features/empty.feature:4        │
+│   an empty todo list                  │ 761ms    │ features/adding-todos.feature:4 │
+│   an empty todo list                  │ 700ms    │ features/empty.feature:4        │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ I add the todo {string}               │ 432.00ms │ support/steps/steps.ts:10       │
+│   I add the todo "buy some cheese"    │ 432ms    │ features/adding-todos.feature:5 │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ my cursor is ready to create a todo   │ 53.00ms  │ support/steps/steps.ts:27       │
+│   my cursor is ready to create a todo │ 101ms    │ features/empty.feature:10       │
+│   my cursor is ready to create a todo │ 5ms      │ features/adding-todos.feature:8 │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ no todos are listed                   │ 46.00ms  │ support/steps/steps.ts:15       │
+│   no todos are listed                 │ 46ms     │ features/empty.feature:7        │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ the todos are:                        │ 31.00ms  │ support/steps/steps.ts:21       │
+│   the todos are:                      │ 31ms     │ features/adding-todos.feature:6 │
+├───────────────────────────────────────┼──────────┼─────────────────────────────────┤
+│ I remove the todo {string}            │ UNUSED   │ support/steps/steps.ts:33       │
+└───────────────────────────────────────┴──────────┴─────────────────────────────────┘
+```
+
+### `usage-json`
+
+Does what the Usage Formatter does, but outputs JSON, which can be output to a file and then consumed by other tools.
+
+### Other officially-supported formatters
+
+* [@cucumber/pretty-formatter](https://www.npmjs.com/package/@cucumber/pretty-formatter) - prints the feature with inline results,  colours and custom themes.

docs/rerun.md

@@ -0,0 +1,58 @@
+# Rerun
+
+*Note: if you want a mechanism to retry flaky scenarios when they fail in CI, take a look at [Retry](./retry.md) instead.*
+
+If you're doing TDD, you might sometimes work like this:
+
+1. Run all the tests to see what fails
+2. Make changes to address failures
+3. Run just the tests that failed
+4. GOTO 2
+
+Rerun makes this kind of workflow convenient, so you don't have to hand-craft command line arguments to run just the tests that failed on the previous run.
+
+First, enable the `rerun` formatter every time you run cucumber-js:
+
+```shell
+--format rerun:@rerun.txt
+```
+
+You can do this via the CLI, or more likely via a [default profile](./profiles.md).
+
+The output file doesn't have to be named `@rerun.txt`, but its name _does_ have to start with `@` - this is how cucumber-js will later distinguish it from feature files. Either way, you should add this file to your `.gitignore` so you don't accidentally commit it. 
+
+Let's say we run cucumber-js for a Todo app we've made, and a few scenarios fail. Our rerun file's contents would look something like this:
+
+```
+features/adding.feature:3:19
+features/editing.feature:8
+```
+
+If this notation looks familiar, it's the same as for specifying scenarios by line on the CLI, and translates to:
+
+- In `adding.feature`, the scenarios on lines 3 and 19
+- In `editing.feature`, the scenario on line 8
+
+So, let's say we've looked at the failure on `editing.feature` and fixed our code. Now let's run cucumber-js again, but pointing at the rerun file:
+
+```shell
+$ cucumber-js @rerun.txt
+```
+
+cucumber-js will unpack this and just run those three failing scenarios accordingly. This time, it goes a bit better - our fix worked, and the rerun file now looks like this:
+
+```
+features/adding.feature:3:19
+```
+
+In other words, the one we fixed has passed and thus dropped off. We can repeat this cycle as many times as needed until eventually the rerun file will be empty because everything has passed. If you run cucumber-js pointing at an empty rerun file, no scenarios will be run.
+
+## Separator
+
+By default, entries in the rerun file are separated by newlines. This can be overwritten via a [format option](./formatters.md#options):
+
+```
+--format-options '{"rerun": {"separator": "<separator>"}}'
+```
+
+This is useful when one needs to rerun failed tests locally by copying a line from a CI log while using a space character as a separator. Note that the rerun file parser can only work with the default separator for now.

docs/retry.md

@@ -1,5 +1,7 @@
 # Retry
 
+*Note: if you want a mechanism to rerun just the failed scenarios when doing TDD, take a look at [Rerun](./rerun.md) instead.*
+
 If you have a flaky scenario (e.g. failing 10% of the time for some reason), you can use *Retry* to have Cucumber attempt it multiple times until either it passes or the maximum number of attempts is reached. You enable this via the `--retry <MAXIMUM_RETRIES>` CLI option, like this:
 
 ```shell

docs/snippets.md

@@ -0,0 +1,60 @@
+# Snippets
+
+Often in a BDD workflow, you'll write one or more steps in a feature file _before_ implementing the corresponding step definition(s). When you run cucumber-js and it finds these undefined steps, it will generate a Snippet for each one - a skeletal chunk of JavaScript with the correct expression and arguments that you can drop into your code to get started. [Formatters](./formatters.md) that are designed for the terminal will output the snippets.
+
+Let's say we're iterating on this feature:
+
+```gherkin
+Feature: Removing todos
+
+  Scenario: Add and then remove a todo from an empty list
+    Given an empty todo list
+    When I add the todo "buy some cheese"
+    And I remove the todo "buy some cheese"
+```
+
+The last step about removing todos isn't implemented yet - we just wrote that into the scenario, so now it's time to write the step definition. If we run cucumber-js, it will announce the undefined step and provide the Snippet in the output:
+
+```js
+Then('I remove the todo {string}', function (string) {
+  // Write code here that turns the phrase above into concrete actions
+  return 'pending';
+});
+```
+
+By default, the snippet uses the "synchronous" style. You can use the `snippetInterface` [format option](./formatters.md#options) to specify one of the styles that supports asynchronous steps:
+
+- "async-await" - Outputs an async function where you can use `await` - probably the best choice if you aren't sure.
+  Running with `--format-options '{"snippetInterface":"async-await"}'` yields:
+  ```js
+  Then('I remove the todo {string}', async function (string) {
+    // Write code here that turns the phrase above into concrete actions
+    return 'pending';
+  });
+  ```
+- "callback" - Outputs a plain function with a callback function as the final argument.
+  Running with `--format-options '{"snippetInterface":"callback"}'` yields:
+  ```js
+  Then('I remove the todo {string}', function (string, callback) {
+    // Write code here that turns the phrase above into concrete actions
+    callback(null, 'pending');
+  });
+  ```
+- "promise" - Outputs a plain function from which you should return a `Promise`.
+  Running with `--format-options '{"snippetInterface":"promise"}'` yields:
+  ```js
+  Then('I remove the todo {string}', function (string) {
+    // Write code here that turns the phrase above into concrete actions
+    return Promise.resolve('pending');
+  });
+  ```
+- "synchronous" - Outputs a plain function with no async pattern (see earlier example).
+
+## Options
+
+These [format options](./formatters.md#options) influence how the snippets are rendered by all formatters that output snippets:
+
+- `snippetInterface` - set to one of 'async-await', 'callback', 'promise', or 'synchronous' to alter the style of generated snippets for undefined steps
+- `snippetSyntax` - module name of path of the [custom snippet syntax](./custom_snippet_syntaxes.md) to be used
+
+

features/step_definition_snippets_custom_syntax.feature

@@ -45,7 +45,7 @@ Feature: step definition snippets custom syntax
       """
 
   Scenario Outline:
-    When I run cucumber-js with `--format-options '{"snippetInterface": "<INTERFACE>", "snippetSyntax": "coffeescript_syntax.js"}'`
+    When I run cucumber-js with `--format-options '{"snippetInterface": "<INTERFACE>", "snippetSyntax": "./coffeescript_syntax.js"}'`
     Then it fails
     And the output contains the text:
       """