Docs: add rule documentation consistency tests (eslint-community#137)

Adds tests to ensure that our rule docs have notices to indicate whether they are fixable and/or in the recommended config. These notices match the format used by ESLint itself. Example: https://eslint.org/docs/rules/no-extra-semi

Also adds tests for various other aspects of rule docs and that each rule has a documentation file and test file.

docs/rules/fixer-return.md

@@ -1,5 +1,7 @@
 # Enforces always return from a fixer function (fixer-return)
 
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
 In a fixable rule, missing return from a fixer function will not apply fixes.
 
 ## Rule Details

docs/rules/meta-property-ordering.md

@@ -1,6 +1,6 @@
 # enforce ordering of meta properties in rule source (meta-property-ordering)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 This rule enforces that meta properties of a rule are placed in a consistent order.
 

docs/rules/no-deprecated-context-methods.md

@@ -1,6 +1,6 @@
 # Disallows usage of deprecated methods on rule context objects (no-deprecated-context-methods)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 This rule disallows the use of deprecated methods on rule `context` objects.
 

docs/rules/no-deprecated-report-api.md

@@ -1,14 +1,16 @@
 # disallow use of the deprecated context.report() API (no-deprecated-report-api)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 ESLint has two APIs that rules can use to report problems. The [deprecated API](http://eslint.org/docs/developer-guide/working-with-rules-deprecated) accepts multiple arguments: `context.report(node, [loc], message)`. The ["new API"](http://eslint.org/docs/developer-guide/working-with-rules#contextreport) accepts a single argument: an object containing information about the reported problem. It is recommended that all rules use the new API.
 
 ## Rule Details
 
 This rule aims to disallow use of the deprecated `context.report(node, [loc], message)` API.
 
-The following patterns are considered warnings:
+Examples of **incorrect** code for this rule:
 
 ```js
 module.exports = {
@@ -19,7 +21,7 @@ module.exports = {
 
 ```
 
-The following patterns are not warnings:
+Examples of **correct** code for this rule:
 
 ```js
 module.exports = {

docs/rules/no-identical-tests.md

@@ -1,14 +1,16 @@
-# Disallow identical tests (no-identical-tests)
-
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
-
-When a rule has a lot of tests, it's sometimes difficult to tell if any tests are duplicates. This rule would warn if any test cases have the same properties.
-
-## Rule Details
-
-Examples of **incorrect** code for this rule:
-
-```js
+# Disallow identical tests (no-identical-tests)
+
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
+
+When a rule has a lot of tests, it's sometimes difficult to tell if any tests are duplicates. This rule would warn if any test cases have the same properties.
+
+## Rule Details
+
+Examples of **incorrect** code for this rule:
+
+```js
 /* eslint eslint-plugin/no-identical-tests: error */
 
 new RuleTester().run('foo', bar, {
@@ -17,12 +19,12 @@ new RuleTester().run('foo', bar, {
     { code: 'foo' },
   ],
   invalid: [],
-});
-```
-
-Examples of **correct** code for this rule:
-
-```js
+});
+```
+
+Examples of **correct** code for this rule:
+
+```js
 /* eslint eslint-plugin/no-identical-tests: error */
 
 new RuleTester().run('foo', bar, {
@@ -31,9 +33,9 @@ new RuleTester().run('foo', bar, {
     { code: 'bar' },
   ],
   invalid: [],
-});
-```
-
-## When Not To Use It
-
-If you want to allow identical tests, do not enable this rule.
+});
+```
+
+## When Not To Use It
+
+If you want to allow identical tests, do not enable this rule.

docs/rules/no-missing-placeholders.md

@@ -1,5 +1,7 @@
 # Disallow missing placeholders in rule report messages (no-missing-placeholders)
 
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
 Report messages in rules can have placeholders surrounded by curly brackets.
 
 ```js

docs/rules/no-unused-placeholders.md

@@ -1,5 +1,7 @@
 # Disallow unused placeholders in rule report messages (no-unused-placeholders)
 
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
 This rule aims to disallow unused placeholders in rule report messages.
 
 ## Rule Details

docs/rules/no-useless-token-range.md

@@ -1,5 +1,9 @@
 # Disallow unnecessary calls to sourceCode.getFirstToken and sourceCode.getLastToken (no-useless-token-range)
 
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
+
 AST nodes always start and end with tokens. As a result, the start index of the first token in a node is the same as the start index of the node itself, and the end index of the last token in a node is the same as the end index of the node itself. Using code like `sourceCode.getFirstToken(node).range[0]` unnecessarily hurts the performance of your rule, and makes your code less readable.
 
 ## Rule Details

docs/rules/prefer-object-rule.md

@@ -1,6 +1,6 @@
 # Disallow rule exports where the export is a function. (prefer-object-rule)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 ## Rule Details
 

docs/rules/prefer-output-null.md

@@ -1,6 +1,6 @@
 # Disallows invalid RuleTester test cases with the output the same as the code. (prefer-output-null)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 Instead of repeating the test case `code`, using `output: null` is more concise and makes it easier to distinguish whether a test case provides an autofix.
 

docs/rules/report-message-format.md

@@ -25,7 +25,7 @@ For example, in order to mandate that all report messages begin with a capital l
 
 Note that since this rule uses static analysis and does not actually run your code, it will attempt to match report messages *before* placeholders are inserted.
 
-The following patterns are considered warnings:
+Examples of **incorrect** code for this rule:
 
 ```js
 /* eslint eslint-plugin/report-message-format: ["error", "^[A-Z].*\\.$"] */
@@ -42,7 +42,7 @@ module.exports = {
 };
 ```
 
-The following patterns are not warnings:
+Examples of **correct** code for this rule:
 
 ```js
 module.exports = {

docs/rules/require-meta-docs-url.md

@@ -1,5 +1,7 @@
 # require rules to implement a meta.docs.url property (require-meta-docs-url)
 
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
+
 A rule can store the URL to its documentation page in `meta.docs.url`. This enables integration tools / IDEs / editors to conveniently provide the link to developers so that they can better understand the rule.
 
 ## Rule Details
@@ -20,7 +22,7 @@ This rule has an option.
 
 If you set the `pattern` option, this rule adds `meta.docs.url` property automatically when you execute `eslint --fix` command.
 
-The following patterns are considered warnings:
+Examples of **incorrect** code for this rule:
 
 ```js
 
@@ -63,7 +65,7 @@ module.exports = {
 
 ```
 
-The following patterns are not warnings:
+Examples of **correct** code for this rule:
 
 ```js
 

docs/rules/require-meta-fixable.md

@@ -1,12 +1,14 @@
 # require rules to implement a meta.fixable property (require-meta-fixable)
 
+✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.
+
 A fixable ESLint rule must have a valid `meta.fixable` property. A rule reports a problem with a `fix()` function but does not export a `meta.fixable` property is likely to cause an unexpected error.
 
 ## Rule Details
 
 This rule aims to require ESLint rules to have a `meta.fixable` property if necessary.
 
-The following patterns are considered warnings:
+Examples of **incorrect** code for this rule:
 
 ```js
 /* eslint eslint-plugin/require-meta-fixable: "error" */
@@ -56,7 +58,7 @@ module.exports = { create (context) {
 } };
 ```
 
-The following patterns are not warnings:
+Examples of **correct** code for this rule:
 
 ```js
 /* eslint eslint-plugin/require-meta-fixable: "error" */

docs/rules/require-meta-has-suggestions.md

@@ -8,7 +8,7 @@ Likewise, rules that do not report suggestions should not enable the `meta.hasSu
 
 This rule aims to require ESLint rules to have a `meta.hasSuggestions` property if necessary.
 
-The following patterns are considered warnings:
+Examples of **incorrect** code for this rule:
 
 ```js
 /* eslint eslint-plugin/require-meta-has-suggestions: "error" */
@@ -44,7 +44,7 @@ module.exports = {
 };
 ```
 
-The following patterns are not warnings:
+Examples of **correct** code for this rule:
 
 ```js
 /* eslint eslint-plugin/require-meta-has-suggestions: "error" */

docs/rules/require-meta-schema.md

@@ -1,5 +1,7 @@
 # require rules to implement a meta.schema property (require-meta-schema)
 
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
+
 Defining a schema for each rule allows eslint to validate that configuration options are passed correctly. Even when there are no options for a rule, a schema should still be defined (as an empty array) so that eslint can validate that no data is passed to the rule.
 
 ## Rule Details

docs/rules/test-case-property-ordering.md

@@ -1,6 +1,6 @@
 # enforce ordering of keys in test cases (test-case-property-ordering)
 
-(fixable) The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) automatically fixes problems reported by this rule.
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
 
 This rule enforces that the properties of RuleTester test cases are arranged in a consistent order.
 

docs/rules/test-case-shorthand-strings.md

@@ -1,5 +1,7 @@
 # Enforce consistent usage of shorthand strings for test cases with no options (test-case-shorthand-strings)
 
+⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.
+
 When writing valid test cases for rules with `RuleTester`, one can optionally include a string as a test case instead of an object, if the the test case does not use any options.
 
 ```js

tests/lib/rule-setup.js

@@ -0,0 +1,112 @@
+'use strict';
+
+const { readdirSync, readFileSync } = require('fs');
+const path = require('path');
+const assert = require('chai').assert;
+const plugin = require('../..');
+
+const RULE_NAMES = Object.keys(plugin.rules);
+const RULE_NAMES_RECOMMENDED = new Set(Object.keys(plugin.configs.recommended.rules));
+
+const MESSAGES = {
+  fixable:
+    '⚒️ The `--fix` option on the [command line](https://eslint.org/docs/user-guide/command-line-interface#-fix) can automatically fix some of the problems reported by this rule.',
+  configRecommended:
+    '✔️ The `"extends": "plugin:eslint-plugin/recommended"` property in a configuration file enables this rule.',
+};
+
+describe('rule setup is correct', () => {
+  it('should have a list of exported rules and rules directory that match', () => {
+    const filePath = path.join(__dirname, '..', 'lib', 'rules');
+    const files = readdirSync(filePath);
+
+    assert.deepStrictEqual(
+      RULE_NAMES,
+      files
+        .filter(file => !file.startsWith('.'))
+        .map(file => file.replace('.js', ''))
+    );
+  });
+
+  it('should have tests for all rules', () => {
+    const filePath = path.join(__dirname, 'rules');
+    const files = readdirSync(filePath);
+
+    assert.deepStrictEqual(
+      RULE_NAMES,
+      files
+        .filter(file => !file.startsWith('.'))
+        .map(file => file.replace('.js', ''))
+    );
+  });
+
+  it('should have documentation for all rules', () => {
+    const filePath = path.join(__dirname, '..', '..', 'docs', 'rules');
+    const files = readdirSync(filePath);
+
+    assert.deepStrictEqual(
+      RULE_NAMES,
+      files
+        .filter(file => !file.startsWith('.'))
+        .map(file => file.replace('.md', ''))
+    );
+  });
+
+  describe('rule documentation files', () => {
+    for (const ruleName of RULE_NAMES) {
+      const rule = plugin.rules[ruleName];
+      const filePath = path.join(
+        __dirname,
+        '..',
+        '..',
+        'docs',
+        'rules',
+        `${ruleName}.md`
+      );
+      const fileContents = readFileSync(filePath, 'utf8');
+      const lines = fileContents.split('\n');
+
+      describe(ruleName, () => {
+        it('should have the right contents (title, notices, etc)', () => {
+          // Title
+          assert.ok(lines[0].endsWith(`(${ruleName})`), 'first line ends with rule name');
+          assert.strictEqual(lines[1], '', 'second line is blank');
+
+          // Rule Details
+          assert.ok(fileContents.includes('## Rule Details'), 'includes "## Rule Details" header');
+
+          // Examples
+          assert.ok(fileContents.includes('Examples of **incorrect** code for this rule'), 'includes incorrect examples');
+          assert.ok(fileContents.includes('Examples of **correct** code for this rule'), 'includes correct examples');
+
+          // Decide which notices should be shown at the top of the doc.
+          const expectedNotices = [];
+          const unexpectedNotices = [];
+          if (RULE_NAMES_RECOMMENDED.has('eslint-plugin/' + ruleName)) {
+            expectedNotices.push('configRecommended');
+          } else {
+            unexpectedNotices.push('configRecommended');
+          }
+          if (rule.meta.fixable) {
+            expectedNotices.push('fixable');
+          } else {
+            unexpectedNotices.push('fixable');
+          }
+
+          // Ensure that expected notices are present in the correct order.
+          let currentLineNumber = 1;
+          for (const expectedNotice of expectedNotices) {
+            assert.strictEqual(lines[currentLineNumber], '');
+            assert.strictEqual(lines[currentLineNumber + 1], MESSAGES[expectedNotice]);
+            currentLineNumber += 2;
+          }
+
+          // Ensure that unexpected notices are not present.
+          for (const unexpectedNotice of unexpectedNotices) {
+            assert.ok(!fileContents.includes(MESSAGES[unexpectedNotice]), 'does not include notice: ' + MESSAGES[unexpectedNotice]);
+          }
+        });
+      });
+    }
+  });
+});