Clippy Book Chapter Updates Reborn: Writing tests

book/src/SUMMARY.md

@@ -14,6 +14,7 @@
     - [Basics](development/basics.md)
     - [Adding Lints](development/adding_lints.md)
     - [Defining Lints](development/defining_lints.md)
+    - [Writing tests](development/writing_tests.md)
     - [Lint Passes](development/lint_passes.md)
     - [Type Checking](development/type_checking.md)
     - [Method Checking](development/method_checking.md)

book/src/development/writing_tests.md

@@ -0,0 +1,223 @@
+# Testing
+
+Developing lints for Clippy is a Test-Driven Development (TDD) process because
+our first task before implementing any logic for a new lint is to write some test cases.
+
+## Develop Lints with Tests
+
+When we develop Clippy, we enter a complex and chaotic realm full of
+programmatic issues, stylistic errors, illogical code and non-adherence to convention.
+Tests are the first layer of order we can leverage to define when and where
+we want a new lint to trigger or not.
+
+Moreover, writing tests first help Clippy developers to find a balance for
+the first iteration of and further enhancements for a lint.
+With test cases on our side, we will not have to worry about over-engineering
+a lint on its first version nor missing out some obvious edge cases of the lint.
+This approach empowers us to iteratively enhance each lint.
+
+## Clippy UI Tests
+
+We use **UI tests** for testing in Clippy. These UI tests check that the output
+of Clippy is exactly as we expect it to be. Each test is just a plain Rust file
+that contains the code we want to check.
+
+The output of Clippy is compared against a `.stderr` file. Note that you don't
+have to create this file yourself. We'll get to generating the `.stderr` files
+with the command [`cargo bless`](#cargo-bless) (seen later on).
+
+### Write Test Cases
+
+Let us now think about some tests for our imaginary `foo_functions` lint. We
+start by opening the test file `tests/ui/foo_functions.rs` that was created by
+`cargo dev new_lint`.
+
+Update the file with some examples to get started:
+
+```rust
+#![warn(clippy::foo_functions)] // < Add this, so the lint is guaranteed to be enabled in this file
+
+// Impl methods
+struct A;
+impl A {
+    pub fn fo(&self) {}
+    pub fn foo(&self) {} //~ ERROR: function called "foo"
+    pub fn food(&self) {}
+}
+
+// Default trait methods
+trait B {
+    fn fo(&self) {}
+    fn foo(&self) {} //~ ERROR: function called "foo"
+    fn food(&self) {}
+}
+
+// Plain functions
+fn fo() {}
+fn foo() {} //~ ERROR: function called "foo"
+fn food() {}
+
+fn main() {
+    // We also don't want to lint method calls
+    foo();
+    let a = A;
+    a.foo();
+}
+```
+
+Without actual lint logic to emit the lint when we see a `foo` function name,
+this test will just pass, because no lint will be emitted. However, we can now
+run the test with the following command:
+
+```sh
+$ TESTNAME=foo_functions cargo uitest
+```
+
+Clippy will compile and it will conclude with an `ok` for the tests:
+
+```
+...Clippy warnings and test outputs...
+test compile_test ... ok
+test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.48s
+```
+
+This is normal. After all, we wrote a bunch of Rust code but we haven't really
+implemented any logic for Clippy to detect `foo` functions and emit a lint.
+
+As we gradually implement our lint logic, we will keep running this UI test command.
+Clippy will begin outputting information that allows us to check if the output is
+turning into what we want it to be.
+
+### Example output
+
+As our `foo_functions` lint is tested, the output would look something like this:
+
+```
+failures:
+
+---- compile_test stdout ----
+normalized stderr:
+error: function called "foo"
+  --> $DIR/foo_functions.rs:6:12
+   |
+LL |     pub fn foo(&self) {}
+   |            ^^^
+   |
+   = note: `-D clippy::foo-functions` implied by `-D warnings`
+
+error: function called "foo"
+  --> $DIR/foo_functions.rs:13:8
+   |
+LL |     fn foo(&self) {}
+   |        ^^^
+
+error: function called "foo"
+  --> $DIR/foo_functions.rs:19:4
+   |
+LL | fn foo() {}
+   |    ^^^
+
+error: aborting due to 3 previous errors
+```
+
+Note the *failures* label at the top of the fragment, we'll get rid of it
+(saving this output) in the next section.
+
+> _Note:_ You can run multiple test files by specifying a comma separated list:
+> `TESTNAME=foo_functions,bar_methods,baz_structs`.
+
+### `cargo bless`
+
+Once we are satisfied with the output, we need to run this command to
+generate or update the `.stderr` file for our lint:
+
+```sh
+$ TESTNAME=foo_functions cargo uibless
+```
+
+This writes the emitted lint suggestions and fixes to the `.stderr` file, with
+the reason for the lint, suggested fixes, and line numbers, etc.
+
+Running `TESTNAME=foo_functions cargo uitest` should pass then. When we commit
+our lint, we need to commit the generated `.stderr` files, too.
+
+In general, you should only commit files changed by `cargo bless` for the
+specific lint you are creating/editing.
+
+> _Note:_ If the generated `.stderr`, and `.fixed` files are empty,
+> they should be removed.
+
+## `toml` Tests
+
+Some lints can be configured through a `clippy.toml` file. Those configuration
+values are tested in `tests/ui-toml`.
+
+To add a new test there, create a new directory and add the files:
+
+- `clippy.toml`: Put here the configuration value you want to test.
+- `lint_name.rs`: A test file where you put the testing code, that should see a
+  different lint behavior according to the configuration set in the
+  `clippy.toml` file.
+
+The potential `.stderr` and `.fixed` files can again be generated with `cargo
+bless`.
+
+## Cargo Lints
+
+The process of testing is different for Cargo lints in that now we are
+interested in the `Cargo.toml` manifest file. In this case, we also need a
+minimal crate associated with that manifest. Those tests are generated in
+`tests/ui-cargo`.
+
+Imagine we have a new example lint that is named `foo_categories`, we can run:
+
+```sh
+$ cargo dev new_lint --name=foo_categories --pass=late --category=cargo
+```
+
+After running `cargo dev new_lint` we will find by default two new crates,
+each with its manifest file:
+
+* `tests/ui-cargo/foo_categories/fail/Cargo.toml`: this file should cause the
+  new lint to raise an error.
+* `tests/ui-cargo/foo_categories/pass/Cargo.toml`: this file should not trigger
+  the lint.
+
+If you need more cases, you can copy one of those crates (under
+`foo_categories`) and rename it.
+
+The process of generating the `.stderr` file is the same as for other lints
+and prepending the `TESTNAME` variable to `cargo uitest` works for Cargo lints too.
+
+## Rustfix Tests
+
+If the lint you are working on is making use of structured suggestions,
+[`rustfix`] will apply the suggestions from the lint to the test file code and
+compare that to the contents of a `.fixed` file.
+
+Structured suggestions tell a user how to fix or re-write certain code that has
+been linted with [`span_lint_and_sugg`].
+
+Should `span_lint_and_sugg` be used to generate a suggestion, but not all
+suggestions lead to valid code, you can use the `//@no-rustfix` comment on top
+of the test file, to not run `rustfix` on that file.
+
+We'll talk about suggestions more in depth in a later chapter.
+<!-- FIXME: (blyxyas) Link to "Emitting lints" when that gets merged -->
+
+Use `cargo bless` to automatically generate the `.fixed` file after running
+the tests.
+
+[`rustfix`]: https://github.com/rust-lang/rustfix
+[`span_lint_and_sugg`]: https://doc.rust-lang.org/beta/nightly-rustc/clippy_utils/diagnostics/fn.span_lint_and_sugg.html
+
+## Testing Manually
+
+Manually testing against an example file can be useful if you have added some
+`println!`s and the test suite output becomes unreadable.
+
+To try Clippy with your local modifications, run from the working copy root.
+
+```sh
+$ cargo dev lint input.rs
+```