`doctest` should not be `opt-in`

[loynoir]

BUG

compile_fail should not be ignored on test function.

Actual

compile_fail is ignored on test function.

$ cargo test && echo OK
...
OK

pub fn add(left: usize, right: usize) -> usize {
    left + right
}

#[cfg(test)]
mod tests {
    use super::*;

    /// ```compile_fail
    /// let mut x = 5;
    /// x += 2;
    /// ```
    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

Expected

compile_fail should not be ignored on test function.

Same as on non-test function.

$ cargo test
...
Test compiled successfully, but it's marked `compile_fail`.
...

/// ```compile_fail
/// let mut x = 5;
/// x += 2;
/// ```
pub fn add(left: usize, right: usize) -> usize {
    left + right
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

Version

• rustc 1.73.0 (cc66ad4 2023-10-03)

• cargo 1.73.0 (9c4383fb5 2023-08-26)

[zachs18]

Rustdoc (currently) only collects doctests on items that exist under --cfg=doctest, not --cfg=test.
https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html?highlight=doctest#include-items-only-when-collecting-doctests

pub fn add(left: usize, right: usize) -> usize {
    left + right
}

#[cfg(any(test, doctest))] // `mod tests` should exists under `--cfg=test` or `--cfg=doctest`
mod tests {
    use super::*;

    /// ```compile_fail
    /// let mut x = 5;
    /// x += 2;
    /// ```
    #[cfg_attr(test, test)] // `#[test]` removes the item without `--cfg=test`, so only apply it if `--cfg=test` is active
    fn it_works() {
        let result = add(2, 2);
        assert_eq!(result, 4);
    }
}

$ cargo test
...
Test compiled successfully, but it's marked `compile_fail`.
...

[loynoir]

This is so bad for situation like below.

pub fn add(left: usize, right: usize) -> usize {
    left + right
}

// #[cfg(any(test, doctest))]
#[cfg(test)]
mod tests {
    use super::*;

    // use `foobar` `generate_test`
    // #[macro_export]
    macro_rules! generate_test {
        () => {
            #[doc = "```compile_fail\nlet mut x = 5; x += 2\n```"]
            // #[cfg_attr(test, test)]
            #[test]
            fn _it_works() {}
        };
    }

    generate_test!();
}

Generated compile_fail are silently ignored.

And nobody will check for

• Are there any tests are silently ignored?

Unless

1. one MUST know tests may ignored

2. one MUST remeber to check

3. one MUST also remember to check two packages code.

4. one MUST count test missing, almost impossible for human eyeballs when test grows as codegen possible.

So, I still consider as a design-bug, which depends on four assumptions, conflicts with

Empowering everyone to build reliable and efficient software.

In short, I think

• test, in this example, doctest, should not be opt-in

[loynoir]

Any clue how to make it also work for tests/xxx.rs ? @zachs18

tests/reproduce.rs

#[cfg(any(test, doctest))]
mod tests {
    /// ```compile_fail
    /// ```
    #[cfg_attr(test, test)]
    fn it_works() {}
}

$ cargo test >/dev/null 2>/dev/null && echo OK
OK

[zachs18]

No; doctests are only run in the crate proper, not in integration tests (tests/xxx.rs). In my understanding, doctests are intended as API documentation examples first, and tests second, and it doesn't make sense to put API documentation in integration tests (since they are not part of the API of the library and do not have documentation generated).

See https://doc.rust-lang.org/rustdoc/write-documentation/documentation-tests.html for more information about doctests.

---

Sidenote: It looks like you want a compile_fail option for unit or integration tests. While I agree that could be useful, I don't think forcing that functionality into rustdoc is a good design.

You may want to look into https://github.com/Manishearth/compiletest-rs also.