Add ability to ignore tests at runtime.

For medium–large scale projects, or projects that have a lot of integrations with third party services. It would nice to be able to ignore tests based on the environment the tests are being run in. One example would be to run tests if an environment variable is present.

You can workaround this with something like the following, however the output of the test is ok and not ignored, misleading the user that the test was successful.

#[test]
fn foo() {
    if env::var("FOO").is_none() { return; }
    let foo = env::var("FOO").unwrap();
}

I would personally rather leave this to custom test frameworks; I think libtest should try to be minimal.

It would also be good to spec this out a bit more. I imagine maybe a proc macro would be enough (i.e., inserting the env) and allowing #[test] to return something like enum TestResult { Passed, Failed, Ignored }.

This is a pain point which is encountered in uutils/coreutils#1041 (comment)

This crate helps you ignore test case when the environment variable is absent.
https://crates.io/crates/test-with

https://github.com/yanganto/test-with/blob/6ac0336a5f47558a467bd245427a540fdad6f46e/examples/env.rs#L3-L13

#[cfg(test)]
mod tests {
    #[test_with::env(PWD)]
    fn test_works() {
        assert!(true);
    }
    #[test_with::env(NOTHING)]
    fn test_ignored() {
        panic!("should be ignored")
    }
}

See this pre-RFC thread on IRLO.

tests: ignore env var dependent tests at runtime (#12805)

See this pre-RFC thread on IRLO.

That thread has some interesting ideas, unfortunately the discussion has been closed for a long time.

I guess the simplest approach would be a well-known marker text in the panic message that the test runner would recognise as an ignored test, instead of a failing one (say, the panic message should be exactly IGNORED TEST). This could be done without introducing any significant API changes. Well, maybe that's too simple 😅

Anyway... it's definitely a needed feature. See ya! 👋

I guess the simplest approach would be a well-known marker text in the panic message that the test runner would recognise as an ignored test

I really don't think such "spooky action at a distance" is wise.

the panic message should be exactly IGNORED TEST)

I also really want a reason for the skip to be communicated somewhere.

I really don't think such "spooky action at a distance" is wise.

This design decision was already made for build.rs, where printing special strings like println!("cargo::rerun-if-changed=src/hello.c"); affects something to cargo.

I think something similar, a-la panic!("cargo::skip-test=This device doesn't support feature XYZ"), would make a lot of sense.

@arilotter tests aren't run by cargo.

Also, integration tests run as separate processes. All you get are an exit code and stdout/stderr.

Those are valid points - I more wanted to mention that Rust has taken a point of view that magic stdout is acceptable.

I think that in the context of the build that is more reasonable (though I have an issue or two about bugs in cargo::rerun-if-changed) as the language doesn't really have a concept of the build graph at compile time (though I would like for a better mechanism for proc_macros to do the reporting…similar bugs creep around there too IIRC). The language can however reify "I cannot reliably perform this test for reason X". If you have some subroutine that determines I cannot run via magic output, there is no opportunity for higher-level logic to say "ok, but I can do this instead" (e.g., say the test first determines that some quota is too low; if you're running as root (as most Linux containerized CI does), it may have permission to attempt to raise that limit and retry). For the "rerun-if-changed", there is no reasonable response to "no, you don't depend on that file" to revoke (you've already, presumably, done the action that makes said file a dependency of the build).

The test-with crate already provide runtime ignoring feature in an example.
https://github.com/yanganto/test-with?tab=readme-ov-file#runtime

Following are the example file.

/// `test.rs` under `/examples` folder
/// `cargo run --example test` to run the test

test_with::runner!(my_mod);

#[test_with::module]
mod my_mod {
    #[test_with::runtime_env(PWD)]
    fn env_test_works() {
        // The test case only run when PWD environment variable is set
        assert!(true);
    }
}

The panic prefix message for ignoring test is rt-ignored: .
Using test-with you can easily to handle this, also you can easy to set up the mock environment as you need.

Navigation Menu

Repository navigation

Add ability to ignore tests at runtime. #68007

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add ability to ignore tests at runtime. #68007

Description

Activity

Mark-Simulacrum commented on Jan 8, 2020

ArniDagur commented on Apr 17, 2021

yanganto commented on Dec 31, 2021

mathstuf commented on Jan 3, 2022

wiktor-k commented on Jul 7, 2025

mathstuf commented on Jul 8, 2025

ethernet8023 commented on Aug 7, 2025

jhpratt commented on Aug 7, 2025

mathstuf commented on Aug 8, 2025

ethernet8023 commented on Aug 8, 2025

mathstuf commented on Aug 8, 2025

yanganto commented on Aug 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions