Warnings-as-errors or similar

[msimberg]

Sometimes tests get excluded with a warning because of issues with the tests. Some examples we've encountered recently:

• Name conflicts (WARNING: skipping test file 'a.py': name conflict error: test 'Foo' from 'a.py' is already defined in 'b.py')
• Version mismatch (WARNING: skipping test file 'a.py': attribute error: module 'reframe' has no attribute 'xfail')
• Other test test issues (WARNING: skipping test file 'a.py': permission error: [Errno 13] Permission denied: '/a/b/c'; the test setup makes assumptions about the system)

Reframe warns about these which is great. However, in automated testing (e.g. CI) none of these are actually noticed without manually looking at logs because the tests are silently ignored instead of causing a failure.

As far as I could tell, there's no way to ask for this currently when running tests. Did I miss it or would it be possible to add such a mode (--warnings-as-errors)?

What I'm thinking of is perhaps not failing immediately on warnings (i.e. still run the tests that are discovered) but exit at the end with an error, even if all tests were otherwise successful.

Similarly, if no tests are found it'd be great to have a way to ask reframe to make that a failure to avoid silently having no tests run.

[vkarak]

@msimberg This request makes sense, but I guess that's more about warnings related to loading the tests.

This is duplicate of #2307 and related to #2306.

[msimberg]

Thanks @vkarak, I missed those other issues. I'm happy to close this in favour of either of those. In particular #2307 is more specific and exactly what I had in mind.

The second part about failing on no tests found is related, but a bit different. Should I open a separate issue for that?

[vkarak]

The second part about failing on no tests found is related, but a bit different. Should I open a separate issue for that?

I'd say so, as this will be likely a different option.

[msimberg]

I opened #3599.