Option of treating XPASS as failure?

[rabbbit]

// enhancement idea
// this might already exist, tried to find it, failed

tldr; I want my xfail'ed tests to cause test suite to fail if they succeed

Hey,

Often I find myself tests that given an two set outputs, sometimes return a value and sometimes raise specific exceptions. I'm currently splitting this in two separate tests, one for "good" values, one for exceptions, and use pytest.raises.

Inspired by docs on parametrizing tests I thought I could use mark.xfail to group these tests in one logical thing.
However, I don't like XPASS's - I'd like to be able to say that these tests have to fail with specific exceptions.
It would basically be a shorthand for splitting a test into two and using pytest.raises.

Ideally, it would work as an extra argument to xfail (strict=True?), rather than command line option, because I cannot set the value globally.

Basically, I'd like this to fail:

import pytest                                                                                                                                                                     

def complicated_validation_function(input_data):                                                                                                                                  
    if input_data in ('should_raise', 'should_raise_too_but_i_made_a_typo'):                                                                                                      
        raise ValueError                                                                                                                                                          
    return True                                                                                                                                                                   

@pytest.mark.parametrize('input_data', [                                                                                                                                          
    (0),                                                                                                                                                                          
    pytest.mark.xfail('should_raise'),                                                                                                                                            
    pytest.mark.xfail('should_raise_too'),                                                                                                                                        
])                                                                                                                                                                                
def test_validation(input_data):                                                                                                                                                  
    assert complicated_validation_function(input_data) is True 

Does this already exist somewhere?
If not, does this sound like a plausible idea?

edit: I dunno how to mark this as proposal per contributing guidelines

[The-Compiler]

See #1299 (and #814 for lots of background and discussion).

I think there's a general consensus for adding a strict parameter to xfail as you said (and maybe a config option to default to that?), it's just that nobody got to it yet. Want to submit a PR? That'd be very appreciated!

[rabbbit]

Ah, cool, I'm not insane then :)

PR - I'll try to take a look next week, but no promises, I suspect this might be beyond me :)

// what would be really cool would be an option to pass a fixture into .raises=my_fixture, and make it actually compare it but .. lets keep that for later :)

I'll close this one, #1299 is good enough.

[RonnyPfannschmidt]

the preference for configuration is good, this one it is

[nicoddemus]

👍

[nicoddemus]

It would be nice to introduce this in 2.9... 😁

[rabbbit]

A note that if we only enable this in pytest.ini (which I believe can only be used 1 per project?) I might not be able to use it -> big repo, cannot change default settings without fixing other peoples tests.

[nicoddemus]

@rabbbit thanks for the clarification.

How about this then:

• pytest.mark.xfail will accept a a strict parameter, which controls if XPASSes fails a test suite. If not given, the value of xfail_strict ini option is used.
• xfail_strict ini option: default value for the strict parameter of pytest.mark.xfail. In the initial release (say 2.9) this will default to False for backward compatibility, but we will warn users in the release announcement that in the next release (2.10) the default will change to True.

Once this is released, you should add xfail_strict=false to your pytest.ini, so in future pytest releases the current behavior is preserved. You can then use pytest.mark.xfail(strict=True) in particular tests.

Does that cover your needs?

[rabbbit]

Sounds awesome :)

Having an option of setting this per (sub)-directory would also work, but I imagine is out of scope here.