Repository navigation
Make xpass a failure again #11467
Description
Activity
This means changing the default of the
strictparameter toTrue?That's would be the starting point, In the long run it should go away
Hi, How we can test this scenario, I can't see xpass as a marker in pytest.
@TanyaAgarwal28 xpass is a test outcome for when a xfail marked test passed "unexpectedly"
Yeah Got it so what we are facing issue here?
Isin't a feature that if xfail testcase is passed then it will came as xpassed in outcome?
import pytest@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6
@TanyaAgarwal28 the problem is that its a misfeature - it hides important change for a non-flaky testsuite - fixing a previously failing test by accident
when non-strict xfail was introduced, it was a major mistake not to use a new mark
So basically we have to revert this change and if xfail testcase is by mistakly fixed then the outcome should be passed??
And for non_strict_xfail we should have new mark??Overview of 4 scenarios for xfail and non_strict_xfail
xfail testcase which is expectedly fail:
import pytest@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 7 # This test is marked as expected fail
xfail testcase which is passed:
import pytest@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail
non_strict_xfail testcase is passed:
import pytest@pytest.mark.non_strict_xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail
So basically we have to revert this change and if xfail testcase is by mistakly fixed then the outcome should be passed??
And for non_strict_xfail we should have new mark??IMHO what should be done:
pytest.mark.xfailhas astrictparameter, which defaults toFalsetoday. We want to issue a warning that this default will change toTruein the future, but only if thestrictflag has not been passed explicitly.- A few major releases later, we change the default to
Trueand remove the warning.
Is this what you have in mind @RonnyPfannschmidt ?
Correct, once the default is switched I'd like to ensure xpass is either a warning or a failure in any case
okay Got it
Thanks- added 6 commits that reference this issue
on Oct 12, 2023 We now have strict mode which enables
strict_xfail. Maybe that's enough to close this issue?this issue is about making it default to true
What's the problem this feature will solve?
Some years ago we accepted a change which made xpass not a failure
I recently hit multiple cases where it was missed that a test was fixed
That's not an acceptable default
Describe the solution you'd like
Alternative Solutions
Additional context