[RFC] Use `-rfE` by default? (reportchars)

[blueyed]

I keep adding this to addopts for projects since it is very useful and nice to have.

Should this be enabled by default?

Unfortunately there is no way to disable this then (would require something like -r= or -r- then, see #5066).

[nicoddemus]

Yeah I also find it extremely useful to have a summary of failures at the end of the test session...

I would be 👍 if we find a way to disable this somehow, like you mention.

[bluetech]

For reference for people who are not familiar with the option (like me):

  -r chars              show extra test summary info as specified by chars:
                        (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed,
                        (p)assed, (P)assed with output, (a)ll except passed
                        (p/P), or (A)ll. (w)arnings are enabled by default (see
                        --disable-warnings).

And the output looks like this added at the end:

================================================================================== short test summary info ===================================================================================
FAILED project/tests/test_quux.py::test_permissions - project.errors.NotAllowed
FAILED project/tests/test_foo.py::test_scenario - project.models.Bar.DoesNotExist: Bar matching query does not exist.

---

My opinion is that it will be nice to add by default. However, according to the help, -ra also includes expected things like X passed and skipped. IMO would be better not to show those by default.

[nicoddemus]

How about this:

• Use fEsxX by default (or alternatively fE as commented in [RFC] Use -rfE by default? (reportchars) #6454 (comment)).
• Include a new N option, meaning None

So to disable the summary completely user can enter -rN.

[bluetech]

@nicoddemus I just think that showing skipped (s) and xfail (x) would be mildly annoying, because it does not indicate a problem, just something intentional the user did, and they will be shown even for successful sessions, which I think are better to keep without a summary.

[nicoddemus]

Fair enough, updated by comment adding that option for consideration. 👍

[bluetech]

(I think X should be included because it does indicate a failure - but maybe I'm confused, because pytest doesn't consider it a failure, just "yellow". I guess the intended use for xfail is more "TODO make it pass" than "ensure this test fails"?).

[blueyed]

Yes, X (XPASS) is unexpected, but people appear to even use it for flaky things.
It was even green until recently (#6425).
Maybe it should be red when it causes failure XPASS(strict) (another issue though).

[bluetech]

Just one more comment, the skipping docs say this:

pytest counts and lists skip and xfail tests separately. Detailed information about skipped/xfailed tests is not shown by default to avoid cluttering the output. You can use the -r option to see details corresponding to the “short” letters shown in the test progress:

pytest -rxXs  # show extra info on xfailed, xpassed, and skipped tests

[nicoddemus]

Well if the objecive is have a "good default" and avoid cluttering then I think using as default fE is the winner?

[asottile]

this would mean one would need to scroll up even further to get to the assertions -- I really dislike how much noise there is before the assertion message already (output, logs, warnings, etc.) and I think this would make it worse?

[nicoddemus]

I don't think it would make it "much" worse actually; if you have one or two failures you will see 2 (ok 3 with header) extra lines. If you have 100s of failures you will see a few 100s of extra lines at the end, but I guess in that case you will have 1000s of lines of traceback to scroll, so the extra summary lines will dwarf the amount of scrolling you will need to do anyway.