Skip to content

List all non-passing tests in test summaries #1740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 15, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
List all non-passing tests in test summaries
This changes the default in pyproject.toml so that pytest lists
a line for each non-passing test at the end of a run, showing the
test name and, where available, condensed information about the
status, such as the "reason" argument for an xfailing or skipped
test.

Previously only failed and errored tests were listed in the
summary. Now skipped, xfailed, and xpassed tests are listed too.
The benefit is in keeping track of the status of tests. Although
showing the full failure output with stack trace and relevant code
under test would be too distracting for tests marked xfail, it is
valuable to not merely run those tests but be able to see a line
showing their names and statuses. Likewise, a number of tests are
currently marked skipped, and while some of them are skipped on a
particular platform because they don't make sense to run on that
platform, a number of others are skipped by raising SkipTest in
response to a failure condition on Windows. (Those consist mostly
of the tests skipped as a result of code discussed in #790.)

This also has the more specific benefit of making it easier to mark
tests as xfail in order to add CI jobs for native Windows, and more
importantly to allow information about their status to later be
used to understand and fix bugs on Windows.)
  • Loading branch information
EliahKagan committed Nov 14, 2023
commit 359116b3b9b7011eb7893fa584a43522f1b9bb57
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.pytest.ini_options]
addopts = "--cov=git --cov-report=term --disable-warnings"
addopts = "--cov=git --cov-report=term --disable-warnings -ra"
filterwarnings = "ignore::DeprecationWarning"
python_files = "test_*.py"
tmp_path_retention_policy = "failed"
Expand All @@ -14,8 +14,8 @@ testpaths = "test" # Space separated list of paths from root e.g test tests doc
# --cov-report html:path # html file at path
# --maxfail # number of errors before giving up
# -disable-warnings # Disable pytest warnings (not codebase warnings)
# -rf # increased reporting of failures
# -rE # increased reporting of errors
# -rfE # default test summary: list fail and error
# -ra # test summary: list all non-passing (fail, error, skip, xfail, xpass)
# --ignore-glob=**/gitdb/* # ignore glob paths
# filterwarnings ignore::WarningType # ignores those warnings

Expand Down