Improve stepwise to not forget failed tests #13122

nicoddemus · 2025-01-10T15:37:47Z

Now --stepwise will remember the last failed test, even if the previous pytest invocations did not receive --stepwise.

Previously it would always clear the cache if not active, which hinders certain interactive workflows, which is the prime use case for the flag.

src/_pytest/stepwise.py

The-Compiler · 2025-01-10T22:36:29Z

while handy to have it remembered within a "working session", would it be unexpected to have it remembered across such sessions?

I'm worried about this too. Imagine someone:

Reads about --stepwise being a thing and gives it a try
Uses pytest normally for the next couple of months
Now actually wants to use --stepwise after a new Python version came out and broke something

If you've long forgotten about your previous session, that'll be quite surprising...

I don't really have a solution unfortunately. A couple of ideas:

Add an argument (or --stepwise=keep or whatever) to opt into keeping it, either when originally using --stepwise, or in the subsequent call without it
Add some sort of option for a stepwise max age (say, a week by default?) after which it's discarded
Invalidate the cache as soon as new tests are added (not sure what happens currently)
Keep the behavior, but make it clearer in the UI: Store a timestamp, and instead of stepwise: skipping 1 already passed items., display something like stepwise: skipping 1 already passed items (cache from 2025-01-10, use --stepwise-reset to discard)

I don't really like the first three as they all come with their own downsides, but maybe the fourth one would at least partially help?

testing/test_cacheprovider.py

nicoddemus · 2025-01-12T11:12:37Z

@The-Compiler thanks for the consideration, indeed now that I think about it more it might be surprising unless the user is aware of the change.

Invalidate the cache as soon as new tests are added (not sure what happens currently)
Keep the behavior, but make it clearer in the UI

I think those two are actually reasonable, and they even make sense to be implemented together. I will take a stab at them.

Thanks!

src/_pytest/stepwise.py

The-Compiler

Amazing work, thanks! Left a couple comments, mainly small nitpicks.

src/_pytest/stepwise.py

testing/test_stepwise.py

src/_pytest/stepwise.py

Now `--stepwise` will remember the last failed test, even if the previous pytest invocations did not pass `--stepwise`. Previously it would always clear the cache if not active, which hinders certain interactive workflows, which is the prime use cases for the flag.

nicoddemus · 2025-01-15T10:54:24Z

@The-Compiler ready for another review. 👍

The-Compiler

Sorry for the delay! Wanted to play around with this a bit rather than just looking at the diff, and didn't get around to it yet.

One last thing I noticed regarding the output, and then I think this is good to go!

The-Compiler · 2025-01-15T13:29:15Z

src/_pytest/stepwise.py

        else:
-            self.report_status = f"skipping {failed_index} already passed items."
+            self.report_status.append(
+                f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date},"


Suggested change

f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date},"

f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date:%c},"

maybe, where %c is "Locale’s appropriate date and time representation."?

Without this, it gets displayed as e.g. cache from 2025-01-15 14:25:45.458813 which seems a bit verbose, and the fractional seconds don't really add any useful information.

I did try that initially, but to me it showed up as the default locale (C): IIUC we need to call locale.setlocale at some point explicitly for %c to work properly, otherwise it will always use the C locale (and we cannot do that because this is a global setting, which pytest definitely should not touch IMO).

But I might be wrong, can you test that in your machine?

If %c does not work, we might explicitly use %YY-%M-%d %h:%m:%s to get rid of the fractional seconds, which I agree is not useful.

Oh wow, looks like it! My locale setup is a bit weird at the moment anyways (experimented with en_DK to get proper dates, but turns out it's not supported everywhere so now I get a mess of formats...). Because of that, I expected that to be an issue on my end.

What do you think about making a timedelta to now and then just formatting that in instead? That would then show up as 0:00:10 ago or 42 days, 1:23:45 ago which seems a much nicer format for the information we actually want to convey here.

edit: Whoops, I completely neglected that this would show microseconds too... I suppose that could be fixed with a datetime.timedelta(seconds=int(td.total_seconds())) (with td being the original delta).

Good idea about timedelta, will do.

Done.

I considered only showing that information if the cache was "old", but then what could be considered an "old" cache? 1 hr? 1 day? In the ended I kept it simple and always show the cache age for now.

Hehe, I have been thinking about that as well, and came to the same conclusion as you did 😅

The-Compiler

Thanks for sticking with this, looks great! Will merge in a day or two if nobody else wants to take a look.

The-Compiler · 2025-01-17T17:51:35Z

Thanks again! ✨

nicoddemus · 2025-01-20T15:26:26Z

Thanks @The-Compiler for merging! However that would be better as a squash probably. 👍

The-Compiler · 2025-01-20T15:29:05Z

I did consider that and decide against it, given that there were significant changes/amendments in the history of this PR.

This PR contains the following updates: | Package | Change | Age | Confidence | |---|---|---|---| | [pytest](https://github.com/pytest-dev/pytest) ([changelog](https://docs.pytest.org/en/stable/changelog.html)) | `<9.0.0,>=8.3.4` -> `<9.1.0,>=9.0.0` | [![age](https://developer.mend.io/api/mc/badges/age/pypi/pytest/9.0.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pytest/8.3.4/9.0.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>pytest-dev/pytest (pytest)</summary> ### [`v9.0.0`](pytest-dev/pytest@8.4.2...9.0.0) [Compare Source](pytest-dev/pytest@8.4.2...9.0.0) ### [`v8.4.2`](https://github.com/pytest-dev/pytest/releases/tag/8.4.2) [Compare Source](pytest-dev/pytest@8.4.1...8.4.2) ### pytest 8.4.2 (2025-09-03) #### Bug fixes - [#13478](pytest-dev/pytest#13478): Fixed a crash when using `console_output_style`{.interpreted-text role="confval"} with `times` and a module is skipped. - [#13530](pytest-dev/pytest#13530): Fixed a crash when using `pytest.approx`{.interpreted-text role="func"} and `decimal.Decimal`{.interpreted-text role="class"} instances with the `decimal.FloatOperation`{.interpreted-text role="class"} trap set. - [#13549](pytest-dev/pytest#13549): No longer evaluate type annotations in Python `3.14` when inspecting function signatures. This prevents crashes during module collection when modules do not explicitly use `from __future__ import annotations` and import types for annotations within a `if TYPE_CHECKING:` block. - [#13559](pytest-dev/pytest#13559): Added missing \[int]{.title-ref} and \[float]{.title-ref} variants to the \[Literal]{.title-ref} type annotation of the \[type]{.title-ref} parameter in `pytest.Parser.addini`{.interpreted-text role="meth"}. - [#13563](pytest-dev/pytest#13563): `pytest.approx`{.interpreted-text role="func"} now only imports `numpy` if NumPy is already in `sys.modules`. This fixes unconditional import behavior introduced in \[8.4.0]{.title-ref}. #### Improved documentation - [#13577](pytest-dev/pytest#13577): Clarify that `pytest_generate_tests` is discovered in test modules/classes; other hooks must be in `conftest.py` or plugins. #### Contributor-facing changes - [#13480](pytest-dev/pytest#13480): Self-testing: fixed a few test failures when run with `-Wdefault` or a similar override. - [#13547](pytest-dev/pytest#13547): Self-testing: corrected expected message for `test_doctest_unexpected_exception` in Python `3.14`. - [#13684](pytest-dev/pytest#13684): Make pytest's own testsuite insensitive to the presence of the `CI` environment variable -- by `ogrisel`{.interpreted-text role="user"}. ### [`v8.4.1`](https://github.com/pytest-dev/pytest/releases/tag/8.4.1) [Compare Source](pytest-dev/pytest@8.4.0...8.4.1) ### pytest 8.4.1 (2025-06-17) #### Bug fixes - [#13461](pytest-dev/pytest#13461): Corrected `_pytest.terminal.TerminalReporter.isatty` to support being called as a method. Before it was just a boolean which could break correct code when using `-o log_cli=true`). - [#13477](pytest-dev/pytest#13477): Reintroduced `pytest.PytestReturnNotNoneWarning`{.interpreted-text role="class"} which was removed by accident in pytest \[8.4]{.title-ref}. This warning is raised when a test functions returns a value other than `None`, which is often a mistake made by beginners. See `return-not-none`{.interpreted-text role="ref"} for more information. - [#13497](pytest-dev/pytest#13497): Fixed compatibility with `Twisted 25+`. #### Improved documentation - [#13492](pytest-dev/pytest#13492): Fixed outdated warning about `faulthandler` not working on Windows. ### [`v8.4.0`](https://github.com/pytest-dev/pytest/releases/tag/8.4.0) [Compare Source](pytest-dev/pytest@8.3.5...8.4.0) ### pytest 8.4.0 (2025-06-02) #### Removals and backward incompatible breaking changes - [#11372](pytest-dev/pytest#11372): Async tests will now fail, instead of warning+skipping, if you don't have any suitable plugin installed. - [#12346](pytest-dev/pytest#12346): Tests will now fail, instead of raising a warning, if they return any value other than None. - [#12874](pytest-dev/pytest#12874): We dropped support for Python 3.8 following its end of life (2024-10-07). - [#12960](pytest-dev/pytest#12960): Test functions containing a yield now cause an explicit error. They have not been run since pytest 4.0, and were previously marked as an expected failure and deprecation warning. See `the docs <yield tests deprecated>`{.interpreted-text role="ref"} for more information. #### Deprecations (removal in next major release) - [#10839](pytest-dev/pytest#10839): Requesting an asynchronous fixture without a \[pytest\_fixture\_setup]{.title-ref} hook that resolves it will now give a DeprecationWarning. This most commonly happens if a sync test requests an async fixture. This should have no effect on a majority of users with async tests or fixtures using async pytest plugins, but may affect non-standard hook setups or `autouse=True`. For guidance on how to work around this warning see `sync-test-async-fixture`{.interpreted-text role="ref"}. #### New features - [#11538](pytest-dev/pytest#11538): Added `pytest.RaisesGroup`{.interpreted-text role="class"} as an equivalent to `pytest.raises`{.interpreted-text role="func"} for expecting `ExceptionGroup`{.interpreted-text role="exc"}. Also adds `pytest.RaisesExc`{.interpreted-text role="class"} which is now the logic behind `pytest.raises`{.interpreted-text role="func"} and used as parameter to `pytest.RaisesGroup`{.interpreted-text role="class"}. `RaisesGroup` includes the ability to specify multiple different expected exceptions, the structure of nested exception groups, and flags for emulating `except* <except_star>`{.interpreted-text role="ref"}. See `assert-matching-exception-groups`{.interpreted-text role="ref"} and docstrings for more information. - [#12081](pytest-dev/pytest#12081): Added `capteesys`{.interpreted-text role="fixture"} to capture AND pass output to next handler set by `--capture=`. - [#12504](pytest-dev/pytest#12504): `pytest.mark.xfail`{.interpreted-text role="func"} now accepts `pytest.RaisesGroup`{.interpreted-text role="class"} for the `raises` parameter when you expect an exception group. You can also pass a `pytest.RaisesExc`{.interpreted-text role="class"} if you e.g. want to make use of the `check` parameter. - [#12713](pytest-dev/pytest#12713): New \[--force-short-summary]{.title-ref} option to force condensed summary output regardless of verbosity level. This lets users still see condensed summary output of failures for quick reference in log files from job outputs, being especially useful if non-condensed output is very verbose. - [#12749](pytest-dev/pytest#12749): pytest traditionally collects classes/functions in the test module namespace even if they are imported from another file. For example: ```python # contents of src/domain.py class Testament: ... # contents of tests/test_testament.py from domain import Testament def test_testament(): ... ``` In this scenario with the default options, pytest will collect the class \[Testament]{.title-ref} from \[tests/test\_testament.py]{.title-ref} because it starts with \[Test]{.title-ref}, even though in this case it is a production class being imported in the test module namespace. This behavior can now be prevented by setting the new `collect_imported_tests`{.interpreted-text role="confval"} configuration option to `false`, which will make pytest collect classes/functions from test files **only** if they are defined in that file. \-- by `FreerGit`{.interpreted-text role="user"} - [#12765](pytest-dev/pytest#12765): Thresholds to trigger snippet truncation can now be set with `truncation_limit_lines`{.interpreted-text role="confval"} and `truncation_limit_chars`{.interpreted-text role="confval"}. See `truncation-params`{.interpreted-text role="ref"} for more information. - [#13125](pytest-dev/pytest#13125): `console_output_style`{.interpreted-text role="confval"} now supports `times` to show execution time of each test. - [#13192](pytest-dev/pytest#13192): `pytest.raises`{.interpreted-text role="func"} will now raise a warning when passing an empty string to `match`, as this will match against any value. Use `match="^$"` if you want to check that an exception has no message. - [#13192](pytest-dev/pytest#13192): `pytest.raises`{.interpreted-text role="func"} will now print a helpful string diff if matching fails and the match parameter has `^` and `$` and is otherwise escaped. - [#13192](pytest-dev/pytest#13192): You can now pass `with pytest.raises(check=fn): <pytest.raises>`{.interpreted-text role="func"}, where `fn` is a function which takes a raised exception and returns a boolean. The `raises` fails if no exception was raised (as usual), passes if an exception is raised and `fn` returns `True` (as well as `match` and the type matching, if specified, which are checked before), and propagates the exception if `fn` returns `False` (which likely also fails the test). - [#13228](pytest-dev/pytest#13228): `hidden-param`{.interpreted-text role="ref"} can now be used in `id` of `pytest.param`{.interpreted-text role="func"} or in `ids` of `Metafunc.parametrize <pytest.Metafunc.parametrize>`{.interpreted-text role="py:func"}. It hides the parameter set from the test name. - [#13253](pytest-dev/pytest#13253): New flag: `--disable-plugin-autoload <disable_plugin_autoload>`{.interpreted-text role="ref"} which works as an alternative to `PYTEST_DISABLE_PLUGIN_AUTOLOAD`{.interpreted-text role="envvar"} when setting environment variables is inconvenient; and allows setting it in config files with `addopts`{.interpreted-text role="confval"}. #### Improvements in existing functionality - [#10224](pytest-dev/pytest#10224): pytest's `short` and `long` traceback styles (`how-to-modifying-python-tb-printing`{.interpreted-text role="ref"}) now have partial `657`{.interpreted-text role="pep"} support and will show specific code segments in the traceback. ```pytest ================================= FAILURES ================================= _______________________ test_gets_correct_tracebacks _______________________ test_tracebacks.py:12: in test_gets_correct_tracebacks assert manhattan_distance(p1, p2) == 1 ^^^^^^^^^^^^^^^^^^^^^^^^^^ test_tracebacks.py:6: in manhattan_distance return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y) ^^^^^^^^^ E AttributeError: 'NoneType' object has no attribute 'x' ``` \-- by `ammaraskar`{.interpreted-text role="user"} - [#11118](pytest-dev/pytest#11118): Now `pythonpath`{.interpreted-text role="confval"} configures \[$PYTHONPATH]{.title-ref} earlier than before during the initialization process, which now also affects plugins loaded via the \[-p]{.title-ref} command-line option. \-- by `millerdev`{.interpreted-text role="user"} - [#11381](pytest-dev/pytest#11381): The `type` parameter of the `parser.addini` method now accepts \["int"]{.title-ref} and `"float"` parameters, facilitating the parsing of configuration values in the configuration file. Example: ```python def pytest_addoption(parser): parser.addini("int_value", type="int", default=2, help="my int value") parser.addini("float_value", type="float", default=4.2, help="my float value") ``` The \[pytest.ini]{.title-ref} file: ```ini [pytest] int_value = 3 float_value = 5.4 ``` - [#11525](pytest-dev/pytest#11525): Fixtures are now clearly represented in the output as a "fixture object", not as a normal function as before, making it easy for beginners to catch mistakes such as referencing a fixture declared in the same module but not requested in the test function. \-- by `the-compiler`{.interpreted-text role="user"} and `glyphack`{.interpreted-text role="user"} - [#12426](pytest-dev/pytest#12426): A warning is now issued when `pytest.mark.usefixtures ref`{.interpreted-text role="ref"} is used without specifying any fixtures. Previously, empty usefixtures markers were silently ignored. - [#12707](pytest-dev/pytest#12707): Exception chains can be navigated when dropped into Pdb in Python 3.13+. - [#12736](pytest-dev/pytest#12736): Added a new attribute \[name]{.title-ref} with the fixed value \["pytest tests"]{.title-ref} to the root tag \[testsuites]{.title-ref} of the junit-xml generated by pytest. This attribute is part of many junit-xml specifications and is even part of the \[junit-10.xsd]{.title-ref} specification that pytest's implementation is based on. - [#12943](pytest-dev/pytest#12943): If a test fails with an exceptiongroup with a single exception, the contained exception will now be displayed in the short test summary info. - [#12958](pytest-dev/pytest#12958): A number of `unraisable <unraisable>`{.interpreted-text role="ref"} enhancements: - Set the unraisable hook as early as possible and unset it as late as possible, to collect the most possible number of unraisable exceptions. - Call the garbage collector just before unsetting the unraisable hook, to collect any straggling exceptions. - Collect multiple unraisable exceptions per test phase. - Report the `tracemalloc`{.interpreted-text role="mod"} allocation traceback (if available). - Avoid using a generator based hook to allow handling `StopIteration`{.interpreted-text role="class"} in test failures. - Report the unraisable exception as the cause of the `pytest.PytestUnraisableExceptionWarning`{.interpreted-text role="class"} exception if raised. - Compute the `repr` of the unraisable object in the unraisable hook so you get the latest information if available, and should help with resurrection of the object. - [#13010](pytest-dev/pytest#13010): `pytest.approx`{.interpreted-text role="func"} now can compare collections that contain numbers and non-numbers mixed. - [#13016](pytest-dev/pytest#13016): A number of `threadexception <unraisable>`{.interpreted-text role="ref"} enhancements: - Set the excepthook as early as possible and unset it as late as possible, to collect the most possible number of unhandled exceptions from threads. - Collect multiple thread exceptions per test phase. - Report the `tracemalloc`{.interpreted-text role="mod"} allocation traceback (if available). - Avoid using a generator based hook to allow handling `StopIteration`{.interpreted-text role="class"} in test failures. - Report the thread exception as the cause of the `pytest.PytestUnhandledThreadExceptionWarning`{.interpreted-text role="class"} exception if raised. - Extract the `name` of the thread object in the excepthook which should help with resurrection of the thread. - [#13031](pytest-dev/pytest#13031): An empty parameter set as in `pytest.mark.parametrize([], ids=idfunc)` will no longer trigger a call to `idfunc` with internal objects. - [#13115](pytest-dev/pytest#13115): Allows supplying `ExceptionGroup[Exception]` and `BaseExceptionGroup[BaseException]` to `pytest.raises` to keep full typing on `ExceptionInfo <pytest.ExceptionInfo>`{.interpreted-text role="class"}: ```python with pytest.raises(ExceptionGroup[Exception]) as exc_info: some_function() ``` Parametrizing with other exception types remains an error - we do not check the types of child exceptions and thus do not permit code that might look like we do. - [#13122](pytest-dev/pytest#13122): The `--stepwise` mode received a number of improvements: - It no longer forgets the last failed test in case pytest is executed later without the flag. This enables the following workflow: 1. Execute pytest with `--stepwise`, pytest then stops at the first failing test; 2. Iteratively update the code and run the test in isolation, without the `--stepwise` flag (for example in an IDE), until it is fixed. 3. Execute pytest with `--stepwise` again and pytest will continue from the previously failed test, and if it passes, continue on to the next tests. Previously, at step 3, pytest would start from the beginning, forgetting the previously failed test. This change however might cause issues if the `--stepwise` mode is used far apart in time, as the state might get stale, so the internal state will be reset automatically in case the test suite changes (for now only the number of tests are considered for this, we might change/improve this on the future). - New `--stepwise-reset`/`--sw-reset` flag, allowing the user to explicitly reset the stepwise state and restart the workflow from the beginning. - [#13308](pytest-dev/pytest#13308): Added official support for Python 3.14. - [#13380](pytest-dev/pytest#13380): Fix `ExceptionGroup`{.interpreted-text role="class"} traceback filtering to exclude pytest internals. - [#13415](pytest-dev/pytest#13415): The author metadata of the BibTex example is now correctly formatted with last names following first names. An example of BibLaTex has been added. BibTex and BibLaTex examples now clearly indicate that what is cited is software. \-- by `willynilly`{.interpreted-text role="user"} - [#13420](pytest-dev/pytest#13420): Improved test collection performance by optimizing path resolution used in `FSCollector`. - [#13457](pytest-dev/pytest#13457): The error message about duplicate parametrization no longer displays an internal stack trace. - [#4112](pytest-dev/pytest#4112): Using `pytest.mark.usefixtures <pytest.mark.usefixtures ref>`{.interpreted-text role="ref"} on `pytest.param`{.interpreted-text role="func"} now produces an error instead of silently doing nothing. - [#5473](pytest-dev/pytest#5473): Replace \[:]{.title-ref} with \[;]{.title-ref} in the assertion rewrite warning message so it can be filtered using standard Python warning filters before calling `pytest.main`{.interpreted-text role="func"}. - ``` [#6985](pytest-dev/pytest#6985): Improved `pytest.approx`{.interpreted-text role="func"} to enhance the readability of value ranges and tolerances between 0.001 and 1000. ``` ```` : - The [repr]{.title-ref} method now provides clearer output for values within those ranges, making it easier to interpret the results. - Previously, the output for those ranges of values and tolerances was displayed in scientific notation (e.g., [42 ± 1.0e+00]{.title-ref}). The updated method now presents the tolerance as a decimal for better readability (e.g., [42 ± 1]{.title-ref}). Example: **Previous Output:** ``` console >>> pytest.approx(42, abs=1) 42 ± 1.0e+00 ``` **Current Output:** ``` console >>> pytest.approx(42, abs=1) 42 ± 1 ``` \-- by `fazeelghafoor`{.interpreted-text role="user"} ```` - [#7683](pytest-dev/pytest#7683): The formerly optional `pygments` dependency is now required, causing output always to be source-highlighted (unless disabled via the `--code-highlight=no` CLI option). #### Bug fixes - [#10404](pytest-dev/pytest#10404): Apply filterwarnings from config/cli as soon as possible, and revert them as late as possible so that warnings as errors are collected throughout the pytest run and before the unraisable and threadexcept hooks are removed. This allows very late warnings and unraisable/threadexcept exceptions to fail the test suite. This also changes the warning that the lsof plugin issues from PytestWarning to the new warning PytestFDWarning so it can be more easily filtered. - [#11067](pytest-dev/pytest#11067): The test report is now consistent regardless if the test xfailed via `pytest.mark.xfail <pytest.mark.xfail ref>`{.interpreted-text role="ref"} or `pytest.fail`{.interpreted-text role="func"}. Previously, *xfailed* tests via the marker would have the string `"reason: "` prefixed to the message, while those *xfailed* via the function did not. The prefix has been removed. - [#12008](pytest-dev/pytest#12008): In `11220`{.interpreted-text role="pr"}, an unintended change in reordering was introduced by changing the way indices were assigned to direct params. More specifically, before that change, the indices of direct params to metafunc's callspecs were assigned after all parametrizations took place. Now, that change is reverted. - [#12863](pytest-dev/pytest#12863): Fix applying markers, including `pytest.mark.parametrize <pytest.mark.parametrize ref>`{.interpreted-text role="ref"} when placed above \[[@staticmethod](https://github.com/staticmethod)]{.title-ref} or \[[@classmethod](https://github.com/classmethod)]{.title-ref}. - [#12929](pytest-dev/pytest#12929): Handle StopIteration from test cases, setup and teardown correctly. - [#12938](pytest-dev/pytest#12938): Fixed `--durations-min` argument not respected if `-vv` is used. - [#12946](pytest-dev/pytest#12946): Fixed missing help for `pdb`{.interpreted-text role="mod"} commands wrapped by pytest -- by `adamchainz`{.interpreted-text role="user"}. - [#12981](pytest-dev/pytest#12981): Prevent exceptions in `pytest.Config.add_cleanup`{.interpreted-text role="func"} callbacks preventing further cleanups. - [#13047](pytest-dev/pytest#13047): Restore `pytest.approx`{.interpreted-text role="func"} handling of equality checks between \[bool]{.title-ref} and \[numpy.bool\_]{.title-ref} types. Comparing \[bool]{.title-ref} and \[numpy.bool\_]{.title-ref} using `pytest.approx`{.interpreted-text role="func"} accidentally changed in version \[8.3.4]{.title-ref} and \[8.3.5]{.title-ref} to no longer match: ```pycon >>> import numpy as np >>> from pytest import approx >>> [np.True_, np.True_] == pytest.approx([True, True]) False ``` This has now been fixed: ```pycon >>> [np.True_, np.True_] == pytest.approx([True, True]) True ``` - [#13119](pytest-dev/pytest#13119): Improved handling of invalid regex patterns for filter warnings by providing a clear error message. - [#13175](pytest-dev/pytest#13175): The diff is now also highlighted correctly when comparing two strings. - [#13248](pytest-dev/pytest#13248): Fixed an issue where passing a `scope` in `Metafunc.parametrize <pytest.Metafunc.parametrize>`{.interpreted-text role="py:func"} with `indirect=True` could result in other fixtures being unable to depend on the parametrized fixture. - [#13291](pytest-dev/pytest#13291): Fixed `repr` of `attrs` objects in assertion failure messages when using `attrs>=25.2`. - [#13312](pytest-dev/pytest#13312): Fixed a possible `KeyError` crash on PyPy during collection of tests involving higher-scoped parameters. - [#13345](pytest-dev/pytest#13345): Fix type hints for `pytest.TestReport.when`{.interpreted-text role="attr"} and `pytest.TestReport.location`{.interpreted-text role="attr"}. - [#13377](pytest-dev/pytest#13377): Fixed handling of test methods with positional-only parameter syntax. Now, methods are supported that formally define `self` as positional-only and/or fixture parameters as keyword-only, e.g.: ```python class TestClass: def test_method(self, /, *, fixture): ... ``` Before, this caused an internal error in pytest. - [#13384](pytest-dev/pytest#13384): Fixed an issue where pytest could report negative durations. - [#13420](pytest-dev/pytest#13420): Added `lru_cache` to `nodes._check_initialpaths_for_relpath`. - [#9037](pytest-dev/pytest#9037): Honor `disable_test_id_escaping_and_forfeit_all_rights_to_community_support`{.interpreted-text role="confval"} when escaping ids in parametrized tests. #### Improved documentation - [#12535](pytest-dev/pytest#12535): \[This example]{.title-ref}<<https://docs.pytest.org/en/latest/example/simple.html#making-test-result-information-available-in-fixtures>> showed `print` statements that do not exactly reflect what the different branches actually do. The fix makes the example more precise. - [#13218](pytest-dev/pytest#13218): Pointed out in the `pytest.approx`{.interpreted-text role="func"} documentation that it considers booleans unequal to numeric zero or one. - [#13221](pytest-dev/pytest#13221): Improved grouping of CLI options in the `--help` output. - [#6649](pytest-dev/pytest#6649): Added `~pytest.TerminalReporter`{.interpreted-text role="class"} to the `api-reference`{.interpreted-text role="ref"} documentation page. - [#8612](pytest-dev/pytest#8612): Add a recipe for handling abstract test classes in the documentation. A new example has been added to the documentation to demonstrate how to use a mixin class to handle abstract test classes without manually setting the `__test__` attribute for subclasses. This ensures that subclasses of abstract test classes are automatically collected by pytest. #### Packaging updates and notes for downstreams - [#13317](pytest-dev/pytest#13317): Specified minimum allowed versions of `colorama`, `iniconfig`, and `packaging`; and bumped the minimum allowed version of `exceptiongroup` for `python_version<'3.11'` from a release candidate to a full release. #### Contributor-facing changes - [#12017](pytest-dev/pytest#12017): Mixed internal improvements: - Migrate formatting to f-strings in some tests. - Use type-safe constructs in JUnitXML tests. - Moved`MockTiming` into `_pytest.timing`. \-- by `RonnyPfannschmidt`{.interpreted-text role="user"} - [#12647](pytest-dev/pytest#12647): Fixed running the test suite with the `hypothesis` pytest plugin. #### Miscellaneous internal changes - [#6649](pytest-dev/pytest#6649): Added `~pytest.TerminalReporter`{.interpreted-text role="class"} to the public pytest API, as it is part of the signature of the `pytest_terminal_summary`{.interpreted-text role="hook"} hook. ### [`v8.3.5`](https://github.com/pytest-dev/pytest/releases/tag/8.3.5) [Compare Source](pytest-dev/pytest@8.3.4...8.3.5) ### pytest 8.3.5 (2025-03-02) #### Bug fixes - [#11777](pytest-dev/pytest#11777): Fixed issue where sequences were still being shortened even with `-vv` verbosity. - [#12888](pytest-dev/pytest#12888): Fixed broken input when using Python 3.13+ and a `libedit` build of Python, such as on macOS or with uv-managed Python binaries from the `python-build-standalone` project. This could manifest e.g. by a broken prompt when using `Pdb`, or seeing empty inputs with manual usage of `input()` and suspended capturing. - [#13026](pytest-dev/pytest#13026): Fixed `AttributeError`{.interpreted-text role="class"} crash when using `--import-mode=importlib` when top-level directory same name as another module of the standard library. - [#13053](pytest-dev/pytest#13053): Fixed a regression in pytest 8.3.4 where, when using `--import-mode=importlib`, a directory containing py file with the same name would cause an `ImportError` - [#13083](pytest-dev/pytest#13083): Fixed issue where pytest could crash if one of the collected directories got removed during collection. #### Improved documentation - [#12842](pytest-dev/pytest#12842): Added dedicated page about using types with pytest. See `types`{.interpreted-text role="ref"} for detailed usage. #### Contributor-facing changes - [#13112](pytest-dev/pytest#13112): Fixed selftest failures in `test_terminal.py` with Pygments >= 2.19.0 - [#13256](pytest-dev/pytest#13256): Support for Towncrier versions released in 2024 has been re-enabled when building Sphinx docs -- by `webknjaz`{.interpreted-text role="user"}. </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.tainton.uk/repos/PwnedPW/pulls/305 Co-authored-by: renovate[bot] <renovate-bot@git.tainton.uk> Co-committed-by: renovate[bot] <renovate-bot@git.tainton.uk>

psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Jan 10, 2025

nicoddemus force-pushed the stepwise-clear-cache branch from b261556 to d7dfd4a Compare January 10, 2025 15:40

nicoddemus commented Jan 10, 2025

View reviewed changes

src/_pytest/stepwise.py Show resolved Hide resolved

nicoddemus force-pushed the stepwise-clear-cache branch 3 times, most recently from 96c3adb to 95d2390 Compare January 10, 2025 21:57

nicoddemus commented Jan 10, 2025

View reviewed changes

testing/test_cacheprovider.py Show resolved Hide resolved

nicoddemus added this to the 8.4 milestone Jan 12, 2025

nicoddemus commented Jan 12, 2025

View reviewed changes

src/_pytest/stepwise.py Outdated Show resolved Hide resolved

nicoddemus requested a review from The-Compiler January 12, 2025 13:23

The-Compiler requested changes Jan 13, 2025

View reviewed changes

nicoddemus requested a review from The-Compiler January 14, 2025 09:41

nicoddemus added 6 commits January 14, 2025 06:41

Add --stepwise-reset to restart the stepwise workflow

40b38f9

Tweak changelog

1837858

Tighten test

b6f1dad

Show cache information and reset cache if test count changes

0c30a58

Code review requests

1f7a151

nicoddemus force-pushed the stepwise-clear-cache branch from cdf06c3 to 1f7a151 Compare January 14, 2025 09:41

The-Compiler approved these changes Jan 15, 2025

View reviewed changes

Show cache age instead of update date

2fcaa79

The-Compiler approved these changes Jan 15, 2025

View reviewed changes

The-Compiler merged commit fc56ae3 into pytest-dev:main Jan 17, 2025
28 checks passed

nicoddemus deleted the stepwise-clear-cache branch January 20, 2025 15:25

	f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date},"
	f"skipping {failed_index} already passed items (cache from {self.cached_info.last_cache_date:%c},"

Uh oh!

Improve stepwise to not forget failed tests #13122

Improve stepwise to not forget failed tests #13122

Uh oh!

Conversation

nicoddemus commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

The-Compiler commented Jan 10, 2025

Uh oh!

Uh oh!

nicoddemus commented Jan 12, 2025

Uh oh!

Uh oh!

The-Compiler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicoddemus commented Jan 15, 2025

Uh oh!

The-Compiler left a comment

Choose a reason for hiding this comment

Uh oh!

The-Compiler Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

nicoddemus Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

The-Compiler Jan 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicoddemus Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

nicoddemus Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

The-Compiler Jan 15, 2025

Choose a reason for hiding this comment

Uh oh!

The-Compiler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

The-Compiler commented Jan 17, 2025

Uh oh!

nicoddemus commented Jan 20, 2025

Uh oh!

The-Compiler commented Jan 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nicoddemus commented Jan 10, 2025 •

edited

Loading

nicoddemus Jan 15, 2025 •

edited

Loading

The-Compiler Jan 15, 2025 •

edited

Loading