Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using --benchmark-only hits assertion in pytest #243

Open
jvesely opened this issue Jun 30, 2023 · 0 comments
Open

Using --benchmark-only hits assertion in pytest #243

jvesely opened this issue Jun 30, 2023 · 0 comments

Comments

@jvesely
Copy link

jvesely commented Jun 30, 2023

$ pytest --dist=no -n0 --benchmark-only --benchmark-enable:

conftest.py::PYCODESTYLE 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 155, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_result.py", line 108, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 80, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 155, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_result.py", line 108, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 80, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/runner.py", line 127, in runtestprotocol
INTERNALERROR>     rep = call_and_report(item, "setup", log)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 133, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_result.py", line 108, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/pluggy/_callers.py", line 80, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/user/py311/lib/python3.11/site-packages/_pytest/reports.py", line 355, in from_item_and_call
INTERNALERROR>     assert line is not None
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR> AssertionError

============================================================================================================ 8 warnings in 22.69s ============================================================================================================

The issue appears to be with test skipping code. Adding -m benchmark to the commandline works around the issue,
pytest --dist=no -n0 --benchmark-only --benchmark-enable -m benchmark
works as expected.
jvesely added a commit to jvesely/PsyNeuLink that referenced this issue Jun 30, 2023
@jvesely
ci: Disable pytest-xdist in benchmark ci job
6b50bfc 
Use "--benchmark-only" together "-m benchmark".
The former will fail if benchmarking is not available.
The latter works around a bug in pytest test collection:
	ionelmc/pytest-benchmark#243

Fixes: d6f8e35
	("tests: Use worksteal xdist balancer (PrincetonUniversity#2670)")

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
This was referenced Jun 30, 2023
Closed
Merged
jvesely added a commit to PrincetonUniversity/PsyNeuLink that referenced this issue Jun 30, 2023
@jvesely
ci: Disable pytest-xdist in benchmark ci job (#2710)
521d3e7 
Use "--benchmark-only" in addition to "-m benchmark".
The former will fail if benchmarking is not available/possible.
The latter works around a bug in pytest test collection:
	ionelmc/pytest-benchmark#243

Fixes: d6f8e35
	("tests: Use worksteal xdist balancer (#2670)")

Signed-off-by: Jan Vesely <jan.vesely@rutgers.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jvesely