Skip to content

feat(test): add support for TAP (Test Anything Protocol) #834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: staging
Choose a base branch
from

Conversation

jenstroeger
Copy link
Owner

@jenstroeger jenstroeger commented Nov 18, 2024

The interesting thing about TAP (Test Anything Protocol) is that test output from different sources can be consolidated and analyzed as one. This change adds a pytest plugin which generated TAP output for tests, thus enabling TAP compatible test output.

However, I think before we would consider this change we need to track PR python-tap/pytest-tap#91 (and maybe PR python-tap/pytest-tap#92).

@jenstroeger
Copy link
Owner Author

With pytest-tap v3.5 released we can now capture and dump the output of individual tests as well:

> make test
Run unit tests...........................................................Passed
- hook id: pytest
- duration: 0.84s

TAP version 13
1..3
ok 1 src/package/something.py::[doctest] package.something.Something.do_something
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#
ok 2 tests/test_something.py::test_something
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#
ok 3 docs/source/index.rst::[doctest] index.rst
# --- Captured Log ---
# 
# --- Captured Out ---
# 
# --- Captured Err ---
#

However, some pytest plugins also print stats after the tests have been run:

---------- coverage: platform darwin, python 3.13.1-final-0 ----------
Name                       Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------
src/package/__init__.py        1      0      0      0   100%
src/package/something.py       4      0      0      0   100%
----------------------------------------------------------------------
TOTAL                          5      0      0      0   100%

Required test coverage of 100.0% reached. Total coverage: 100.00%
============================ Hypothesis Statistics =============================

tests/test_something.py::test_something:

  - during generate phase (0.00 seconds):
    - Typical runtimes: ~ 0-1 ms, of which < 1ms in data generation
    - 2 passing examples, 0 failing examples, 0 invalid examples

  - Stopped because nothing left to do


============================== slowest durations ===============================
0.01s call     tests/test_something.py::test_something
0.00s call     src/package/something.py::package.something.Something.do_something
0.00s setup    src/package/something.py::package.something.Something.do_something
0.00s call     docs/source/index.rst::index.rst
0.00s teardown src/package/something.py::package.something.Something.do_something
0.00s setup    docs/source/index.rst::index.rst
0.00s teardown docs/source/index.rst::index.rst
0.00s setup    tests/test_something.py::test_something
0.00s teardown tests/test_something.py::test_something
============================== 3 passed in 0.08s ===============================

and those are still missing. We’ll have to dig around some more and check if we can capture those as well, and output them…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant