Help running coverage with pytest --testmon

[Tomasz-Kluczkowski]

Hi,

Is there any chance that coverage report could be written when running it with pytest-testmon package? Is there anything one can do to get them cooperating with each other to run only the tests for changed files and generate an updated coverage report for only what needs updating?

Tom

[nedbat]

@Tomasz-Kluczkowski I'm not familiar with pytest-testmon, so I'm not sure what it would take. Have you talked to the maintainers?

[Tomasz-Kluczkowski]

Hey, Thanks for coming back on this. I think after reading coverage’s docs little bit more I know why pytest-testmon has issues. It is down to using Python’s stack tracing feature and fighting with coverage for it. What I use pytest-testmon is running selective tests based on changes in the code. I wanted to combine running those tests with updating of the coverage report. Unfortunately it is not possible:(. But I know you guys work on v.5.0 (correct me if I am wrong please) where you implement a feature similar to what pytest-testmon uses - a database of test to code relationships which could be used in identical way to run tests for only the changed code and update the coverage report at the same time. I really do hope you have success with this kind of API :). It would help a lot and make coverage even better than it is now (and it is awesome 😎 already). Best regards, Tomasz On 17 Feb 2019, at 23:05, Ned Batchelder <notifications@github.com<mailto:notifications@github.com>> wrote: @Tomasz-Kluczkowski<https://github.com/Tomasz-Kluczkowski> I'm not familiar with pytest-testmon, so I'm not sure what it would take. Have you talked to the maintainers? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#774 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AY1UaSe2eyGuE3VgJMIxgb34CMoMwTPZks5vOeApgaJpZM4a2dl8>.

[blueyed]

See also the following issues:

• concurrent usage of settrace doesn't work in Python ecosystem  tarpas/pytest-testmon#97
• Coverage with/after pdb.set_trace #729
• coverage.py doesn't seem to work with pdb.set_trace() #647

[tarpas]

Hey, pytest-testmon maintainer here. We track this issue here tarpas/pytest-testmon#86

As far as I can comprehend the new format in coverage 5.0 is not usable for pytest-testmon, and given that it has more features and config options it will be more difficult to implement the enhancement in our project then with coverage 4.x

[nedbat]

@tarpas The new data collected in 5.0 should make testmon's job easier I would think?

[tarpas]

@nedbat thanks for reaching out.
Some of our design goals, which I think are different to yours:

• data stability even if some lines change (testmon avoids using line numbers), "lossy compression" (we cannot reconstruct line by line coverage, we only store method checksums)
• progressively store data so it's possible to interrupt the test run and continue where we left off
• storing failures (exceptions)
• We are and want to be very tightly integrated with pytest.

For that reasons I don't think our efforts overlap much and don't foresee testmon could scrap some part of code and just build on top of coverage 5.0

So sorry, I have to reiterate that implementing our ticket #86 also produce standard .coverage file with .testmondata will be more difficult with 5.0 than before. I now know ho to implement it, I just didn't have yet time to do it. (a tiny PR to coveragepy wil be a part of that)