Current state of Partialtesting

[MGough]

I've been using partialtesting and wanted to raise a few issues & open some discussion here as it's rather quiet. I think there's some points that people should be aware of and consider before using this tool. It would be interesting to know if anyone is using it (or similar alternatives) on larger projects.

Here are the things I've noticed:

Non-python files triggering full test run
This is documented, non-project files result in a full test run. There's no way to ignore certain files, should a config file change trigger a full test run? Probably. Should a README file change trigger a full test run? Probably not. It seems like it would be useful for the user to specify which files should cause this.

Can't ignore specific tests
Similar to the above, if I have some tests which aren't part of my regular test suite (perhaps they are integration tests to run against a specific environment) then there's no easy way to just instruct partialtesting to ignore them.

CI Pipeline integration
This is always going to be tricky. If two PRs are opened which have some overlap and both pipelines pass, when one is merged into master the coverage file there will change, and the partialtest in the unmerged PR is no longer valid.

Partial/Full/No Test
The way to identify whether to run a partial, full or no test is one of the following:

• No file created
• Empty file created
• File with contents created

It would be much clearer and consistent to handle if a file was always created containing a known & consistent format, like JSON to be parsed with jq.

[fersarr]

Hi Merlin, nice to hear you are using PT!

For your points 1 & 2 (Non-python files triggering full test run & Can't ignore specific tests), I think this might be helpful:
https://github.com/man-group/partialtesting/blob/master/partialtesting/partialtesting.py#L35

Basically, you can use --special-extensions and --special-files to customize how you run partialtesting.

This should be docummented in the README but it's not. Currently it's only visible via partialtesting --help I believe. Feel free to send a PR for the README if you have time. Also, I thought that .md (READMEs) was included in the list, but it's not, so feel free to add that too.

[MGough]

Thanks for responding (and so quickly too!). I had a look through that file before the end of 2020, but I think I'd misunderstood how those were being used. I see now that I can override those defaults via the CLI.

From a few lines below in that file it looks like .md files are being ignored by default, I've probably just done something silly there when running PT in my pipeline. Hopefully I can give those options a try soon and if I have a chance I'll open a PR to document them. I'll post an update if I start on that to avoid duplicating efforts.

[MGough]

I'll need to review the code you linked further. For some of our cases I think it will work, but there is a case where we want to completely ignore some files. Much like the NO_TESTS_EXTENSIONS (https://github.com/man-group/partialtesting/blob/master/partialtesting/partialtesting.py#L57), a NO_TESTS_FILES would be useful

[fersarr]

From a few lines below in that file it looks like .md files are being ignored by default

Perhaps NO_TESTS_EXTENSIONS isn't working as expected. It looks like there are 2 tests that specifically test for READMEs in case you wanna have a look. The first one (integration test) seems more useful for this case: https://github.com/man-group/partialtesting/blob/master/tests/integration/test_partialtesting_integration.py#L307 and https://github.com/man-group/partialtesting/blob/master/tests/unit/test_partial_testing.py#L203

Much like the NO_TESTS_EXTENSIONS (https://github.com/man-group/partialtesting/blob/master/partialtesting/partialtesting.py#L57), a NO_TESTS_FILES would be useful

Yea, I think it sounds like a good idea to have NO_TESTS_FILES.