A review of the PHPUnit cache

[theofidry]

I have some questions regarding the PHPUnit cache.

First of all: when does one benefit from it?

The only piece of documentation I can find about it is this:

Configure a directory where the PHPUnit test runner can cache data such as test results (required for reordering tests based on previous failures, for instance) or information about tested code as well as test code generated using static analysis (significantly improves performance of code coverage analysis, for instance).

What I understand:

• if you use the defects order type, it will use it. By default it uses default, which unfortunately is not documented. At least here it is listed, but it does not say what it does, so you do not know if by default it is used or not.
• Static analysis of some sorts may use it. Now, I know it is third-party so hard to have a reliable list, but maybe it would be good have a few listed? Personally I have no idea which kind uses it, and when.

Performance

Having a brief look at the cache directly, that is a lot of files. On Infection it is about 850 of them. I do not know if there is any extra cost for generating the cache, but I can only assume there's bound to be a cost for at least writing that many files.

I have no idea of the performance cost, I could not notice it on a naive benchmark I did with hyperfine.

That being said, if seems like a wasteful operation and if it proves to have a cost, would it make sense to have an option to disable it in the CI?

[sebastianbergmann]

Test Results

The CLI option --do-not-cache-result can be used to prevent the writing of .phpunit.cache/test-results (default path). The same effect can be achieved by setting cacheResult="false" in the XML configuration file.

By the way, this should have never been refered to as "caching". We simply store the results of the previous test suite execution to sort the test suite based on this information for the current test suite execution.

Static Analysis

The code coverage functionality needs static analysis. When this static analysis is performed then we store the result in .phpunit.cache/code-coverage (default path). Unless the --warm-coverage-cache CLI option is used, this is only done when code coverage reporting is requested.

[sebastianbergmann]

That being said, if seems like a wasteful operation and if it proves to have a cost, would it make sense to have an option to disable it in the CI?

Writing the static analysis cache in CI is wasteful when it is not kept between CI runs (for the same branch). I do not think it would make sense to introduce configuration options for not storing the static analysis result in the cache.

[theofidry]

The code coverage functionality needs static analysis.

Could you expend a tiny bit more about this?

[sebastianbergmann]

Could you expend a tiny bit more about this?

This is the code that performs the static analysis: https://github.com/sebastianbergmann/php-code-coverage/tree/main/src/StaticAnalysis.

[theofidry]

So do I understand correctly that unless you're displaying any sort of coverage report, this basically has no use? And otherwise it would be in your interest to store them because it would save some I/O?

[sebastianbergmann]

So do I understand correctly that unless you're displaying any sort of coverage report, this basically has no use?

When no code coverage reporting is requested, the static analysis required for this is not performed and no result of static analysis is written to the cache (as there is no result).

[theofidry]

Ok! So sounds like it's only a doc issue then 😄

I'll try to get to it soonish. Maybe you want to rephrase the title & labels