Find out why coverage dropped when testing with PyPy3.8

[pepoluan]

When doing fixes/upgrades/enhancements to GitHub CI, I found out that coverage dropped if the Python version was PyPy3.8, necessitating a skip on coverage testing specifically for PyPy3.8

Here's an example: https://github.com/aio-libs/aiosmtpd/actions/runs/3695940538/jobs/6259040475

Strangely, now that I have the time to peruse the full result of the action, it seems to affect only PyPy3.8 on Ubuntu-{18,20}.04 ?

Is it worth it to delve deeper?

[pepoluan]

In commit 5b62b88 I chose to only explicitly exclude the combination of PyPy3.8 + Ubuntu-{18,20}.04.

If test is green I'll just leave it at that and not delve deeper.

[pepoluan]

Update: No, it's not just limited to {18,20}.04

See this: https://github.com/aio-libs/aiosmtpd/actions/runs/3710309144

I think I'll "revert" to the strategy of not doing any coverage testing for PyPy3.8

[pepoluan]

This will still need to be investigated, though. So I'm keeping this issue alive.

[pepoluan]

After some investigation, it's because PyPy3.8 changes some decorator behavior since v7.3.10.

This is described in nedbat/coveragepy#1515

Until a new coverage.py is released (>7.0.1) there is nothing we can do.

The new behavior has been accounted for by coverage.py for PyPy>=3.9, though, so when GHCI finally adds PyPy3.9 runner, we should be good.

Therefore I'm going to 'ignore' this issue for now until a new coverage.py version that accounts for v7.3.10's behavior change is released.

[waynew]

👍 seems reasonable to me

[pepoluan]

Aaaand 7.0.3 has been released several hours ago (at time of my commenting)

I'll revisit this in a couple of days.

[pepoluan]

Tried the latest coverage-7.0.5 and it still fails if the test is running on PyPy3.8 7.3.10-final, but works on PyPy3.8 7.3.11-final.

7.3.10-final is apparently the version that gets installed on MacOS-11, MacOS-12, and Windows-2019. Other OSes in the GitHub Action runners installs 7.3.11-final.

Because we have successfully run the tests on PyPy3.9 anyways, I think it's just too much effort to tweak the tests so that coverage is 100% on PyPy3.8. Besides, the 'canonical' coverage test is based on CPython 3.8 on Ubuntu 20.04 anyways.

Therefore I will let the current GHCI config (no coverage for PyPy3.8) to stand.