Re-think our pypy3 CI runs

[fselmo]

There is a very delicate balance we are playing with the RAM game on pypy3 CI runner. We currently have the size-xl-x64 which has 32GB of RAM (ref). The more we increase the load, the worse this dance gets (obviously) but it's not just OOM on the whole process. Sometimes single runners are running OOM on particular tests and erring out instead of exiting with code 143.

• Single runners erring out (example)
• Full run OOM code 143 (example)

I recently did some playing around and was able to tune this to account for including up-to-Amsterdam tests here. But note how delicate this balance is. I was not able to use 24GB / 32GB (8 runners with 3G) for the runners because I believe we need more than 8GB for OS / other processes as it seems the entire process OOMs with these numbers. I could not find a good balance here even with 5 runners and 3G. Adding Amsterdam took us from ~63k to ~109k tests, I think we need some better options.

All this to say we need more than 32GB for this task or we need to reconsider how many tests we run for pypy3.

Possible solutions

• 🩹 We could increase the RAM on our runner. This will work for some time but we'll eventually run into these issues again as, only adding Amsterdam, we went from 60k tests to 109k because Amsterdam fills all forks + its tests, etc. This is the "bandaid" fix.
• 📊 We could be more mindful about the kinds of runs we spin up. We could create a nightly build that splits at the full run into at most 4 forks (or something like this) and triggers separate runs of pypy3 broken up into 4-fork runs for all forks. Then we can leave only the latest fork or N and N-1 to run on every PR. We can add a note that reviewers should manually run previous fork tests if they are affected for a PR.

Or some other option that is more sustainable. Thoughts and ideas welcome.

[danceratopz]

These PyPy jobs have been bothering me for a while. PyPy should be faster, but the py3 job takes on average ~12 mins longer (analysis below).

Could we consider dropping PyPy support entirely? Who's our PyPy user?

If we decide to keep it, I would suggest that we reduce the tox pypy3 environment to perform a smoke test (of PyPy) instead of a full test run that fills all fixtures. It currently duplicates the CPython run in py3. This seems overkill. Could

Reducing the test set to perform a smoke tests would of course also resolve the RAM issues.

Why is it slower?

This is what I found out with some quick research, it makes sense to me as we make a lot of calls to C libraries for, e.g., hashing:

• PyPy's speed comes from JIT-compiling pure Python code.
• C extensions use CPython's internal C API; PyPy emulates this via cpyext.
• Every Python → C call pays a translation tax.
• Crypto libs make many small, fast C calls — overhead dominates.
• PyPy's JIT can't optimize across the boundary; C code is a black box.

TL;DR: Pure Python = fast on PyPy. C extensions = slow. Crypto-heavy codebases are the worst case.

PyPy3 vs CPython3 CI Job Timing Analysis

Analysis of 5 recent successful runs of the test.yaml workflow from ethereum/execution-specs.

fill (py3 vs pypy3)

Run	py3	pypy3	Overhead
#4308	37m 24s	45m 5s	+7m 41s
#4306	42m 11s	60m 24s	+18m 13s
#4304	36m 36s	48m 12s	+11m 36s
#4303	37m 44s	47m 29s	+9m 45s
#4302	39m 27s	55m 47s	+16m 20s
Average	38m 40s	51m 23s	+12m 43s (33% slower)

Testing Unit Tests (tests_pytest_py3 vs tests_pytest_pypy3)

Run	py3	pypy3	Overhead
#4308	2m 36s	3m 49s	+1m 13s
#4306	2m 42s	3m 46s	+1m 4s
#4304	2m 42s	4m 17s	+1m 35s
#4303	2m 34s	4m 3s	+1m 29s
#4302	2m 32s	3m 55s	+1m 23s
Average	2m 37s	3m 58s	+1m 20s (51% slower)