Reference tests' coverage

[hmijail]

My team is writing a verified EVM, and we are working on running the reference tests on it. So we would like to compare our results to the execution specs' test runs.

For that, I'm trying to understand how much of the EVM and of the tests is currently covered by the execution specs; browsing through the source I see that not all tests are run.

Is there any better/easier way to grasp the current status than reading the source and understanding all the testing machinery?

Relatedly, is there any connection between the execution specs and the PyEVM?

[SamWilsn]

For that, I'm trying to understand how much of the EVM and of the tests is currently covered by the execution specs; browsing through the source I see that not all tests are run.

As far as I am aware, we implement the full EVM up to but excluding the Paris fork. We do skip some tests, but #597 is unskipping a bunch.

Is there any better/easier way to grasp the current status than reading the source and understanding all the testing machinery?

I believe (and perhaps @gurukamath can confirm) that if you run the tests all the fixtures are listed and the xfails are skipped because we don't support them.

Relatedly, is there any connection between the execution specs and the PyEVM?

Nope, this is all pretty fresh.

[gurukamath]

Yes. You can simply run the state tests with the -v option to display the fixtures that are run.

For example, if you want to see the frontier tests that are run

pytest -v tests/frontier/test_state_transition.py