Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

bluetech · 2024-01-04T12:35:11Z

Example: https://github.com/pytest-dev/pytest/actions/runs/7408595776/job/20157160097

This doesn't always happen, maybe 50% of the time. Started in the last few days. I checked if the PyPy version changed during these days, but seems not (it's 3.8.16).

The error is:

 __________________ ERROR collecting testing/_py/test_local.py __________________
/opt/hostedtoolcache/PyPy/3.8.16/x64/lib/pypy3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1023: in _gcd_import
    ???
<frozen importlib._bootstrap>:1000: in _find_and_load
    ???
<frozen importlib._bootstrap>:984: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:166: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:350: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, but got <_ast.Store object at 0x0000000002142880>

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

The text was updated successfully, but these errors were encountered:

See pytest-dev/pytest#11771

skirpichev · 2024-02-25T03:57:13Z

I have same failure on older versions as well. E.g. 7.4.3:
https://github.com/mpmath/mpmath/actions/runs/8030300102?pr=759

This happens for all tested in the mpmath pypy versions (3.8-3.10). But most often on 3.10.

skirpichev · 2024-03-30T06:08:57Z

Another example: #11168 (comment)

@cfbolz, I don't see a bugreport in pypy issues. Is there any chance this might be related to #11168 (an fixed in pypy)?

cfbolz · 2024-03-30T13:58:06Z

@skirpichev it's not impossible that it's the same problem, yes. You can try a nightly PyPy build (or wait for the next release but that is still a bit out).

skirpichev · 2024-03-31T02:30:04Z

@cfbolz, thanks for a suggestion. I did tests for nightly builds (on 3.8-3.10) and, after ~20 re-runs, I got a TypeError on pypy3.8, which is similar to this issue:

============================= test session starts ==============================
platform linux -- Python 3.8.16[pypy-7.3.12-alpha], pytest-8.1.1, pluggy-1.4.0
mpmath backend: python
mpmath mp class: <mpmath.ctx_mp.MPContext object at 0x00000000018deb10>
mpmath version: 1.4.0a1.dev43+g534bc79
Python version: 3.8.16 (d8e6b37f5231c7b59b2b639a651dd696a3732cc3, Jan 09 2023, 00:30:15)
[PyPy 7.3.12-alpha0 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
rootdir: /home/runner/work/mpmath/mpmath
configfile: pyproject.toml
testpaths: mpmath, docs
plugins: cov-5.0.0, xdist-3.5.0
created: 4/4 workers
4 workers [2294 items]

........................................................................ [  3%]
........................................................................ [  6%]
........................................................................ [  9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 69%]
........................................................................ [ 72%]
......................................................................... [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................x............................... [ 91%]
.......................................x................................ [ 94%]
...................................s.................................... [ 97%]
.............................................................            [100%]
==================================== ERRORS ====================================
_______________ ERROR collecting mpmath/tests/test_basic_ops.py ________________
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1023: in _gcd_import
    ???
<frozen importlib._bootstrap>:1000: in _find_and_load
    ???
<frozen importlib._bootstrap>:984: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:169: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:353: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920>
============================= slowest 20 durations =============================
34.04s call     mpmath/tests/test_functions2.py::test_bessel_zeros_extra
28.52s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False23]
16.64s call     mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyper
11.94s call     mpmath/tests/test_extra_zeta.py::test_zetazero[241389216-97490234.22767118]
10.72s call     mpmath/functions/zeta.py::mpmath.functions.zeta.secondzeta
9.91s call     mpmath/tests/test_elliptic.py::test_elliptic_integrals
9.25s call     mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.bessely
8.75s call     mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyp3f2
8.59s call     mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.siegelz
8.44s call     mpmath/calculus/extrapolation.py::mpmath.calculus.extrapolation.nsum
8.39s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False9]
7.87s call     mpmath/functions/zetazeros.py::mpmath.functions.zetazeros.nzeros
7.58s call     mpmath/tests/test_torture.py::test_asymp[f_wrapped-1500-False1]
7.53s call     mpmath/tests/test_quad.py::test_quadosc
7.50s call     mpmath/calculus/quadrature.py::mpmath.calculus.quadrature.QuadratureMethods.quad
7.45s call     mpmath/tests/test_torture.py::test_asymp[f_wrapped-90-False]
7.27s call     mpmath/tests/test_torture.py::test_asymp[f-5000-True1]
7.22s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False27]
7.00s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False37]
6.74s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False36]
=========================== short test summary info ============================
ERROR mpmath/tests/test_basic_ops.py - TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920>
======= 2291 passed, 1 skipped, 2 xfailed, 1 error in 203.43s (0:03:23) ========

Summary page: https://github.com/mpmath/mpmath/actions/runs/8491432157?pr=772

~~Maybe this fix isn't merged yet to the 3.8 branch (as it was stated in your blog post)? If so,~~ I'll continue testing.

cfbolz · 2024-03-31T09:15:24Z

Yes, the 3.8 variants are not maintained any more, you need to try 3.9 or 3.10. Thanks for trying this out!

skirpichev · 2024-03-31T13:30:17Z

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures. Given the previous statistics, probably there is a chance that this issue was fixed in PyPy's nightly builds. I'll switch regular tests for pypy3.9/10 to them.

Thanks for bugfix!

cfbolz · 2024-04-02T09:05:24Z

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures.

thank you for trying this, that's very valuable feedback!

webknjaz · 2024-07-01T15:56:50Z

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

During the sprints, we were seeing 3–4 variants of different AST object reprs there. Hopefully, bumping PyPy in CI will fix this. Ronny suggested this, but nobody got to actually doing it...

webknjaz · 2024-07-01T15:59:43Z

FTR @LilyFoote linked https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html in #11168 (comment) but that reference never got cross-posted into this issue.

webknjaz · 2024-07-01T16:04:20Z

Action item for whoever gets to prepare the PR:

Update this to 3.9 https://github.com/pytest-dev/pytest/blob/51ee388/.github/workflows/test.yml#L153
Mention the above article in the contrib-type change note
Add a Resolves #11771 somewhere in the commit message or the PR description

x612skm · 2024-07-01T17:09:22Z

Working on this issue

PyPy 3.8 has a flaky bug in the garbage collector that is not going to be fixed there since that version is EoL, but newer version have it addressed. Here's the problem we're seeing in the CI - TypeError: expected some sort of stmt, but got <_ast.Load object ...> Here's the Link to the explanation - pytest-dev#11771 (comment) Resolved pytest-dev#11771

webknjaz · 2024-07-02T14:32:59Z

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

I even saw a case of NameError, which seems to have the same root cause:

 E   NameError: name '@py_assert2' is not defined

(https://github.com/pytest-dev/pytest/actions/runs/9746885084/job/26898335196#step:6:97)

PyPy 3.8 has a flaky bug in the garbage collector that is not going to be fixed there since that version is EoL, but newer version have it addressed. Here's the problem we're seeing in the CI ```TypeError: expected some sort of stmt, but got <_ast.Load object ...> ``` Here's the Link to the explanation Resolves pytest-dev#11771

PyPy 3.8 has a flaky bug in the garbage collector that is not going to be fixed there since that version is EoL, but newer version have it addressed. Here's the problem we're seeing in the CI: ``` TypeError: expected some sort of stmt, but got <_ast.Load object ...> ``` Here's the Link to the explanation Resolves pytest-dev#11771

PyPy 3.8 has a flaky bug in the garbage collector that is not going to be fixed there since that version is EoL, but newer version have it addressed. Here's the problem we're seeing in the CI: "TypeError: expected some sort of stmt, but got <_ast.Load object ...>" Here's the Link to the explanation Resolves pytest-dev#11771

PyPy 3.8 has a flaky bug in the garbage collector that is not going to be fixed there since that version is EoL, but newer versions have it addressed. The problem manifests itself in CI as follows: ```console TypeError: expected some sort of stmt, but got <_ast.Load object ...> ``` This patch makes use of the PyPy version that is still supported and contains a fix for the said bug. Ref: https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html Resolves pytest-dev#11771.

Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>

bluetech added the type: selftests a problem in the tests of pytest label Jan 4, 2024

skirpichev added a commit to skirpichev/mpmath that referenced this issue Feb 24, 2024

Freeze pytest deps

9c95964

See pytest-dev/pytest#11771

skirpichev added a commit to skirpichev/mpmath that referenced this issue Feb 24, 2024

Freeze pytest deps

42014bc

See pytest-dev/pytest#11771

skirpichev mentioned this issue Mar 1, 2024

Random failures in CI on various PyPy versions: TypeError: expected some sort of stmt, but got <ast.Name/Load... mpmath/mpmath#762

Closed

webknjaz added the good first issue easy issue that is friendly to new contributor label Jul 1, 2024

webknjaz assigned x612skm Jul 1, 2024

x612skm mentioned this issue Jul 1, 2024

Bump PyPy runtime to v3.9 @ GHA #12557

Merged

x612skm added a commit to x612skm/pytest-dev--pytest that referenced this issue Jul 4, 2024

Add a change note for PR pytest-dev#11771

3363f7e

alexfikl mentioned this issue Jul 6, 2024

Remove a bunch of unused pylint supressions inducer/pytools#243

Merged

webknjaz linked a pull request Jul 19, 2024 that will close this issue

Bump PyPy runtime to v3.9 @ GHA #12557

Merged

x612skm added a commit to x612skm/pytest-dev--pytest that referenced this issue Jul 19, 2024

Add a change note for PR pytest-dev#11771

ced7072

Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>

webknjaz closed this as completed in #12557 Jul 19, 2024

patchback bot mentioned this issue Jul 19, 2024

[PR #12557/a7d5a8eb backport][8.2.x] Bump PyPy runtime to v3.9 @ GHA #12632

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

bluetech commented Jan 4, 2024

skirpichev commented Feb 25, 2024

skirpichev commented Mar 30, 2024

cfbolz commented Mar 30, 2024

skirpichev commented Mar 31, 2024 •

edited

Loading

cfbolz commented Mar 31, 2024

skirpichev commented Mar 31, 2024

cfbolz commented Apr 2, 2024

webknjaz commented Jul 1, 2024

webknjaz commented Jul 1, 2024

webknjaz commented Jul 1, 2024 •

edited

Loading

x612skm commented Jul 1, 2024

webknjaz commented Jul 2, 2024

Selftests sometimes fail on pypy: TypeError: expected some sort of stmt, but got <_ast.Load object ...> #11771

Selftests sometimes fail on pypy: TypeError: expected some sort of stmt, but got <_ast.Load object ...> #11771

Comments

bluetech commented Jan 4, 2024

skirpichev commented Feb 25, 2024

skirpichev commented Mar 30, 2024

cfbolz commented Mar 30, 2024

skirpichev commented Mar 31, 2024 • edited Loading

cfbolz commented Mar 31, 2024

skirpichev commented Mar 31, 2024

cfbolz commented Apr 2, 2024

webknjaz commented Jul 1, 2024

webknjaz commented Jul 1, 2024

webknjaz commented Jul 1, 2024 • edited Loading

x612skm commented Jul 1, 2024

webknjaz commented Jul 2, 2024

Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

skirpichev commented Mar 31, 2024 •

edited

Loading

webknjaz commented Jul 1, 2024 •

edited

Loading