[BUG]: tracking issue for PyPy 7.3.6 support (PyPy 3.8)

[henryiii]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

We are having issues with the PyPy update, so I'm going to avoid making it and just list the problems here, eventually opening up another PR (currently mixed into #3368). This will also focus only on supporting it, not on building binaries with it, which is a bad idea (on PyPy3.7, anyway) - users should wait till 7.3.7 to build binaries (and existing ones like possibly NumPy won't work).

For both 3.7 and 3.8:

• test_to_python returns 1 from cstats.alive()

For 3.8 Unix:

• test_int_convert does not produce a warning (was not included before)
• test_numpy_int_convert does not produce a warning (Also was not included before)
• PyLong_AsLong implementation does not contain the updates made in CPython to call __index__.
• test_eval_empty_globals does not contain __builtins__ (or anything).
• test_python_alreadyset_in_destructor returns False from triggered[0]

For 3.8 Windows:

• Dies with ImportError: initialization failed. Probably from the issue we've always had on 3.8 not being able to run the cross module tests on Windows.

For 2.7 macOS:

It seems the parallel setuptools_helpers test hangs. Could be intermittent, a CI issue, or something else, but could be related to PyPy 7.3.6. Edit: rerunning made it pass.

[henryiii]

The now-failing test links to https://foss.heptapod.net/pypy/pypy/-/issues/2444 (CC @mattip), but instead of working, it now seems to be leaking, perhaps? Since this runs on both CPython and PyPY, I'm assuming the test was working before, even though it has an issue at the top.

pybind11/tests/test_buffers.py

Lines 38 to 70 in d45a881

	# https://foss.heptapod.net/pypy/pypy/-/issues/2444
	def test_to_python():
	mat = m.Matrix(5, 4)
	assert memoryview(mat).shape == (5, 4)
	assert mat[2, 3] == 0
	mat[2, 3] = 4.0
	mat[3, 2] = 7.0
	assert mat[2, 3] == 4
	assert mat[3, 2] == 7
	assert struct.unpack_from("f", mat, (3 * 4 + 2) * 4) == (7,)
	assert struct.unpack_from("f", mat, (2 * 4 + 3) * 4) == (4,)
	mat2 = np.array(mat, copy=False)
	assert mat2.shape == (5, 4)
	assert abs(mat2).sum() == 11
	assert mat2[2, 3] == 4 and mat2[3, 2] == 7
	mat2[2, 3] = 5
	assert mat2[2, 3] == 5
	cstats = ConstructorStats.get(m.Matrix)
	assert cstats.alive() == 1
	del mat
	pytest.gc_collect()
	assert cstats.alive() == 1
	del mat2 # holds a mat reference
	pytest.gc_collect()
	assert cstats.alive() == 0
	assert cstats.values() == ["5x4 matrix"]
	assert cstats.copy_constructions == 0
	# assert cstats.move_constructions >= 0 # Don't invoke any
	assert cstats.copy_assignments == 0
	assert cstats.move_assignments == 0

For the two tests not producing warnings for int conversion, I think that's a difference in implementation (but needs to be checked, CC @YannickJadoul.

pybind11/tests/test_builtin_casters.py

Lines 335 to 342 in d45a881

	# The implicit conversion from np.float32 is undesirable but currently accepted.
	# TODO: Avoid DeprecationWarning in `PyLong_AsLong` (and similar)
	if (3, 8) <= env.PY < (3, 10):
	with env.deprecated_call():
	assert convert(np.float32(3.14159)) == 3
	else:
	assert convert(np.float32(3.14159)) == 3
	require_implicit(np.float32(3.14159))

For test_eval_empty_globals, this seems to be a difference between CPython and PyPy; starting in 3.8, CPython injects __builtins__ with PyRun_String, while PyPy doesn't. :'( We can make our workaround include PyPy on 3.8. include/pybind11/eval.h contains the workaround, and it's pretty safe, just costs an extra Python containership check.

pybind11/include/pybind11/eval.h

Lines 21 to 32 in d45a881

	inline void ensure_builtins_in_globals(object &global) {
	#if PY_VERSION_HEX < 0x03080000
	// Running exec and eval on Python 2 and 3 adds `builtins` module under
	// `__builtins__` key to globals if not yet present.
	// Python 3.8 made PyRun_String behave similarly. Let's also do that for
	// older versions, for consistency.
	if (!global.contains("__builtins__"))
	global["__builtins__"] = module_::import(PYBIND11_BUILTINS_MODULE);
	#else
	(void) global;
	#endif
	}

Looks like the block patching in the hook in 3.8 doesn't work in PyPy 3.8.

pybind11/tests/test_exceptions.py

Lines 102 to 122 in d45a881

	hooked = False
	triggered = [False] # mutable, so Python 2.7 closure can modify it
	if hasattr(sys, "unraisablehook"): # Python 3.8+
	hooked = True
	# Don't take `sys.unraisablehook`, as that's overwritten by pytest
	default_hook = sys.__unraisablehook__
	def hook(unraisable_hook_args):
	exc_type, exc_value, exc_tb, err_msg, obj = unraisable_hook_args
	if obj == "already_set demo":
	triggered[0] = True
	default_hook(unraisable_hook_args)
	return
	# Use monkeypatch so pytest can apply and remove the patch as appropriate
	monkeypatch.setattr(sys, "unraisablehook", hook)
	assert m.python_alreadyset_in_destructor("already_set demo") is True
	if hooked:
	assert triggered[0] is True

[mattip]

The now-failing test links to https://foss.heptapod.net/pypy/pypy/-/issues/2444, but instead of working, it now seems to be leaking, perhaps?

It would be good to track this down, since that issue is closed. We yet again modified buffer handling for PyPy3.8 and maybe missed something. With that, the implementation of pytest.gc_collect() calls gc.collect() twice. In NumPy's implementation, I needed 3 calls.

starting in 3.8, CPython injects builtins with PyRun_String, while PyPy doesn't.

Do you know where/how this change happened? I don't see it documented for PyRun_String nor in the 3.8 changelog. Maybe this is not 3.8-specific or is it more general to all invocations of eval?

For sys.unraisablehook, I opened https://foss.heptapod.net/pypy/pypy/-/issues/3583

[henryiii]

Do you know where/how this change happened?

#2616 points at python/cpython#13362, the implementation of PEP 587.

[mattip]

For __builtins__, I opened https://foss.heptapod.net/pypy/pypy/-/issues/3584

[mattip]

For sys.unraisablehook, I opened https://foss.heptapod.net/pypy/pypy/-/issues/3583

I think that might have been on PyPy3.7, sys.unraisablehook is there on PyPy3.8.

[henryiii]

I think it's there, but it doesn't work. The code path does trigger to set it, but it doesn't do anything.

[mattip]

Strange. sys.unraisablehook = ... is used in Lib/test/support for a catch_unraisable_exceptions context manager, and, for instance, we are passing test_coroutines::test_fatal_coro_warning which uses it, so I am not sure what is going on.

[henryiii]

I'm rerunning it (as part of something else) in #3419 - I'll see if I can see what part is failing.

[henryiii]

Interesting, for failure 1, that happens on PyPy3.7 7.3.7 as well (wow that's a lot of threes and sevens). So it's a change with all PyPy, not just the 3.8 branch. It used to behave more like CPython, and now isn't - though like anything related to the garbage collector, I'm not sure it's so critical. I'll try multiple calls, which I think is what was suggested above.

[henryiii]

Another PyPy 3.8 issue: Before Python 3.8, PyLong_AsLong does not pick up on obj.__index__. PyPy 3.8 seems to trigger the old 3.7 and before behavior, rather than using __index__ (at least that's what it looks like at the moment).

Also, there's a missing DeprecationWarning in PyLong_AsLong. I think that's from CPython and not us.

I see difference between PyPy and CPython's PyIndex_Check. Pretty sure that's unrelated.

[henryiii]

We can (and do) backport that behavior for 3.7 and before, but it should be supported by PyPy3.8 directly, I think.

[mattip]

It is a bit hard to manage all this in a github issue conversation. Can you point to the failing tests?