Skip to content

Drop support for PyPy < 7.3.1 and clean up old PyPy workarounds #2456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 5, 2020
Merged

Drop support for PyPy < 7.3.1 and clean up old PyPy workarounds #2456

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b885208
Remove code inside 'PYPY_VERSION_NUM < 0x06000000' preprocessor if br…
YannickJadoul Sep 2, 2020
35e6c21
fix: more cleanup
henryiii Sep 2, 2020
9d346cc
Remove more references to PyPy 5.7 and 5.9 in the docs
YannickJadoul Sep 5, 2020
9d08a18
Update comment on PyUnicode_UTF* in PyPy
YannickJadoul Oct 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ Goodies
In addition to the core functionality, pybind11 provides some extra
goodies:

- Python 2.7, 3.5+, and PyPy (tested on 7.3) are supported with an
- Python 2.7, 3.5+, and PyPy/PyPy3 7.3 are supported with an
implementation-agnostic interface.

- It is possible to bind C++11 lambda functions with captured
Expand Down
8 changes: 4 additions & 4 deletions docs/advanced/misc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ avoids this issue involves weak reference with a cleanup callback:

.. code-block:: cpp

// Register a callback function that is invoked when the BaseClass object is colelcted
// Register a callback function that is invoked when the BaseClass object is collected
py::cpp_function cleanup_callback(
[](py::handle weakref) {
// perform cleanup here -- this function is called with the GIL held
Expand All @@ -237,9 +237,9 @@ avoids this issue involves weak reference with a cleanup callback:

.. note::

PyPy (at least version 5.9) does not garbage collect objects when the
interpreter exits. An alternative approach (which also works on CPython) is to use
the :py:mod:`atexit` module [#f7]_, for example:
PyPy does not garbage collect objects when the interpreter exits. An alternative
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to re-confirm, this, though. I did not do that, yet.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have tests for that. Some tests explicitly call gc.collect() more than once because of PyPy, so it shouldn't be hard to verify.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll give it a quick try, but it's actually documented: https://doc.pypy.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies

Last note: CPython tries to do a gc.collect() automatically when the program finishes; not PyPy. (It is possible in both CPython and PyPy to design a case where several gc.collect() are needed before all objects die. This makes CPython’s approach only work “most of the time” anyway.)

Should we add this to the docs, or are we fine as is?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I actually can't find any tests on the garbage collector passing by when the interpreter exits, though. So not sure how to apply @bstaletic's suggestion.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about calling PyGC_Collect()? Would that work across the board?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good plan, but seems PyPy doesn't have PyGC_Collect :-(

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than gc.collect(), which isn't C, there's PyGC_Collect() and a bunch of "didn't mean to make it public" functions that got removed in python 3.9. On the other hand PyGC_Collect() was never documented.

Conclusion: CPython is a horrible mess.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PyPy devs, struggling to keep up with massive amounts of C API every release, are the last ones you need to convince of that statement, I'm afraid ;-)

approach (which also works on CPython) is to use the :py:mod:`atexit` module [#f7]_,
for example:

.. code-block:: cpp

Expand Down
6 changes: 2 additions & 4 deletions include/pybind11/cast.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1277,10 +1277,8 @@ template <typename StringType, bool IsView = false> struct string_caster {
UTF_N == 16 ? PyUnicode_DecodeUTF16(buffer, nbytes, nullptr, nullptr) :
PyUnicode_DecodeUTF32(buffer, nbytes, nullptr, nullptr);
#else
// PyPy seems to have multiple problems related to PyUnicode_UTF*: the UTF8 version
// sometimes segfaults for unknown reasons, while the UTF16 and 32 versions require a
// non-const char * arguments, which is also a nuisance, so bypass the whole thing by just
// passing the encoding as a string value, which works properly:
// PyPy segfaults when on PyUnicode_DecodeUTF16 (and possibly on PyUnicode_DecodeUTF32 as well),
// so bypass the whole thing by just passing the encoding as a string value, which works properly:
return PyUnicode_Decode(buffer, nbytes, UTF_N == 8 ? "utf-8" : UTF_N == 16 ? "utf-16" : "utf-32", nullptr);
#endif
}
Expand Down
4 changes: 0 additions & 4 deletions include/pybind11/detail/class.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,10 +478,6 @@ extern "C" inline int pybind11_clear(PyObject *self) {
/// Give instances of this type a `__dict__` and opt into garbage collection.
inline void enable_dynamic_attributes(PyHeapTypeObject *heap_type) {
auto type = &heap_type->ht_type;
#if defined(PYPY_VERSION) && (PYPY_VERSION_NUM < 0x06000000)
pybind11_fail(get_fully_qualified_tp_name(type) + ": dynamic attributes are currently not "
"supported in conjunction with PyPy!");
#endif
type->tp_flags |= Py_TPFLAGS_HAVE_GC;
type->tp_dictoffset = type->tp_basicsize; // place dict at the end
type->tp_basicsize += (ssize_t)sizeof(PyObject *); // and allocate enough space for it
Expand Down
2 changes: 1 addition & 1 deletion include/pybind11/eval.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ void exec(const char (&s)[N], object global = globals(), object local = object()
eval<eval_statements>(s, global, local);
}

#if defined(PYPY_VERSION) && PY_VERSION_HEX >= 0x3000000
#if defined(PYPY_VERSION) && PY_VERSION_HEX >= 0x03000000
template <eval_mode mode = eval_statements>
object eval_file(str, object, object) {
pybind11_fail("eval_file not supported in PyPy3. Use eval");
Expand Down
2 changes: 0 additions & 2 deletions tests/test_multiple_inheritance.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,14 +193,12 @@ TEST_SUBMODULE(multiple_inheritance, m) {
.def_readwrite_static("static_value", &VanillaStaticMix2::static_value);


#if !(defined(PYPY_VERSION) && (PYPY_VERSION_NUM < 0x06000000))
struct WithDict { };
struct VanillaDictMix1 : Vanilla, WithDict { };
struct VanillaDictMix2 : WithDict, Vanilla { };
py::class_<WithDict>(m, "WithDict", py::dynamic_attr()).def(py::init<>());
py::class_<VanillaDictMix1, Vanilla, WithDict>(m, "VanillaDictMix1").def(py::init<>());
py::class_<VanillaDictMix2, WithDict, Vanilla>(m, "VanillaDictMix2").def(py::init<>());
#endif

// test_diamond_inheritance
// Issue #959: segfault when constructing diamond inheritance instance
Expand Down