Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable implicit conversion of 0 to pybind11::handle. #4008

Merged
merged 19 commits into from
Jul 14, 2022
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ca8d9fd
Disable implicit conversion from `0` to `pybind11::handle`.
Jun 14, 2022
21aa772
Reverse or-ed condition in an attempt to resolve GCC 8.3.0 errors (i3…
Jun 15, 2022
4ff51fd
Trying the simpler `std::is_same<T, PyObject *>`
Jun 15, 2022
e322611
Add implicit_conversion_from_pytorch_THPObjectPtr_to_handle test.
Jun 15, 2022
02fdc20
Accommodate types with implicit conversions to `PyObject *`, other th…
Jun 15, 2022
75317d8
Fix copy-paste mishap (picked wrong name).
Jun 15, 2022
05283e2
Revamp SFINAE construct to actually fix the pytorch issue (already va…
Jun 16, 2022
5410da7
Use `NOLINT(performance-noexcept-move-constructor)` for reduced code …
Jun 16, 2022
ded9c66
Use any_of, all_of, negation. It turns out to clang-format nicer.
Jun 16, 2022
6aa92e1
Clean up comments for changed code.
Jun 16, 2022
468addb
Reduce pytorch situation further, add test for operator ... const.
Jun 17, 2022
bc22bc8
Use `none_of` as suggested by @skylion007
Jun 17, 2022
5340256
Add `pure_compile_tests_for_handle_from_PyObject_pointers()`
Jun 17, 2022
a2781c6
Fix inconsequential oversight (retested).
Jun 18, 2022
8e6ad64
Factor our `is_pyobj_ptr_or_nullptr_t` to make the SFINAE conditions …
Jun 18, 2022
0bb8284
Remove stray line (oversight).
Jun 18, 2022
dbab18a
Make the `pure_compile_tests_for_handle_from_PyObject_pointers()` "rh…
Jun 18, 2022
5d42ec7
Remove the temporary PYBIND11_UNDO_PR4008 `#ifdef`.
Jun 18, 2022
ac94635
Merge branch 'master' into implicit_conversion_from_0_to_handle
Jul 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions include/pybind11/pytypes.h
Original file line number Diff line number Diff line change
Expand Up @@ -211,9 +211,27 @@ class handle : public detail::object_api<handle> {
public:
/// The default constructor creates a handle with a ``nullptr``-valued pointer
handle() = default;
/// Creates a ``handle`` from the given raw Python object pointer

/// Enable implicit conversion from ``PyObject *`` and ``nullptr``.
/// Not using ``handle(PyObject *ptr)`` to avoid implicit conversion from ``0``.
template <typename T,
detail::enable_if_t<detail::any_of<std::is_same<T, PyObject *>,
std::is_same<T, std::nullptr_t>>::value,
int> = 0>
// NOLINTNEXTLINE(google-explicit-constructor)
handle(T ptr) : m_ptr(ptr) {}

/// Enable implicit conversion through ``T::operator PyObject *()``.
template <typename T,
detail::enable_if_t<
detail::all_of<detail::negation<detail::any_of<std::is_base_of<handle, T>,
std::is_same<T, PyObject *>,
std::is_same<T, PyObject *const>,
std::is_same<T, std::nullptr_t>>>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
detail::enable_if_t<
detail::all_of<detail::negation<detail::any_of<std::is_base_of<handle, T>,
std::is_same<T, PyObject *>,
std::is_same<T, PyObject *const>,
std::is_same<T, std::nullptr_t>>>,
detail::enable_if_t<
detail::all_of<detail::none_of<std::is_base_of<handle, T>,
std::is_same<T, PyObject *>,
std::is_same<T, PyObject *const>,
std::is_same<T, std::nullptr_t>>

This probably still isn't the best way to do this, will this match volatile, mutable etc pointers. I think we may need to use detail::remove_cv<T>::type or something similar here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also there is detail::satisifies_none_of for the std::is_same block, but i am not sure its that much cleaner. Might be a bit when using remove_cv_ref or other behavior.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was still thinking about the cv cases.
The most important thing is to come up with corresponding unit tests first. Without we (me at least) are certain to get lost in the incomprehensible maze of C++ rules and special cases.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rwgk I feel as though some template library has to have this implemented already with all the edge cases we need and we can just use that.

Copy link
Collaborator Author

@rwgk rwgk Jun 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"feel" and "probably" comments like this are massively unhelpful.
You gave up really quickly (#4006).
I'm presenting a solution that is already globally tested.
If you think this can be done better, please show how.
With tests.
I'm always happy to accept something that is provably better, and will be happy to globally test that for you.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Apologies, I didn't intend to come off rude or dismissive of your efforts. I was just making a generic note for future discussions.

Thanks for providing this really helpful PR so far and testing against all the edge case in PyTorch. I just meant to comment on how surprised I was to find that disabling this troublesome implicit conversion in C++ is so difficult. Thanks for all the effort.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Skylion007 I added all the tests that I could think of (5340256) to validate that the only behavior change is to disable the implicit conversion from 0 to handle. I left a temporary #ifdef in case you have ideas for additional tests, or want to validate yourself that the commented-out code fails to compile before and after this PR.

Between

  • the existing unit tests
  • the added operator PyObject *() test reduced from the wild (pytorch)
  • the added operator PyObject *() const test
  • the added pure_compile_tests_for_handle_from_PyObject_pointers() tests
  • and global testing

I think we are extremely well covered. Maybe the SFINAE constructs could be polished, but unless we have more additional tests that expose a hole, I'd say this is as good as it gets.

I want to rerun the global testing though, although I don't think the change to none_of would break anything.

std::is_convertible<T, PyObject *>>::value,
int> = 0>
// NOLINTNEXTLINE(google-explicit-constructor)
handle(PyObject *ptr) : m_ptr(ptr) {} // Allow implicit conversion from PyObject*
handle(T &obj) : m_ptr(obj) {}

/// Return the underlying ``PyObject *`` pointer
PyObject *ptr() const { return m_ptr; }
Expand Down
35 changes: 35 additions & 0 deletions tests/test_pytypes.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,42 @@ class float_ : public py::object {
};
} // namespace external

namespace implicit_conversion_from_0_to_handle {
// Uncomment to trigger compiler error. Note: Before PR #4008 this used to compile successfully.
// void expected_to_trigger_compiler_error() { py::handle(0); }
} // namespace implicit_conversion_from_0_to_handle

namespace pytorch_object_ptr_h_reduced {

// Reduced from
// https://github.com/pytorch/pytorch/blob/279634f384662b7c3a9f8bf7ccc3a6afd2f05657/torch/csrc/utils/object_ptr.h
template <class T>
class THPPointer {
public:
explicit THPPointer(T *ptr) noexcept : ptr(ptr){};
THPPointer(THPPointer &&p) { // NOLINT(performance-noexcept-move-constructor)
// free();
ptr = p.ptr;
p.ptr = nullptr;
};
operator T *() { return ptr; } // NOLINT(google-explicit-constructor)
private:
T *ptr = nullptr;
};

using THPObjectPtr = THPPointer<PyObject>;

bool handle_from_obj_ptr() {
THPObjectPtr obj_ptr(Py_None);
auto h = py::handle(obj_ptr); // Critical part of test: does this compile?
return h.ptr() == Py_None; // Just something.
}

} // namespace pytorch_object_ptr_h_reduced

TEST_SUBMODULE(pytypes, m) {
m.def("handle_from_pytorch_obj_ptr", pytorch_object_ptr_h_reduced::handle_from_obj_ptr);

// test_bool
m.def("get_bool", [] { return py::bool_(false); });
// test_int
Expand Down
4 changes: 4 additions & 0 deletions tests/test_pytypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@
from pybind11_tests import pytypes as m


def test_handle_from_pytorch_obj_ptr():
assert m.handle_from_pytorch_obj_ptr()


def test_bool(doc):
assert doc(m.get_bool) == "get_bool() -> bool"

Expand Down