[BUG] def_property_readonly doesn't respect return_value_policy

[hzhangxyz]

pybind11 version is ace4deb
python version is

Python 3.8.6 (default, Sep 30 2020, 04:00:38)
[GCC 10.2.0] on linux

and gcc version is gcc (GCC) 10.2.0

my minimum working example is

#include "/home/hzhangxyz/Desktop/TAT/pybind11/include/pybind11/pybind11.h"

struct A {
  int a;
};
struct B {
  int *b;
};

namespace py = pybind11;
PYBIND11_MODULE(T, tm) {
  py::class_<A>(tm, "A")
      .def(py::init<int>())
      .def_readonly("a", &A::a)
      .def_property_readonly(
          "b", [](A &a) { return B{&a.a}; }, py::keep_alive<0, 1>());
  py::class_<B>(tm, "B").def_property_readonly("b",
                                               [](const B &b) { return *b.b; });
}

save it as T.cpp

now run g++ T.cpp -I/usr/include/python3.8 -lpython3.8 -fPIC -shared -o T.cpython-38-x86_64-linux-gnu.so and get T.cpython-38-x86_64-linux-gnu.so

test with python -c 'import T, sys; a=T.A(233); b=a.b; print(sys.getrefcount(a))' and get output 2, it is incorrect, it should be 3

so when I do python -c 'import T; a=T.A(233); b=a.b; print(b.b); del a; print(b.b)' it gives the wrong output

233
279652016

or simply python -c 'import T; print(T.A(233).b.b)' give incorrect 652495216

Any one can tell me how to let code like T.A(233).b.b valid?

In my project, I have code like this, the common thing is the def_property_readonly return a object containing the reference(or pointer) to self

[YannickJadoul]

I might be wrong, but a first quick look at the code seems that a return_value_policy or keep_alive isn't correctly passed to the property-getter, indeed.

[marc-chiesa]

Is this also true of call_guard? I had a deadlock in a readonly getter because it didn't release the gil (the internal property is wrapped in a mutex), even though I had put a py::call_guard<py::gil_scoped_release>() on it. Making it a lambda with a manually added gil_scoped_release resolved the issue.

[YannickJadoul]

@marc-chiesa Possibly. Can you confirm that manually creating a py::cpp_function, passing py::call_guard<py::gil_scoped_release>() to that constructor, and using that to call def_property_readonly doés work?

[daquexian]

Can you confirm that manually creating a py::cpp_function, passing py::call_guard<py::gil_scoped_release>() to that constructor, and using that to call def_property_readonly doés work?

@YannickJadoul it works, while .def_property_readonly("x", x, py::call_guard<py::gil_scoped_release>()) doesn't actually release the gil

[tttapa]

Is there a technical limitation that makes it impossible to make use of py::keep_alive for properties?
If so, there should probably be a note about it in the docs, because I spent way longer than I'd like to admit trying to figure out why I was having lifetime issues 😄

For what it's worth, manually wrapping the getter in a py::cpp_function seems to work as a workaround:

// View that wraps a pointer to an integer.
struct MemberView {
    const int *p;
    MemberView(const int &i) : p{&i} { py::print("MemberView()"); }
    MemberView(const MemberView &o) : p{o.p} { py::print("MemberView(const MemberView &)"); }
    MemberView(MemberView &&) = delete;
    MemberView &operator=(const MemberView &) = delete;
    MemberView &operator=(MemberView &&) = delete;
    ~MemberView() { py::print("~MemberView()"); }
    int value() const { return *p; }
};

// Has an integer member that can be viewed.
struct ExampleClass {
    int i;
    ExampleClass(int i) : i(i) { py::print("ExampleClass(int)"); }
    ExampleClass(const ExampleClass &o) = delete;
    MemberView get_view() const {
        py::print("ExampleClass::get_view()");
        return i;
    }
    ~ExampleClass() { py::print("~ExampleClass()"); }
};

py::class_<MemberView>(m, "MemberView") //
    .def_property_readonly("value", &MemberView::value);

py::class_<ExampleClass>(m, "ExampleClass") //
    .def(py::init<int>())
    // Normal function works:
    .def("get_view", &ExampleClass::get_view, py::return_value_policy::copy,
            py::keep_alive<0, 1>())
    // Property doesn't work
    .def_property_readonly("view", &ExampleClass::get_view, py::return_value_policy::copy,
                            py::keep_alive<0, 1>())
    // Property with the getter wrapped in py::cpp_function works
    .def_property_readonly("view_hack", py::cpp_function(&ExampleClass::get_view,
                                                            py::return_value_policy::copy,
                                                            py::keep_alive<0, 1>()));

def test_get_view():
    def function():
        print("enter function()")
        c = m.ExampleClass(0x12345678)
        view = c.get_view()
        print("return from function()")
        return view
    view = function()
    assert view.value == 0x12345678

def test_view():
    def function():
        print("enter function()")
        c = m.ExampleClass(0x12345678)
        view = c.view
        print("return from function()")
        return view
    view = function()
    assert view.value == 0x12345678

def test_view_hack():
    def function():
        print("enter function()")
        c = m.ExampleClass(0x12345678)
        view = c.view_hack
        print("return from function()")
        return view
    view = function()
    assert view.value == 0x12345678

=========================================== FAILURES ===========================================
__________________________________________ test_view ___________________________________________

~MemberView()
    def test_view():
        def function():
            print("enter function()")
            c = m.ExampleClass(0x12345678)
            view = c.view
            print("return from function()")
            return view
        view = function()
>       assert view.value == 0x12345678
E       assert 2135319984 == 305419896
E        +  where 2135319984 = <module._module.MemberView object at 0x7f4c7f4665b0>.value

module/python/test/test_module.py:58: AssertionError
------------------------------------- Captured stdout call -------------------------------------
enter function()
ExampleClass(int)
ExampleClass::get_view()
MemberView()
MemberView(const MemberView &)
~MemberView()
return from function()
~ExampleClass()
============================================ PASSES ============================================
________________________________________ test_get_view _________________________________________
------------------------------------- Captured stdout call -------------------------------------
enter function()
ExampleClass(int)
ExampleClass::get_view()
MemberView()
MemberView(const MemberView &)
~MemberView()
return from function()
~MemberView()
~ExampleClass()
________________________________________ test_view_hack ________________________________________
------------------------------------- Captured stdout call -------------------------------------
enter function()
ExampleClass(int)
ExampleClass::get_view()
MemberView()
MemberView(const MemberView &)
~MemberView()
return from function()
~MemberView()
~ExampleClass()
================================= 1 failed, 2 passed in 0.19s ==================================