Leaking of C++ objects if they are constructed with a python function

[yeganer]

Issue description

C++ objects that take a python function as argument are not properly deleted. The issue seems to be that the Python garbage collector cannot break the reference cycle. I hope the example code illustrates the issue well. If there are any questions, you can also find me in the gitter.

Reproducible example code

yeganer/pybind11@d27b39e:test

[yeganer]

For easier reproducibility I made a PR (#1301 ).
Can you please give me any advice on how to debug this? Is this an issue with PyBind11, CPython in general or with the C++ code?

Any help would be greatly appreciated.

[jagerman]

I don't quite understand why this is happening: it seems the func captured in the lambda in functional.h isn't properly destroyed when the type_caster itself gets destroyed. It should destroy the std::function<Return(Args...)> value; member, which should itself destroy the lambda-with-capture, including the captured variables.

As a workaround, it seems like making func a private member variable (rather than local variable in load), then capturing this instead of func in the lambda solves the issue.

But like I said, I don't understand why the current code doesn't work.

Oh, I wasn't looking closely enough: the std::function is copied into the TestObject, which is what keeps it alive. I'm still not seeing what's keeping the TestObject alive, though.

[jagerman]

I think this falls into the pit of "Bugs and caveats" with interpreter finalization. Especially given that your PR is passing on all the 2.7 builds, but failing on the 3.6 builds points to this being a Python-level issue. (Somewhat odd, mind you, is that the i386/Python 3.6 build passes).

I really have no idea how to solve this (other than a "don't do that" recommendation about storing globals).

[yeganer]

@jagerman Thanks for your response.
I made a small modification to #1301 which now causes a failure on all platforms. All I did was change the global dictionary to a locally generated one.

Maybe I'm doing something wrong but to me it looks like any pybind object that has a python function as member should outlive the Interpreter and it's destructor will never be called.

Is there a flaw in my goal?