Convert py::object Numpy array to py::array

[kohr-h]

I'm playing around with embedding the interpreter, and I'd like to access a Numpy array stored in a py::object using pybind11's array handling functionality, and possibly hand it off to other code.

Here's a minimal example:

#include <pybind11/embed.h>
namespace py = pybind11;

int main() {
    py::scoped_interpreter guard{};
    py::object np = py::module::import("numpy");
    py::object arr = np.attr("ones")(3);
    // want it as py::array object now and call C++ functions on it
}

Here probably my ignorance of C++ shows, but how is this conversion done? Is there functionality for that in pybind11?

A follow-up on that question would then be: Can I stop the interpreter from deleting the object when the interpreter goes out of scope? If not, or if it requires too much hacking, I'd be okay with a copy.

If there's an elegant solution, I think it would be a nice addition to the docs on embedding the interpreter.

[jagerman]

The built-in python wrappers like py::array are implicitly convertible from py::objects, so you can simply write:

py::array_t<double> arr = np.attr("ones")(3);

[jagerman]

A follow-up on that question would then be: Can I stop the interpreter from deleting the object when the interpreter goes out of scope? If not, or if it requires too much hacking, I'd be okay with a copy.

No. When the interpreter goes out of scope there won't actually be a Python object left and so there would be no way back into the object.

It is possible, however, to have your data survive a interpreter shutdown and restart by having numpy operate on your own memory: pass a pointer to allocated memory (of at least the required size) into the py::array/py::array_t constructors. Then then numpy object won't allocate anything itself, but will just directly access/modify your memory.

[kohr-h]

Great, thanks @jagerman for the explanation! Would it make sense to add this kind of tiny example to the docs? I can make a PR.

[jagerman]

Sure; it would fit as a new section in docs/advanced/pycpp/object.rst, I suppose.

It would also be worth mentioning that the implicit conversion does a type-check and possibly a conversion at runtime, raising an exception if the object type isn't right and conversion isn't possible. (For example, if you tried: py::dict d = np.attr("ones")(3); you'd get an exception. On the other hand, I believe py::list l = np.attr("ones")(3); is allowed because a python list can be constructed from anything iterable).

[EricCousineau-TRI]

It is possible, however, to have your data survive a interpreter shutdown and restart by having numpy operate on your own memory: pass a pointer to allocated memory (of at least the required size) into the py::array/py::array_t constructors. Then then numpy object won't allocate anything itself, but will just directly access/modify your memory.

FWIW, in my experience, it seems that NumPy is not robust against restarting an interpreter. From how the C API is imported, I don't think there's any trigger for C extensions to recognize when a new instance of NumPy is loaded, so it may have stale API entries (e.g. for pybind11, npy_api::get() could be stale, if the solib is unloaded / then reloaded?). Ran into segfaults in this piece of code (which had relatively simple usages of NumPy), had to change it up a bit:
https://github.com/RobotLocomotion/drake/blob/68c65489cba2c69fbbe49f922e663169de51f1b4/bindings/pydrake/util/test/cpp_template_pybind_test.cc#L108

[kohr-h]

FWIW, in my experience, it seems that NumPy is not robust against restarting an interpreter.

Good to know!

I'm now working on a project that makes use of pybind11 to run Python in a C++ (and others) application. So far it works really nice.
And please don't close this due to inactivity from my side, I'm gonna write that PR, just haven't had time so far.

[bstaletic]

I'd have closed this one, but:

And please don't close this due to inactivity from my side, I'm gonna write that PR, just haven't had time so far.

So instead, this is just a ping.

[kohr-h]

Thanks for the ping! I made a PR.