Segfault from pyo3 trying to call `PyBuffer::drop` after Python interpreter already finalized

[kylebarron]

Bug Description

I got a segfault in my Python library from pyo3 trying to call PyBuffer::drop after the Python interpreter was already finalized. (See kylebarron/arro3#230).

Trying to acquire the GIL made this assert_ne! panic.

The similar C++-based Python library pyarrow performs a check for whether the interpreter is initialized before trying to drop the value.

I already worked around this in kylebarron/arro3#231 by checking if the interpreter is initialized before calling drop.
But perhaps this is something that should be fixed upstream in pyo3.

Steps to Reproduce

git clone https://github.com/kylebarron/arro3
cd arro3
git checkout 692e40bd557b9a5402bfc37df28c2b9d11afed53
# Install uv if necessary
# https://docs.astral.sh/uv/
uv run maturin develop -m arro3-core/Cargo.toml
uv run maturin develop -m arro3-compute/Cargo.toml
uv run maturin develop -m arro3-io/Cargo.toml
# Enter a Python environment
uv run python

Then run

import numpy as np
import pyarrow as pa
from arro3.core import Array

numpy_arr = np.array([0, 1, 2, 3], dtype=np.float64)
arro3_arr = Array(numpy_arr)
pyarrow_arr = pa.array(arro3_arr)

Then exit the Python interpreter, e.g. with control+D. You should see a segfault backtrace. This appears to be consistently reproducible for me.

Backtrace

thread '<unnamed>' panicked at /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.22.3/src/gil.rs:198:21:
assertion `left != right` failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled.

Consider calling `pyo3::prepare_freethreaded_python()` before attempting to use Python APIs.
  left: 0
 right: 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at core/src/panicking.rs:221:5:
panic in a function that cannot unwind
stack backtrace:
   0:        0x11889c274 - _PyInit__core
   1:        0x1188b4e28 - _PyInit__core
   2:        0x11889a8ec - _PyInit__core
   3:        0x11889d3f0 - _PyInit__core
   4:        0x11889cfa0 - _PyInit__core
   5:        0x11889de6c - _PyInit__core
   6:        0x11889d870 - _PyInit__core
   7:        0x11889c738 - _PyInit__core
   8:        0x11889d560 - _PyInit__core
   9:        0x1188de260 - _PyInit__core
  10:        0x1188de2d8 - _PyInit__core
  11:        0x1188de450 - _PyInit__core
  12:        0x1187a5620 - _PyInit__core
  13:        0x128018cf0 - __ZNSt3__120__shared_ptr_emplaceIN5arrow12_GLOBAL__N_117ImportedArrayDataENS_9allocatorIS3_EEE16__on_zero_sharedEv
  14:        0x12801ac3c - __ZN5arrow12_GLOBAL__N_114ImportedBufferD1Ev
  15:        0x128019d14 - __ZN5arrow9ArrayDataD2Ev
  16:        0x1281a4a00 - __ZN5arrow12NumericArrayINS_10DoubleTypeEED1Ev
  17:        0x105fc5b34 - __ZL36__pyx_tp_dealloc_7pyarrow_3lib_ArrayP7_object
  18:        0x105226690 - _dictkeys_decref
  19:        0x105228e6c - _dict_dealloc
  20:        0x10523938c - _module_clear
  21:        0x1053364b8 - _gc_collect_main
  22:        0x1053358f8 - __PyGC_CollectNoFail
  23:        0x10530f914 - _finalize_modules
  24:        0x10530ec9c - _Py_FinalizeEx
  25:        0x1053346c0 - _Py_RunMain
  26:        0x1053350c4 - _pymain_main
  27:        0x105335164 - _Py_BytesMain
thread caused non-unwinding panic. aborting.

Your operating system and version

MacOS Sonoma 14.6.1

Your Python version (python --version)

3.11.8

Your Rust version (rustc --version)

1.82.0

Your PyO3 version

0.22

How did you install python? Did you use a virtualenv?

Installed Python via pyenv. Virtualenv provided by uv.

Additional Info

No response

[davidhewitt]

It feels to me like we should consider changing PyBuffer::drop to be more like Py<T>::drop where we don't try to automatically acquire the GIL (this can in theory lead to deadlocks) and instead defer the release until we know we have the GIL? 🤔

[kylebarron]

In my own case I assume that the only instance where the interpreter isn't initialized is when the Python session is ending, and then I assume if I "leak" the memory, it'll still get cleaned up when the process closes.

[davidhewitt]

True; it's probably correct to add a check that the interpreter is initialized somewhere in that machinery. I think it's complicated a bit by the possibility of finalizing and restarting the interpreter (or a subinterpreter), but we don't really support that anyway (yet, probably not for a while).