Skip to content

Commit

Permalink
add second lifetime to PyRef and PyRefMut
Browse files Browse the repository at this point in the history
  • Loading branch information
Icxolu committed Nov 19, 2024
1 parent 2e2cf36 commit 1da83f0
Show file tree
Hide file tree
Showing 31 changed files with 662 additions and 573 deletions.
68 changes: 34 additions & 34 deletions guide/src/class.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

PyO3 exposes a group of attributes powered by Rust's proc macro system for defining Python classes as Rust structs.

The main attribute is `#[pyclass]`, which is placed upon a Rust `struct` or `enum` to generate a Python type for it. They will usually also have *one* `#[pymethods]`-annotated `impl` block for the struct, which is used to define Python methods and constants for the generated Python type. (If the [`multiple-pymethods`] feature is enabled, each `#[pyclass]` is allowed to have multiple `#[pymethods]` blocks.) `#[pymethods]` may also have implementations for Python magic methods such as `__str__`.
The main attribute is `#[pyclass]`, which is placed upon a Rust `struct` or `enum` to generate a Python type for it. They will usually also have _one_ `#[pymethods]`-annotated `impl` block for the struct, which is used to define Python methods and constants for the generated Python type. (If the [`multiple-pymethods`] feature is enabled, each `#[pyclass]` is allowed to have multiple `#[pymethods]` blocks.) `#[pymethods]` may also have implementations for Python magic methods such as `__str__`.

This chapter will discuss the functionality and configuration these attributes offer. Below is a list of links to the relevant section of this chapter for each:

Expand All @@ -22,6 +22,7 @@ This chapter will discuss the functionality and configuration these attributes o
## Defining a new class

To define a custom Python class, add the `#[pyclass]` attribute to a Rust struct or enum.

```rust
# #![allow(dead_code)]
use pyo3::prelude::*;
Expand Down Expand Up @@ -122,7 +123,8 @@ create_interface!(FloatClass, String);
#### Must be thread-safe

Python objects are freely shared between threads by the Python interpreter. This means that:
- Python objects may be created and destroyed by different Python threads; therefore #[pyclass]` objects must be `Send`.

- Python objects may be created and destroyed by different Python threads; therefore #[pyclass]`objects must be`Send`.
- Python objects may be accessed by multiple python threads simultaneously; therefore `#[pyclass]` objects must be `Sync`.

For now, don't worry about these requirements; simple classes will already be thread-safe. There is a [detailed discussion on thread-safety](./class/thread-safety.md) later in the guide.
Expand Down Expand Up @@ -202,13 +204,14 @@ fn my_module(m: &Bound<'_, PyModule>) -> PyResult<()> {

Often is useful to turn a `#[pyclass]` type `T` into a Python object and access it from Rust code. The [`Py<T>`] and [`Bound<'py, T>`] smart pointers are the ways to represent a Python object in PyO3's API. More detail can be found about them [in the Python objects](./types.md#pyo3s-smart-pointers) section of the guide.

Most Python objects do not offer exclusive (`&mut`) access (see the [section on Python's memory model](./python-from-rust.md#pythons-memory-model)). However, Rust structs wrapped as Python objects (called `pyclass` types) often *do* need `&mut` access. Due to the GIL, PyO3 *can* guarantee exclusive access to them.
Most Python objects do not offer exclusive (`&mut`) access (see the [section on Python's memory model](./python-from-rust.md#pythons-memory-model)). However, Rust structs wrapped as Python objects (called `pyclass` types) often _do_ need `&mut` access. Due to the GIL, PyO3 _can_ guarantee exclusive access to them.

The Rust borrow checker cannot reason about `&mut` references once an object's ownership has been passed to the Python interpreter. This means that borrow checking is done at runtime using with a scheme very similar to `std::cell::RefCell<T>`. This is known as [interior mutability](https://doc.rust-lang.org/book/ch15-05-interior-mutability.html).

Users who are familiar with `RefCell<T>` can use `Py<T>` and `Bound<'py, T>` just like `RefCell<T>`.

For users who are not very familiar with `RefCell<T>`, here is a reminder of Rust's rules of borrowing:

- At any given time, you can have either (but not both of) one mutable reference or any number of immutable references.
- References can never outlast the data they refer to.

Expand Down Expand Up @@ -309,11 +312,11 @@ Generally, `#[new]` methods have to return `T: Into<PyClassInitializer<Self>>` o
For constructors that may fail, you should wrap the return type in a PyResult as well.
Consult the table below to determine which type your constructor should return:

| | **Cannot fail** | **May fail** |
|-----------------------------|---------------------------|-----------------------------------|
|**No inheritance** | `T` | `PyResult<T>` |
|**Inheritance(T Inherits U)**| `(T, U)` | `PyResult<(T, U)>` |
|**Inheritance(General Case)**| [`PyClassInitializer<T>`] | `PyResult<PyClassInitializer<T>>` |
| | **Cannot fail** | **May fail** |
| ----------------------------- | ------------------------- | --------------------------------- |
| **No inheritance** | `T` | `PyResult<T>` |
| **Inheritance(T Inherits U)** | `(T, U)` | `PyResult<(T, U)>` |
| **Inheritance(General Case)** | [`PyClassInitializer<T>`] | `PyResult<PyClassInitializer<T>>` |

## Inheritance

Expand All @@ -323,7 +326,6 @@ Currently, only classes defined in Rust and builtins provided by PyO3 can be inh
from; inheriting from other classes defined in Python is not yet supported
([#991](https://github.com/PyO3/pyo3/issues/991)).


For convenience, `(T, U)` implements `Into<PyClassInitializer<T>>` where `U` is the
base class of `T`.
But for a more deeply nested inheritance, you have to return `PyClassInitializer<T>`
Expand Down Expand Up @@ -370,7 +372,7 @@ impl SubClass {
(SubClass { val2: 15 }, BaseClass::new())
}

fn method2(self_: PyRef<'_, Self>) -> PyResult<usize> {
fn method2(self_: PyRef<'_, '_, Self>) -> PyResult<usize> {
let super_ = self_.as_super(); // Get &PyRef<BaseClass>
super_.method1().map(|x| x * self_.val2)
}
Expand All @@ -388,24 +390,24 @@ impl SubSubClass {
PyClassInitializer::from(SubClass::new()).add_subclass(SubSubClass { val3: 20 })
}

fn method3(self_: PyRef<'_, Self>) -> PyResult<usize> {
fn method3(self_: PyRef<'_, '_, Self>) -> PyResult<usize> {
let base = self_.as_super().as_super(); // Get &PyRef<'_, BaseClass>
base.method1().map(|x| x * self_.val3)
}

fn method4(self_: PyRef<'_, Self>) -> PyResult<usize> {
fn method4(self_: PyRef<'_, '_, Self>) -> PyResult<usize> {
let v = self_.val3;
let super_ = self_.into_super(); // Get PyRef<'_, SubClass>
SubClass::method2(super_).map(|x| x * v)
}

fn get_values(self_: PyRef<'_, Self>) -> (usize, usize, usize) {
fn get_values(self_: PyRef<'_, '_, Self>) -> (usize, usize, usize) {
let val1 = self_.as_super().as_super().val1;
let val2 = self_.as_super().val2;
(val1, val2, self_.val3)
}

fn double_values(mut self_: PyRefMut<'_, Self>) {
fn double_values(mut self_: PyRefMut<'_, '_, Self>) {
self_.as_super().as_super().val1 *= 2;
self_.as_super().val2 *= 2;
self_.val3 *= 2;
Expand Down Expand Up @@ -481,6 +483,7 @@ impl DictWithCounter {
```

If `SubClass` does not provide a base class initialization, the compilation fails.

```rust,compile_fail
# use pyo3::prelude::*;
Expand All @@ -504,7 +507,7 @@ impl SubClass {
```

The `__new__` constructor of a native base class is called implicitly when
creating a new instance from Python. Be sure to accept arguments in the
creating a new instance from Python. Be sure to accept arguments in the
`#[new]` method that you want the base class to get, even if they are not used
in that `fn`:

Expand Down Expand Up @@ -542,6 +545,7 @@ initial items, such as `MyDict(item_sequence)` or `MyDict(a=1, b=2)`.
## Object properties

PyO3 supports two ways to add properties to your `#[pyclass]`:

- For simple struct fields with no side effects, a `#[pyo3(get, set)]` attribute can be added directly to the field definition in the `#[pyclass]`.
- For properties which require computation you can define `#[getter]` and `#[setter]` functions in the [`#[pymethods]`](#instance-methods) block.

Expand All @@ -566,6 +570,7 @@ The above would make the `num` field available for reading and writing as a `sel
Properties can be readonly or writeonly by using just `#[pyo3(get)]` or `#[pyo3(set)]` respectively.

To use these annotations, your field type must implement some conversion traits:

- For `get` the field type must implement both `IntoPy<PyObject>` and `Clone`.
- For `set` the field type must implement `FromPyObject`.

Expand Down Expand Up @@ -733,15 +738,16 @@ impl MyClass {

Declares a class method callable from Python.

* The first parameter is the type object of the class on which the method is called.
- The first parameter is the type object of the class on which the method is called.
This may be the type object of a derived class.
* The first parameter implicitly has type `&Bound<'_, PyType>`.
* For details on `parameter-list`, see the documentation of `Method arguments` section.
* The return type must be `PyResult<T>` or `T` for some `T` that implements `IntoPy<PyObject>`.
- The first parameter implicitly has type `&Bound<'_, PyType>`.
- For details on `parameter-list`, see the documentation of `Method arguments` section.
- The return type must be `PyResult<T>` or `T` for some `T` that implements `IntoPy<PyObject>`.

### Constructors which accept a class argument

To create a constructor which takes a positional class argument, you can combine the `#[classmethod]` and `#[new]` modifiers:

```rust
# #![allow(dead_code)]
# use pyo3::prelude::*;
Expand Down Expand Up @@ -807,7 +813,7 @@ Python::with_gil(|py| {
```

> Note: if the method has a `Result` return type and returns an `Err`, PyO3 will panic during
class creation.
> class creation.
If the class attribute is defined with `const` code only, one can also annotate associated
constants:
Expand Down Expand Up @@ -844,7 +850,7 @@ fn increment_field(my_class: &mut MyClass) {
// Take a reference wrapper when borrowing should be automatic,
// but interaction with the underlying `Bound` is desired.
#[pyfunction]
fn print_field(my_class: PyRef<'_, MyClass>) {
fn print_field(my_class: PyRef<'_, '_, MyClass>) {
println!("{}", my_class.my_field);
}

Expand Down Expand Up @@ -1193,7 +1199,7 @@ Python::with_gil(|py| {
```

Ordering of enum variants is optionally added using `#[pyo3(ord)]`.
*Note: Implementation of the `PartialOrd` trait is required when passing the `ord` argument. If not implemented, a compile time error is raised.*
_Note: Implementation of the `PartialOrd` trait is required when passing the `ord` argument. If not implemented, a compile time error is raised._

```rust
# use pyo3::prelude::*;
Expand Down Expand Up @@ -1390,22 +1396,22 @@ impl pyo3::PyClass for MyClass {
type Frozen = pyo3::pyclass::boolean_struct::False;
}

impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a MyClass
impl<'a, 'holder, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'holder, 'py> for &'holder MyClass
{
type Holder = ::std::option::Option<pyo3::PyRef<'py, MyClass>>;
type Holder = ::std::option::Option<pyo3::PyRef<'a, 'py, MyClass>>;

#[inline]
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'a mut Self::Holder) -> pyo3::PyResult<Self> {
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'holder mut Self::Holder) -> pyo3::PyResult<Self> {
pyo3::impl_::extract_argument::extract_pyclass_ref(obj, holder)
}
}

impl<'a, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'py> for &'a mut MyClass
impl<'a, 'holder, 'py> pyo3::impl_::extract_argument::PyFunctionArgument<'a, 'holder, 'py> for &'holder mut MyClass
{
type Holder = ::std::option::Option<pyo3::PyRefMut<'py, MyClass>>;
type Holder = ::std::option::Option<pyo3::PyRefMut<'a, 'py, MyClass>>;

#[inline]
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'a mut Self::Holder) -> pyo3::PyResult<Self> {
fn extract(obj: &'a pyo3::Bound<'py, PyAny>, holder: &'holder mut Self::Holder) -> pyo3::PyResult<Self> {
pyo3::impl_::extract_argument::extract_pyclass_ref_mut(obj, holder)
}
}
Expand Down Expand Up @@ -1459,22 +1465,16 @@ impl pyo3::impl_::pyclass::PyClassImpl for MyClass {
# }
```


[`PyTypeInfo`]: {{#PYO3_DOCS_URL}}/pyo3/type_object/trait.PyTypeInfo.html

[`Py`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Py.html
[`Bound<'_, T>`]: {{#PYO3_DOCS_URL}}/pyo3/struct.Bound.html
[`PyClass`]: {{#PYO3_DOCS_URL}}/pyo3/pyclass/trait.PyClass.html
[`PyRef`]: {{#PYO3_DOCS_URL}}/pyo3/pycell/struct.PyRef.html
[`PyRefMut`]: {{#PYO3_DOCS_URL}}/pyo3/pycell/struct.PyRefMut.html
[`PyClassInitializer<T>`]: {{#PYO3_DOCS_URL}}/pyo3/pyclass_init/struct.PyClassInitializer.html

[`Arc`]: https://doc.rust-lang.org/std/sync/struct.Arc.html
[`RefCell`]: https://doc.rust-lang.org/std/cell/struct.RefCell.html

[classattr]: https://docs.python.org/3/tutorial/classes.html#class-and-instance-variables

[`multiple-pymethods`]: features.md#multiple-pymethods

[lifetime-elision]: https://doc.rust-lang.org/reference/lifetime-elision.html
[compiler-error-e0106]: https://doc.rust-lang.org/error_codes/E0106.html
16 changes: 10 additions & 6 deletions guide/src/class/numeric.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
At this point we have a `Number` class that we can't actually do any math on!

Before proceeding, we should think about how we want to handle overflows. There are three obvious solutions:

- We can have infinite precision just like Python's `int`. However that would be quite boring - we'd
be reinventing the wheel.
be reinventing the wheel.
- We can raise exceptions whenever `Number` overflows, but that makes the API painful to use.
- We can wrap around the boundary of `i32`. This is the approach we'll take here. To do that we'll just forward to `i32`'s
`wrapping_*` methods.
`wrapping_*` methods.

### Fixing our constructor

Expand Down Expand Up @@ -42,6 +43,7 @@ fn wrap(obj: &Bound<'_, PyAny>) -> PyResult<i32> {
Ok(val as i32)
}
```

We also add documentation, via `///` comments, which are visible to Python users.

```rust
Expand All @@ -68,8 +70,8 @@ impl Number {
}
```


With that out of the way, let's implement some operators:

```rust
use pyo3::exceptions::{PyZeroDivisionError, PyValueError};

Expand Down Expand Up @@ -132,7 +134,7 @@ impl Number {
#
#[pymethods]
impl Number {
fn __pos__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> {
fn __pos__<'a, 'py>(slf: PyRef<'a, 'py, Self>) -> PyRef<'a, 'py, Self> {
slf
}

Expand Down Expand Up @@ -178,7 +180,7 @@ impl Number {

We do not implement the in-place operations like `__iadd__` because we do not wish to mutate `Number`.
Similarly we're not interested in supporting operations with different types, so we do not implement
the reflected operations like `__radd__` either.
the reflected operations like `__radd__` either.

Now Python can use our `Number` class:

Expand Down Expand Up @@ -405,13 +407,15 @@ function that does:
unsigned long PyLong_AsUnsignedLongMask(PyObject *obj)
```
We can call this function from Rust by using [`pyo3::ffi::PyLong_AsUnsignedLongMask`]. This is an *unsafe*
We can call this function from Rust by using [`pyo3::ffi::PyLong_AsUnsignedLongMask`]. This is an _unsafe_
function, which means we have to use an unsafe block to call it and take responsibility for upholding
the contracts of this function. Let's review those contracts:
- The GIL must be held. If it's not, calling this function causes a data race.
- The pointer must be valid, i.e. it must be properly aligned and point to a valid Python object.
Let's create that helper function. The signature has to be `fn(&Bound<'_, PyAny>) -> PyResult<T>`.
- `&Bound<'_, PyAny>` represents a checked borrowed reference, so the pointer derived from it is valid (and not null).
- Whenever we have borrowed references to Python objects in scope, it is guaranteed that the GIL is held. This reference is also where we can get a [`Python`] token to use in our call to [`PyErr::take`].
Expand Down
Loading

0 comments on commit 1da83f0

Please sign in to comment.