Get rid of specialization

[konstin]

The last step for #5 is getting rid of specialization. I've remove most uses in 7c0379b and subsequent commits, except for the following 17 10 9 7 4: rust-lang/rust#64564 changed the rules so there are again many default fn.

src/class/descr.rs
76:    default fn tp_descr_get() -> Option<ffi::descrgetfunc> {
97:    default fn tp_descr_set() -> Option<ffi::descrsetfunc> {
130:    default fn tp_as_descr(_type_object: &mut ffi::PyTypeObject) {}

src/class/buffer.rs
46:    default fn tp_as_buffer() -> Option<ffi::PyBufferProcs> {
74:    default fn cb_bf_getbuffer() -> Option<ffi::getbufferproc> {
110:    default fn cb_bf_releasebuffer() -> Option<ffi::releasebufferproc> {

src/class/gc.rs
27:    default fn update_type_object(_type_object: &mut ffi::PyTypeObject) {}
72:    default fn tp_traverse() -> Option<ffi::traverseproc> {
124:    default fn tp_clear() -> Option<ffi::inquiry> {

src/class/iter.rs
49:    default fn tp_as_iter(_typeob: &mut ffi::PyTypeObject) {}
71:    default fn tp_iter() -> Option<ffi::getiterfunc> {
94:    default fn tp_iternext() -> Option<ffi::iternextfunc> {

src/class/basic.rs
152:    default fn tp_as_object(_type_object: &mut ffi::PyTypeObject) {}
153:    default fn nb_bool_fn() -> Option<ffi::inquiry> {
183:    default fn tp_getattro() -> Option<ffi::binaryfunc> {
252:        default fn set_attr() -> Option<ffi::setattrofunc> {
274:        default fn del_attr() -> Option<ffi::setattrofunc> {
296:        default fn set_del_attr() -> Option<ffi::setattrofunc> {
324:    default fn tp_str() -> Option<ffi::unaryfunc> {
344:    default fn tp_repr() -> Option<ffi::unaryfunc> {
364:    default fn tp_hash() -> Option<ffi::hashfunc> {
389:    default fn nb_bool() -> Option<ffi::inquiry> {
409:    default fn tp_richcompare() -> Option<ffi::richcmpfunc> {

src/class/mapping.rs
83:    default fn tp_as_mapping() -> Option<ffi::PyMappingMethods> {
116:    default fn mp_length() -> Option<ffi::lenfunc> {
139:    default fn mp_subscript() -> Option<ffi::binaryfunc> {
162:    default fn mp_ass_subscript() -> Option<ffi::objobjargproc> {
187:    default fn mp_del_subscript() -> Option<ffi::objobjargproc> {
201:    default fn det_set_dispatch() -> Option<ffi::objobjargproc> {

src/class/pyasync.rs
94:    default fn tp_as_async() -> Option<ffi::PyAsyncMethods> {
121:    default fn am_await() -> Option<ffi::unaryfunc> {
144:    default fn am_aiter() -> Option<ffi::unaryfunc> {
167:    default fn am_anext() -> Option<ffi::unaryfunc> {

src/conversion.rs
107:    default fn with_borrowed_ptr<F, R>(&self, py: Python, f: F) -> R

src/class/sequence.rs
135:    default fn tp_as_sequence() -> Option<ffi::PySequenceMethods> {
168:    default fn sq_length() -> Option<ffi::lenfunc> {
190:    default fn sq_item() -> Option<ffi::ssizeargfunc> {
239:        default fn set_item() -> Option<ffi::ssizeobjargproc> {
285:        default fn del_item() -> Option<ffi::ssizeobjargproc> {
328:        default fn del_set_item() -> Option<ffi::ssizeobjargproc> {
372:    default fn sq_contains() -> Option<ffi::objobjproc> {
394:    default fn sq_concat() -> Option<ffi::binaryfunc> {
416:    default fn sq_repeat() -> Option<ffi::ssizeargfunc> {
438:    default fn sq_inplace_concat() -> Option<ffi::binaryfunc> {
465:    default fn sq_inplace_repeat() -> Option<ffi::ssizeargfunc> {

src/class/number.rs
625:    default fn tp_as_number() -> Option<ffi::PyNumberMethods> {
692:    default fn nb_add() -> Option<ffi::binaryfunc> {
714:    default fn nb_subtract() -> Option<ffi::binaryfunc> {
736:    default fn nb_multiply() -> Option<ffi::binaryfunc> {
758:    default fn nb_matrix_multiply() -> Option<ffi::binaryfunc> {
780:    default fn nb_true_divide() -> Option<ffi::binaryfunc> {
802:    default fn nb_floor_divide() -> Option<ffi::binaryfunc> {
824:    default fn nb_remainder() -> Option<ffi::binaryfunc> {
846:    default fn nb_divmod() -> Option<ffi::binaryfunc> {
868:    default fn nb_power() -> Option<ffi::ternaryfunc> {
890:    default fn nb_lshift() -> Option<ffi::binaryfunc> {
912:    default fn nb_rshift() -> Option<ffi::binaryfunc> {
934:    default fn nb_and() -> Option<ffi::binaryfunc> {
956:    default fn nb_xor() -> Option<ffi::binaryfunc> {
978:    default fn nb_or() -> Option<ffi::binaryfunc> {
1000:    default fn nb_inplace_add() -> Option<ffi::binaryfunc> {
1022:    default fn nb_inplace_subtract() -> Option<ffi::binaryfunc> {
1044:    default fn nb_inplace_multiply() -> Option<ffi::binaryfunc> {
1066:    default fn nb_inplace_matrix_multiply() -> Option<ffi::binaryfunc> {
1088:    default fn nb_inplace_true_divide() -> Option<ffi::binaryfunc> {
1110:    default fn nb_inplace_floor_divide() -> Option<ffi::binaryfunc> {
1132:    default fn nb_inplace_remainder() -> Option<ffi::binaryfunc> {
1154:    default fn nb_inplace_power() -> Option<ffi::ternaryfunc> {
1176:    default fn nb_inplace_lshift() -> Option<ffi::binaryfunc> {
1198:    default fn nb_inplace_rshift() -> Option<ffi::binaryfunc> {
1220:    default fn nb_inplace_and() -> Option<ffi::binaryfunc> {
1242:    default fn nb_inplace_xor() -> Option<ffi::binaryfunc> {
1264:    default fn nb_inplace_or() -> Option<ffi::binaryfunc> {
1287:    default fn nb_add_fallback() -> Option<ffi::binaryfunc> {
1309:    default fn nb_sub_fallback() -> Option<ffi::binaryfunc> {
1331:    default fn nb_mul_fallback() -> Option<ffi::binaryfunc> {
1353:    default fn nb_matmul_fallback() -> Option<ffi::binaryfunc> {
1375:    default fn nb_truediv_fallback() -> Option<ffi::binaryfunc> {
1397:    default fn nb_floordiv_fallback() -> Option<ffi::binaryfunc> {
1419:    default fn nb_mod_fallback() -> Option<ffi::binaryfunc> {
1441:    default fn nb_divmod_fallback() -> Option<ffi::binaryfunc> {
1463:    default fn nb_pow_fallback() -> Option<ffi::ternaryfunc> {
1485:    default fn nb_lshift_fallback() -> Option<ffi::binaryfunc> {
1507:    default fn nb_rshift_fallback() -> Option<ffi::binaryfunc> {
1529:    default fn nb_and_fallback() -> Option<ffi::binaryfunc> {
1551:    default fn nb_xor_fallback() -> Option<ffi::binaryfunc> {
1573:    default fn nb_or_fallback() -> Option<ffi::binaryfunc> {
1595:    default fn nb_negative() -> Option<ffi::unaryfunc> {
1618:    default fn nb_positive() -> Option<ffi::unaryfunc> {
1640:    default fn nb_absolute() -> Option<ffi::unaryfunc> {
1662:    default fn nb_invert() -> Option<ffi::unaryfunc> {
1684:    default fn nb_int() -> Option<ffi::unaryfunc> {
1706:    default fn nb_float() -> Option<ffi::unaryfunc> {
1728:    default fn nb_index() -> Option<ffi::unaryfunc> {

src/instance.rs
387:    default fn to_managed_py_ref<'p>(&self, py: Python<'p>) -> ManagedPyRef<'p, Self> {
396:    default fn drop_impl(borrowed: &mut ManagedPyRef<Self>) {

src/types/sequence.rs
268:                default fn extract(obj: &'a PyAny) -> PyResult<Self> {
307:    default fn extract(obj: &'a PyAny) -> PyResult<Self> {

Once those are removed, we pyo3 can be stable on rust 1.30! I'm happy to take pull requests tackling one or more of those default fn uses

[ijl]

The codegen in pyo3-derive-backend is a problem once the above functions are removed. For example:

pyo3/pyo3-derive-backend/src/py_class.rs

Lines 329 to 336 in 5100676

	impl ::pyo3::class::methods::PyPropMethodsProtocolImpl for #cls {
	fn py_methods() -> &'static [::pyo3::class::PyMethodDefType] {
	static METHODS: &'static [::pyo3::class::PyMethodDefType] = &[
	#(#py_methods),*
	];
	METHODS
	}
	}

pyo3/pyo3-derive-backend/src/py_impl.rs

Lines 33 to 40 in 264f642

	impl ::pyo3::class::methods::PyMethodsProtocolImpl for #ty {
	fn py_methods() -> &'static [::pyo3::class::PyMethodDefType] {
	static METHODS: &'static [::pyo3::class::PyMethodDefType] = &[
	#(#methods),*
	];
	METHODS
	}
	}

I am not familiar enough with it to have a solution.

[konstin]

I've also already ran against that and haven't found a good solution either :/

I think the most viable way forward would be not to have a default implementation of PyMethodsProtocolImpl, but instead have #[pymethods] implement it. For cases where there is no pymethods block, one could use #[pyclass(methods = false)] which then implements py_methods().

[konstin]

#332 fixes the PyMethodsProtocolImpl problem

[bspeice]

Is this still an issue given #332 being merged? I'm interested in helping push this to stable, and would appreciate advice for where to start.

EDIT: It looks like the first two instances in conversion are trivially removable, and I'm not sure what the implications are for removing the attempted buffer.

[kngwyu]

@bspeice
I think now the problem is around #426.
I'm looking for a better design but have not come up with it.

[konstin]

@bspeice #210 (comment) is fixed by #332, while some specialization are still there.

The main question is how we can rewrite the following so we don't have a conflicting implementation without specialization anymore:

pub trait FooImpl {
    fn foo() -> Option<ffi::Foo> {
        None
    }
}

impl<T> FooImpl for T {}

impl<'p, T> FooImpl for T
where
    T: Bar<'p>,
{
    fn foo() -> Option<ffi::PySequenceMethods> {
        return bar();
    }
}

(You can see a lot of those in the error message by removing #![feature(specialization)] from lib.rs)

[Alphare]

As a contributor to Mercurial where we started using Rust a few months ago, I can say that we very much look forward to using PyO3, and the progress towards a build on stable is really encouraging.
However I wonder if that last issue is fixable at all, considering that it is exactly the use case for specialization.

[konstin]

It is technically fixable. We mainly have two options:

Inventory

Instead of implementing traits, the protocol implementations are submit to an inventory and added collected at class initialization time. The disadvantage here is that this essentially a well established linker hack, which can cause some weird interactions with the c api (e.g. #341).

Inventory is already used for pymethods blocks (so that we can have 0 to n pymethods blocks).

Manual annotations

You could force explicit annotations on pyclass blocks about which protocols will be implemented. The pyclass block would add default implementations for all protocol you didn't specify. If we wanted to completely get rid of inventory, we'd also need to forbid multiple pymethod block (or add an awkward syntax for numbering them) and annotate whether a struct has one or zero pymethod blocks.