Can I change the signature of `ob_item` of PyTupleObject and PyListObject?

[Fidget-Spinner]

I want to change

typedef struct {
    PyObject_VAR_HEAD
    PyObject **ob_item;
    Py_ssize_t allocated;
} PyListObject;

to

typedef struct {
    PyObject_VAR_HEAD
    _PyHeapRef *ob_item;
    Py_ssize_t allocated;
} PyListObject;

There is no change in size of the struct or offsets, _PyHeapRef is an opaque struct.

Does this require a PEP?

Discourse issue here https://discuss.python.org/t/open-question-can-we-change-the-field-type-of-pytupleobject-and-pylistobject/91318

I mainly want to do this to supported tagged integers in CPython list and tuples.

Vote

• @encukou
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @vstinner
• @zooba

[vstinner]

IMO too many C extensions currently access directly to ob_item, either via the ob_item member or by abusing &PyList_GET_ITEM(list, 0) and &PyTuple_GET_ITEM(tuple, 0) macros. You cannot change the type from PyObject** to _PyHeapRef*.

When I wrote PEP 674 (Disallow using macros as l-values), I left &PyList_GET_ITEM(list, 0) and &PyTuple_GET_ITEM(tuple, 0) unchanged because it's a common pattern: https://peps.python.org/pep-0674/#pytuple-get-item-and-pylist-get-item-are-left-unchanged

I proposed an API to replace direct access to PyObject**, but I failed to convince:

• https://mail.python.org/archives/list/python-dev@python.org/thread/632CV42376SWVYAZTHG4ROOV2HRHOVZ7/
• https://peps.python.org/pep-0620/#avoid-functions-returning-pyobject
• (I failed to find other previous discussions about this topic.)

Does this require a PEP?

I would say yes, but be prepared to get it rejected.

I suggest to conduct a study to see how ob_item is used in the wild, and list impacted projects.

Also, consider designing an API to still give a direct access to an array of PyObject** since it's convenient. The API would convert PyObject* to _PyHeapRef once complete.

[Fidget-Spinner]

Thanks Victor, I sort of expected your answer.

An alternative proposal I thought of is to store the tagged integer directly in the PyObject pointer. However that would mean possible segfault if dereferencing an object that is obtained directly via ob_item. Do you know if enough people use PyTuple/List_GET_ITEM? I can make that safe but not the other ones.

[encukou]

Yes, this would need a PEP.

Using ((PyListObject*)(obj))->ob_item[n] directly is, unfortunately, public API covered by PEP 387. It can be deprecated, if you don't mind your work blocked for years (and in this case, even the recommended 5 years might be too short).

Personally, I like your workaround; and I'd aim for later exposing a new PyRef-based API.

[Fidget-Spinner]

Ok. Makes sense to me. Thank you Petr and Victor!