Limited API support for Py_buffer

[tiran]

BPO	45459
Nosy	@pitrou, @vstinner, @tiran, @benjaminp, @alex, @encukou, @skrah, @pmp-p, @serhiy-storchaka, @rdb, @miss-islington, @erlend-aasland
PRs	bpo-45459: Add Py_buffer to limited API #29035 bpo-45459: Add Py_buffer to limited API #29991 bpo-45459: Rename buffer.h to pybuffer.h #31201 bpo-45459: Add pytypedefs.h header file #31527 bpo-45459: C API uses type names rather than structure names #31528 bpo-45459: Fix object.h: don't define PyObject type twice #31539 bpo-45459: Fix PyModuleDef_Slot type in the limited C API #31668 bpo-45459: Use type names in the internal C API #31669

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-02-25.23:25:59.650>
created_at = <Date 2021-10-13.09:55:13.771>
labels = ['expert-C-API', 'type-feature', '3.11']
title = 'Limited API support for Py_buffer'
updated_at = <Date 2022-03-03.22:08:16.420>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-03-03.22:08:16.420>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-25.23:25:59.650>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-10-13.09:55:13.771>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45459
keywords = ['patch']
message_count = 42.0
messages = ['403813', '403820', '403822', '403825', '403828', '404201', '404274', '404294', '404377', '404400', '404402', '404403', '404404', '404409', '404418', '405466', '405467', '406794', '408016', '408020', '408023', '408030', '408031', '408035', '408037', '408039', '408078', '408084', '408089', '412364', '412369', '412774', '412776', '413750', '413752', '413773', '413792', '413924', '413927', '414049', '414478', '414481']
nosy_count = 12.0
nosy_names = ['pitrou', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'alex', 'petr.viktorin', 'skrah', 'pmpp', 'serhiy.storchaka', 'rdb', 'miss-islington', 'erlendaasland']
pr_nums = ['29035', '29991', '31201', '31527', '31528', '31539', '31668', '31669']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45459'
versions = ['Python 3.11']

[tiran]

Currently all APIs related to Py_buffer are excluded from the limited API. It's neither possible to use Py_buffer from a C extension with limited API nor is it possible to define heap types with buffer support using the stable ABI.

The lack of Py_buffer support prevents prominent projects like NumPy or Pillow to use the limited API and produce abi3 binary wheel. To be fair it's not the only reason why these projects have not adopted the stable abi3 yet. Still Py_buffer is a necessary but not sufficient condition. Stable abi3 support would enable NumPy stack to build binary wheels that work with any Python version >= 3.11, < 4.0.

The limited API excludes any C API that references Py_buffer:

• 8 PyBuffer_*() functions
• 21 PyBUF_* constants
• PyMemoryView_FromBuffer()
• PyObject_GetBuffer
• Py_bf_getbuffer / Py_bf_releasebuffer type slots for PyBufferProcs

It should not be terribly complicated to add Py_buffer to the stable API. All it takes are an opaque struct definition of Py_buffer, an allocate function, a free function and a bunch of getters and setters. The hard part is to figure out which getters and setters are needed and how many struct members must be exposed by getters and setters. I recommend to get feedback from NumPy, Pillow, and Cython devs first.

Prototype
---------

typedef struct bufferinfo Py_buffer;

/* allocate a new Py_buffer object on the heap and initialize all members to NULL / 0 */
Py_buffer*
PyBuffer_New()
{
    Py_buffer *view = PyMem_Calloc(1, sizeof(Py_buffer));
    if (view == NULL) {
        PyErr_NoMemory();
    }
    return view;
}

/* convenience function */
Py_buffer*
PyBuffer_NewEx(PyObject *obj, void *buf,  Py_ssize_t len, Py_ssize_t itemsize,
               int readonly, int ndim, char *format, Py_ssize_t *shape, Py_ssize_t *strides,
               Py_ssize_t *suboffsets, void *internal)
{
    ...
}

/* release and free buffer */
void
PyBuffer_Free(Py_buffer *view)
{
    if (view != NULL) {
        PyBuffer_Release(view);
        PyMem_Free(view);
    }
}

[vstinner]

Py_buffer.shape requires a Py_ssize_t* pointer. It's not convenient. For example, the array module uses:

static int
array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags)
{
    ...
    if ((flags & PyBUF_ND)==PyBUF_ND) {
        view->shape = &((PyVarObject*)self)->ob_size;
    }
    ...
    return 0;
}

This code is not compatible with a fully opaque PyObject structure:
https://bugs.python.org/issue39573#msg401395

[serhiy-storchaka]

shape is a pointer to array of Py_ssize_t of size ndim. array and memoryview do a trick to avoid memory allocation, but _testbuffer.ndarray allocates it dynamically in the heap. We can add a small static buffer in Py_buffer to avoid additional memory allocation in common cases.

[tiran]

IIRC shape, strides, and suboffsets are all arrays of ndims length.

We could optimize allocation if we would require users to specify the value of ndims and don't allow them to change the value afterwards. PyBuffer_New(int ndims) then would allocate view of size sizeof(Py_buffer) + (3 * ndims * sizeof(Py_ssize_t *)). This would give us sufficient space to memcpy() shape, strides, and suboffsets arguments into memory after the Py_buffer struct.

[serhiy-storchaka]

ndim is not known before calling PyObject_GetBuffer(), so we will need a new API which combines PyObject_GetBuffer() and PyBuffer_New().

[tiran]

CC Antoine for his expertise of the buffer protocol

Opaque Py_Buffer and PyObject structs will require a different approach and prevent some optimizations. The consumer will have to malloc() a Py_buffer struct on the heap. In non-trivial cases the producer (exporter) may have to malloc() another blob and store it in Py_buffer.internal [1]. I'm not particularly worried about the performance of malloc here.

[1] https://docs.python.org/3/c-api/buffer.html?highlight=pybuffer#c.Py_buffer.internal

[serhiy-storchaka]

Py_buffer is often used for handling arguments if the function supports bytes, bytearray and other bytes-like objects. For example bytes.partition(). Any additional memory allocation would add significant overhead here. bytes.join() creates Py_buffer for every item, it would be a deoptimization if it would need to allocate them all separately.

We should allow to allocate Py_buffer on stack. Currently it has too complex structure and we cannot guarantee its stability (although there were no changes for years). I propose to split Py_buffer on transparent and opaque parts and standardize the transparent structure. It should include: obj, buf, len, possible flags (to distinguish read-only from writeable) and a pointer to opaque data. For bytes, bytearray, BytesIO, mmap and most other classes the pointer to opaque data is NULL. For array and memoryview objects the opaque data could be embedded into the object.

[encukou]

I recommend to get feedback from NumPy, Pillow, and Cython devs first.

Could you split this into two PRs: one to add the new API, and another to add things to the limited set?

There's no rush to add it to the limited API, esp. since it can't be tested with the intended consumers yet. And it's a hassle to remove mistakes from the limited API, even in alphas/betas.

To try this out I suggest making the struct opaque when something like _PyBuffer_IS_OPAQUE is #defined. Then you can let the compiler tell you what needs to change in those projects :)

[vstinner]

Maybe a PEP is needed to collect usages of the Py_buffer API and check if the ABI is future proof. A PEP may help to discuss with other projects which currently consume this API.

I suggest to start with the smallest possible API and then slowly extend it. It's too easy to make mistakes :-( Once it's added to the stable ABI, it will be really hard to change it.

For example, PyBuffer.format is a "char*", but who owns the string? For a stable ABI, I would suggest to duplicate the string.

For shape, stripes and suboffsets arrays, I would also suggest to allocate these arrays on the heap people to ensure that it cannot be modified from the outside.

In your PR, PyBuffer_GetLayout() gives indirectly access to the internal Py_buffer structure members and allows to modify them. One way is to avoid this issue is to return a *copy* of these arrays.

I would prefer to require to call "Set" functions to modify a Py_buffer to ensure that a buffer always remains consistency.

PyBuffer_NewEx(PyObject *obj, void *buf, Py_ssize_t len, Py_ssize_t itemsize, int readonly, int ndim, char *format, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t *suboffsets, void *internal)

This API looks like PyCode_New() which was broken *often* so it looks like a bad pattern for a stable ABI.

Maybe PyBuffer_New() + many Set() functions would be more future proof.

But I don't know which Py_buffer members are mandatory to have a "valid" buffer.

What if tomorrow we add new members. Will it be possible to initalize them to a reasonable default value?