[WIP] bpo-35199: PyTuple_GET_ITEM() becomes a function in debug mode

[vstinner]

When Python is compiled in debug mode (Py_DEBUG), PyTuple_GET_ITEM()
macro becomes a function which implements sanity checks using
_PyObject_ASSERT() and assert(). It should help to detect misusage of
the C API.

https://bugs.python.org/issue35199

[vstinner]

@serhiy-storchaka, @pablogsal, @pitrou, @nascheme: That's a first change to show what I would like to do with the C API. What do you think?

[vstinner]

This change breaks the backward compatibility when a C extension is compiled with Py_DEBUG. So I'm not sure that it should be merged into Python 3.8. Maybe we should add a new opt-in experimental option to enable it, instead of Py_DEBUG?

.

[vstinner]

I had to modify Python to avoid "&PyTuple_GET_ITEM(op, i)": commit d17a693.

[pitrou]

That looks like a bad idea to me. People who use PyTuple_GET_ITEM know what they are getting.
Also in my experience it's extremely rare to index a tuple out of bounds by mistake.

[serhiy-storchaka]

This will break the code that:

1. Uses &PyTuple_GET_ITEM(obj, 0).
2. First set tuple size to smaller value, then call PyTuple_SET_ITEM(), and finally increase the tuple size to necessary limit.

[pablogsal]

First set tuple size to smaller value, then call PyTuple_SET_ITEM(), and finally increase the tuple size to necessary limit.

Even if this is "valid" shouldn't this pattern be disregarded? I mean, the code that calls PyTuple_SET_ITEM must promise that it will call PyTuple_Resize at some point in the future, so why not doing it before the PyTuple_SET_ITEM?

Maybe I don't understand the comment, do you mean for example setting the tuple to size 1, setting the 2nd item and then resizing to size 2, right?

[nascheme]

Hello Victor,

I think I agree with Antoine and Serhiy that changing a macro like this to a function is not a good idea. I support the general idea of what you are trying to do and I don't want to discourage you from trying to improve the API.

Your C-API project has two objectives. First, you would like the make the Python limited API more useful (really it is a stable ABI). Second objective is to avoid leaking implementation details via the API. That will allow alternative Python's like PyPy to use extensions written for CPython without huge performance penalties. PyTuple_GET_ITEM is an example of leaking too many implementation details. E.g. the callers get back something that can be used as an l-value.

For external users of the C-API, I think you want three options:

A) use backwards compatible API (what you get now if you don't define Py_LIMITED_API)

B) use limited API, have stable ABI (define Py_LIMITED_API)

C) use new API that doesn't expose implementation (potentially unstable ABI).

We don't have option C now. Option B is available but the API coverage is small.

PyTuple_GET_ITEM would still be available for option A. It must still be usable as an l-value for example, to avoid code breakage. It is already hidden if the limited API is enabled (option B). For option C, I think we need a new API. I don't know what you call it but it would be high performance version of PyTuple_GET_ITEM() without exposing implementation. So, it would probably be an inline function. If option C is enabled, you would not have access to PyTuple_GET_ITEM.

I see a major task of the C-API project would be to implement the support for building extensions with option C enabled. You would have to provide an API like PyTuple_GET_ITEM but that is an inline function. To avoid breaking a lot of code, you have to avoid changing PyTuple_GET_ITEM.

[vstinner]

Antoine:

Also in my experience it's extremely rare to index a tuple out of bounds by mistake.

One of my friend contacted me for a segfault today. He was reading out of the bound of a tuple :-)

Serhiy:

This will break the code that:

• Uses &PyTuple_GET_ITEM(obj, 0).

Yes, I described it in the NEWS entry :-) I wrote this PR to ask if we can affort to break the C API. It seems like it's a strong no. Ok :-)

For external users of the C-API, I think you want three options:
A) use backwards compatible API (what you get now if you don't define Py_LIMITED_API)
B) use limited API, have stable ABI (define Py_LIMITED_API)
C) use new API that doesn't expose implementation (potentially unstable ABI).

I wasn't sure if this PR was in (A) or (B), but since it breaks the backward compatibility, it seems to close strictly in (C): a new C API.

It seems like abusing the existing Py_DEBUG define is a bad idea. I should instead do something similar to my "proof of concept" implementation of my new C API:
https://github.com/pythoncapi/cpython/

=> use a new define to opt-in for this new C API

I will now close this PR, because I dislike discussing such very large changes on a pull request. I will instead discuss on python-dev. But I should think about how do to that. Have a plan :-)

[vstinner]

I created https://bugs.python.org/issue35206 "Add a new experimental _Py_CAPI2 API" and sent an email to python-dev: https://mail.python.org/pipermail/python-dev/2018-November/155702.html [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged).