[C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)

[vstinner]

BPO	43688
Nosy	@vstinner, @encukou
PRs	bpo-43688: Support the limited C API in debug mode #25131 bpo-43688: Support "make regen-limited-abi" in debug mode #25133 bpo-43688: Run make regen-limited-abi #25134 bpo-43688: Fix Py_LIMITED_API version of xxlimited #25135
Files	bench_limited.py bench_testcapi.py bench.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-26.22:20:07.057>
created_at = <Date 2021-04-01.06:47:31.015>
labels = ['expert-C-API', '3.10']
title = '[C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)'
updated_at = <Date 2021-05-26.22:20:07.056>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-26.22:20:07.056>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-26.22:20:07.057>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-04-01.06:47:31.015>
creator = 'vstinner'
dependencies = []
files = ['49925', '49926', '49927']
hgrepos = []
issue_num = 43688
keywords = ['patch']
message_count = 12.0
messages = ['389956', '389960', '389962', '389971', '389982', '389983', '390002', '390057', '390062', '390063', '390067', '394478']
nosy_count = 2.0
nosy_names = ['vstinner', 'petr.viktorin']
pr_nums = ['25131', '25133', '25134', '25135']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43688'
versions = ['Python 3.10']

[vstinner]

Currently, setup.py doesn't build xxlimited and xxlimited_35 extension modules with the limited C API if Python is built in debug mode. I only found two functions affected by Py_DEBUG macro in the limited C API: Py_INCREF() and Py_DECREF().

Example:

#if defined(Py_DEBUG) && !defined(Py_REF_DEBUG)
#define Py_REF_DEBUG
#endif

static inline void _Py_INCREF(PyObject *op)
{
#ifdef Py_REF_DEBUG
    _Py_RefTotal++;
#endif
    op->ob_refcnt++;
}
#define Py_INCREF(op) _Py_INCREF(_PyObject_CAST(op))

If Py_DEBUG is defined (Python built in debug mode), Py_INCREF() increments the private _Py_RefTotal variable.

The limited C API is supposed to provide a stable ABI, but Py_INCREF() leaks _Py_RefTotal implementation if Python is built in debug mode.

I propose to modify Py_INCREF() and Py_DECREF() of the limited C API to always declare them as opaque function calls. The regular (non limited) C API will continue to use the same static inline functions.

See also #25115 and bpo-41111 "[C API] Convert a few stdlib extensions to the limited C API (PEP-384)".

Note: Py_XINCREF() and Py_XDECREF() should be fine.

[vstinner]

One nice advantage of this change is that it becomes possible to check for reference leaks in C extensions written with the limited C API!

$ ./python -m test test_xxlimited -R 3:3
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_xxlimited
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

1 test OK.

Total duration: 367 ms
Tests result: SUCCESS

[vstinner]

New changeset baf10da by Victor Stinner in branch 'master':
bpo-43688: Run make regen-limited-abi (GH-25134)
baf10da

[encukou]

If you do this, please check the performance impact. Py_INCREF/Py_DECREF are very common.

[vstinner]

New changeset 2ac0515 by Victor Stinner in branch 'master':
bpo-43688: Fix Py_LIMITED_API version of xxlimited (GH-25135)
2ac0515

[vstinner]

New changeset baf10da by Victor Stinner in branch 'master':
bpo-43688: Run make regen-limited-abi (GH-25134)

With this change, "Tests / Check if generated files are up to date" job started fails (on PR 25135): "Some symbols from the limited API are missing: PyType_HasFeature".

I fixed this issue in bpo-43690 which removes PyType_HasFeature from Doc/data/stable_abi.dat.

[vstinner]

I wrote a microbenchmark on Py_INCREF()+Py_DECREF():

$ python3 -m pyperf compare_to ref.json limited.json 
Mean +- std dev: [ref] 3.45 ns +- 0.17 ns -> [limited] 6.03 ns +- 0.21 ns: 1.75x slower

If a function is only made of Py_INCREF() and Py_DECREF(), it can be up to 1.8x slower in the worst case. But in practice, I don't think that functions are only made of Py_INCREF() and Py_DECREF(). They do a few other things.

I'm not sure how to run a "macro benchmark" on my PR 25131, since I don't know any C extension doing anything useful. There is xxlimited, but it does almost nothing.

What would be a fair benchmark for this change?

--

To run my microbenchmark:

git apply bench.patch
./configure --with-lto --enable-optimizations
make
./python -m venv env
./env/bin/python -m pip install pyperf
./env/bin/python ../bench_limited.py -o ../limited.json -v
./env/bin/python ../bench_testcapi.py -o ../ref.json -v