C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[vstinner]

BPO	37250
Nosy	@ncoghlan, @scoder, @encukou, @ambv, @jdemeyer, @zooba, @willingc
PRs	bpo-37250: define tp_print for backwards compatibility #14009 [3.8] bpo-37250: put back tp_print for backwards compatibility #14193

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-01.13:40:34.626>
created_at = <Date 2019-06-12.13:08:30.505>
labels = ['interpreter-core', 'build', '3.8']
title = 'C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed'
updated_at = <Date 2019-07-01.13:40:34.626>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-07-01.13:40:34.626>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-01.13:40:34.626>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2019-06-12.13:08:30.505>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 37250
keywords = ['patch']
message_count = 69.0
messages = ['345337', '345340', '345341', '345342', '345343', '345344', '345345', '345346', '345347', '345348', '345349', '345350', '345351', '345352', '345353', '345355', '345356', '345357', '345359', '345382', '345383', '345402', '345405', '345409', '345459', '345591', '345965', '345966', '345969', '345988', '345989', '345990', '345992', '345994', '345996', '345998', '346013', '346064', '346067', '346069', '346075', '346083', '346084', '346085', '346087', '346099', '346100', '346105', '346123', '346125', '346128', '346153', '346161', '346163', '346177', '346185', '346190', '346194', '346213', '346218', '346220', '346222', '346223', '346238', '346373', '346388', '346424', '346438', '346507']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'scoder', 'petr.viktorin', 'lukasz.langa', 'jdemeyer', 'steve.dower', 'willingc']
pr_nums = ['14009', '14193']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue37250'
versions = ['Python 3.8']

[vstinner]

Copy of Stefan Behnel's msg345305 in bpo-37221:
"""
Note that PyCode_New() is not the only change in 3.8 beta1 that breaks Cython generated code. The renaming of "tp_print" to "tp_vectorcall" is equally disruptive, because Cython has (or had) a work-around for CPython (mis-)behaviour that reset the field explicitly to NULL after calling PyType_Ready(), which could set it arbitrarily without good reason.

So, either revert that field renaming, too, or ignore Cython generated modules for the reasoning about the change in this ticket.

I'm fine with keeping things as they are now in beta-1, but we could obviously adapt to whatever beta-2 wants to change again.
"""

There are 2 problems:

• source compatibility
• ABI compatibility

The following change removed PyTypeObject.tp_print and replaced it with PyTypeObject.tp_vectorcall_offset:

commit aacc77f
Author: Jeroen Demeyer <J.Demeyer@UGent.be>
Date: Wed May 29 20:31:52 2019 +0200

bpo-36974: implement PEP-590 (GH-13185)


Co-authored-by: Jeroen Demeyer <J.Demeyer@UGent.be>
Co-authored-by: Mark Shannon <mark@hotpy.org>

== ABI compatibility ==

In term of ABI, it means that C extensions using static type ("static PyTypeObject mytype = { .... };") is broken by this change.

Honestly, I'm not sure if we ever provided any forward compatibility for static types. bpo-32388 removed "cross-version binary compatibility" on purpose in Python 3.8. It's an on-going topic, see also my notes about ABI and PyTypeObject:
https://pythoncapi.readthedocs.io/type_object.html

Maybe we can simply ignore this part of the problem.

== Source compatibility ==

Cython generates C code setting tp_print explicitly to NULL.

To avoid depending on Cython at installation, most (if not all) projects using Cython include C files generated by Cython in files they distribute (like tarballs). Because of that, the commit aacc77f requires all these projects to regenerate their C files using Cython.

In Fedora, we fixed many Python packages to always re-run Cython to regenerate all C files. But Fedora is just one way to distribute packages, it doesn't solve the problem of files distributed on PyPI, nor other Linux distribution (for example).

Jeroen Demeyer proposed PR 14009 to fix the source compatibility:

   #define tp_print tp_vectorcall

[vstinner]

"printfunc tp_print;" has been replaced with "Py_ssize_t tp_vectorcall_offset;": the type was basically a pointer and has been replaced with an integer.

With "#define tp_print tp_vectorcall", "type->tp_print = NULL;" becomes "type->tp_vectorcall = NULL;".

If -Werror is used, "type->tp_vectorcall = NULL;" would fail with a compilation error, since NULL is not exactly an integer, no? I'm not sure that it's an issue, I'm just thinking aloud.

[jdemeyer]

Actually, Cython generates code setting tp_print to 0 (not NULL).

[vstinner]

inherit_slots() uses:

        /* Inherit tp_vectorcall_offset only if tp_call is not overridden */
        if (!type->tp_call) {
            COPYSLOT(tp_vectorcall_offset);
        }

PyType_Ready() contains the following assertion:

    /* Consistency checks for PEP 590:
     * - Py_TPFLAGS_METHOD_DESCRIPTOR requires tp_descr_get
     * - _Py_TPFLAGS_HAVE_VECTORCALL requires tp_call and
     *   tp_vectorcall_offset > 0
     * To avoid mistakes, we require this before inheriting.
     */
    if (type->tp_flags & Py_TPFLAGS_METHOD_DESCRIPTOR) {
        _PyObject_ASSERT((PyObject *)type, type->tp_descr_get != NULL);
    }
    if (type->tp_flags & _Py_TPFLAGS_HAVE_VECTORCALL) {
        _PyObject_ASSERT((PyObject *)type, type->tp_vectorcall_offset > 0);
        _PyObject_ASSERT((PyObject *)type, type->tp_call != NULL);
    }

I understand that tp_vectorcall_offset=0 is fine and the expected value for a type which doesn't have the flag _Py_TPFLAGS_HAVE_VECTORCALL.

[vstinner]

IMHO the simplest and safest option for this issue is to revert tp_print removal and move tp_vectorcall_offset at the end of PyTypeObject.

Is PEP-590 fully public in Python 3.8? It seems like _Py_TPFLAGS_HAVE_VECTORCALL at least is private.

Maybe we can attempt again to remove tp_print from Python 3.9? Once Cython will handle tp_print removal? Or even keep it in Python 3.9?

Python 3 had unused tp_print for 10 years and nobody complained. There are a few PyTypeObject instances in a Python process. A single pointer isn't a giant waste of memory.

[jdemeyer]

If -Werror is used, "type->tp_vectorcall = NULL;" would fail with a compilation error, since NULL is not exactly an integer, no?

No because tp_vectorcall is a function pointer. You're confusing with tp_vectorcall_offset which is an integer.

[vstinner]

No because tp_vectorcall is a function pointer. You're confusing with tp_vectorcall_offset which is an integer.

Oh. I didn't notice that your PR makes tp_print an alias to tp_vectorcall rather than tp_vectorcall_offset. Ok.

[vstinner]

If we decide to not reintroduce tp_print, tp_print removal must be documented at:
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api

[jdemeyer]

I understand that tp_vectorcall_offset=0 is fine and the expected value for a type which doesn't have the flag _Py_TPFLAGS_HAVE_VECTORCALL.

Not necessarily. There are subtleties involved when subclassing: there are cases where tp_vectorcall_offset needs to be non-zero despite _Py_TPFLAGS_HAVE_VECTORCALL not being set. See also PR 13858.

[vstinner]

Cython issue: cython/cython#2976

[jdemeyer]

IMHO the simplest and safest option for this issue is to revert tp_print removal and move tp_vectorcall_offset at the end of PyTypeObject.

That's a drastic solution to a rather simple problem. PR 14009 would fix Cython just fine.

[vstinner]

me:

I understand that tp_vectorcall_offset=0 is fine and the expected value for a type which doesn't have the flag _Py_TPFLAGS_HAVE_VECTORCALL.

Jeroen Demeyer:

Not necessarily. There are subtleties involved when subclassing: there are cases where tp_vectorcall_offset needs to be non-zero despite _Py_TPFLAGS_HAVE_VECTORCALL not being set. See also PR 13858.

Cython generates C code which looks like:

if (PyType_Ready(type) < 0) { ... handle error ... }
type->tp_print = 0;

This code can cause subtle and annoying issue if PR 13858 is merged. So that's another argument in favor of reintroducing tp_print in Python 3.8.

--

Cython has already been fixed to no longer set tp_print to 0 on Python 3.8:

cython/cython@f10a0a3

But again, this problem is not about correctness, but more about practical backward compatibility issues (see my first message).