Clarify which "identifiers" in the C API are macros

[soegaard]

Documentation

The "Python/C API Reference Manual" is an excellent resource but could be even better
if C macros were marked as such.

For a programmer that writes directly in C it doesn't matter which identifiers are real C functions
and which are C macros. However, when Python is used through libpython only the
C functions are available. This is the case for languages embedding Python through ffilib.

As an example: Consider the C macro PyImport_ImportModuleEx and the C function PyImport_ImportModuleLevel. They are documented in a way that makes it impossible
to guess that one is a C macro.

PyObject *PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)

PyObject *PyImport_ImportModuleLevel(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level)

A look in "Python.c" (or friends import.h here) will reveal that PyImport_ImportModuleEx is a C macro.
But it would be a quality of life-improvement, if a simple "C Macro" were added below the
signature of C macros.

This "C macro" annotation could have the same style as the "Return value is a new reference" annotation.
(maybe with a different color).

+1

In my opinion, even better would be the requirement that the C API functions, macros, and so on -- that are described in the documentation -- also have a corresponding link to the file in which they are defined. Of course, there are other ways to find where something is defined, but it could be very helpful if it were a part of the documentation.

The line number shouldn't be factored in since it would very quickly become invalidated through PRs that don't even touch these API items. The link to the file should only be changed whenever one wishes to add/remove an API item from a file (perhaps to relocate the definition to a different file).

I could be missing other considerations though.

[soegaard]

The documentation is writting using "sphinx".
In the manual [1] I see that C functions are marked with "c:function".
However, the comment says:

This is also used to describe function-like preprocessor macros.

The annotation "c:macro" is described as:

Describes a “simple” C macro. Simple macros are macros which are used for code expansion, but which do not take arguments so cannot be described as functions.

Is step one to extend Sphinx with a new annotation "c:macrofunction" ?

[1] https://pvbookmarks.readthedocs.io/en/master/documentation/doc_generators/sphinx/rest_sphinx/domain/c_domain.html#c-function

[ericvsmith]

Some of these (maybe all?) are deliberately left underspecified so we can switch if they're macros or functions.

[soegaard]

In that case, I think, should state that fact explicitly.

Btw - I see it as a positive that there is an ongoing effort to convert macros into static, inline functions:
#89653

If we were to specify whether it is a macro or a function, is there something documented or about the status quo that would dictate that it must stay that way for x amount of time? I think we mainly guarantee behavior (via the docs) and very rarely the implementation, no?

If not, I think the net benefit of distinguishing them on the documentation is positive, no?

[ericvsmith]

There are no guarantees beyond the limited API and stable ABI. Maybe @vstinner can give his opinion.

[vstinner]

Documentation patches are welcomed ;-)

[soegaard]

@vstinner

Documentation patches are welcomed ;-)
Great to hear.

Before I make a patch, I need to know, what kind of change is wanted.

Take the C macro PyImport_ImportModuleEx as an example.
Currently the documentation source [1] looks like this:

.. c:function:: PyObject* PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)

   .. index:: builtin: __import__

   Import a module.  This is best described by referring to the built-in Python
   function :func:`__import__`.
   <omitted>

When I look at the documentation for Sphinx [2], I see:

    c:function:: function prototype
    Describes a C function. The signature should be given as in C, e.g.:
    <omitted>

This suggests that I should change c:function to something else.
I would like to change it to c:macro but @ericvsmith mentions that it
is unspecified whether PyImport_ImportModuleEx is a C function or a C macro.

The Right™ solution would be to add an c:function_or_macro annotation to Sphinx.

The faster way is to add an explanation in the text:

"Currently a C macro, but could change to a C function in the future."

.. c:function:: PyObject* PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)
    "Currently a C macro, but could change to a C function in the future."

   .. index:: builtin: __import__

   Import a module.  This is best described by referring to the built-in Python
   function :func:`__import__`.
   <omitted>

Is this phrasing acceptable - or is it better to avoid the word "currently".

[1] https://github.com/python/cpython/blob/main/Doc/c-api/import.rst

[2] https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html

[vstinner]

I suggest adding "Function added as a macro." If it becomes a regular tomorrow, this sentence can be removed.

cc @encukou

[encukou]

well if you ask me...
IMO the correct solution is to change the code: add an actual exported function in addition to the macro.

[vstinner]

IMO the correct solution is to change the code: add an actual exported function in addition to the macro.

Well, if you ask me, macros should be converted to functions: https://peps.python.org/pep-0670/ :-)

I converted a few macros to static inline functions, but not to regular functions yet. As said in PEP 670, there is a question about the performance cost of such change. Since there are very few users requesting to call functions currently implemented as macros or static inline functions, there is active work on converting these to regular functions. It's more done on a case by case basis. Well, in general, IMO the overhead is not significan or can be ignored.

@soegaard Will you be making a PR? FWIW, if you wanted to just add to the text, I would suggest ``Function-like macro'' or going with Victor's suggestion.

Following Petr's suggestion, it may be worth creating a separate issue to see how other maintainers feel about creating actual functions in addition to the existing function-like macros in the C API to facilitate embedding Python. I wouldn't try to address this in your PR.

[soegaard]

@oda-gitso

I'll make some PR, but first let me know if the example below looks okay.

I added the phrase "Function-like macro" to import.rst which
contains the documentation of PyImport_ImportModuleEx.
It is the only function-like macro in import.h.

I used your phrase "Function-like macro" since that's the term used in the C specification.

The rendered documentation looks like this:

Ideally I would have liked no line between "Return value: New reference." and "Function-like macro.",
but I think that requires modifying Sphinx to fix.

If this looks okay, I'll look at the other files and make a PR.

The diff is here:

main...soegaard:mark-function-like-macros

[soegaard]

Here is a version with "Function-like macro" in italics.
I think it looks a little better.