Naming convention for new C API functions

[vstinner]

Are you ok with the naming convention of adding Ref suffix when adding a variant of a function which returns a strong reference instead of a borrowed reference? Do we need to go further in terms of standard?

See also issue #14: "PyLong and PyUnicode don't match the Python type names".

Point raised by @markshannon in my PR adding PyDict_GetItemRef().

(*) I would like to fix the C API to avoid functions returning borrewed references. For that, I add new variants of existing functions which return strong references, by adding a Ref suffix to their name:

• PyModule_AddObject() => PyModule_AddObjectRef()
• PyImport_AddModule() => PyImport_AddModuleRef()
• PyWeakref_GetObject() => PyWeakref_GetRef()
• PyDict_GetItemWithError() => PyDict_GetItemRef() -- it also replaces PyDict_GetItem()

In the past, other naming convention were used (other suffixes were added).

(*) To add support for Unicode filenames in the import machinery, I add variants of functions taking char* by adding Object suffix: function taking PyObject* argument. Examples:

• PyRun_SimpleFileExFlags() => PyRun_SimpleFileObject()
• PyRun_InteractiveOneFlags() => PyRun_InteractiveOneFlagsObject()
• Py_CompileStringExFlags() => Py_CompileStringObject()
• PyErr_WarnExplicit() = PyErr_WarnExplicitObject()
• PyErr_SyntaxLocationEx() => PyErr_SyntaxLocationObject()

(*) Previously, another trend was simply to add Ex suffix. Windows API has a similar pattern: WaitForMultipleObjects() => WaitForMultipleObjectsEx(). Examples in Python:

• PyErr_Print() => PyErr_PrintEx()
• Py_Finalize() => Py_FinalizeEx()
• PyErr_SyntaxLocationObject() => PyErr_SyntaxLocationObjectEx() -- this one now combines two suffix, Object and Ex :-)

(*) A fourth trend (!) is to use WithSomething suffix. Examples:

• PyDict_GetItem() => PyDict_GetItemWithError()
• PyImport_ExecCodeModuleEx() => PyImport_ExecCodeModuleWithPathnames()
• PyCFunction => PyCFunctionWithKeywords
• PyType_FromSpec() => PyType_FromSpecWithBases()

(*) I also saw another suffix, I don't know if it can be called a convention, a variant of the previous name: add Flags.

• PyRun_AnyFileEx() => PyRun_AnyFileExFlags() -- combo! Ex and Flags suffixes :-)
• PyRun_String() => PyRun_StringFlags()
• _PyStructSequence_InitBuiltin() => _PyStructSequence_InitBuiltinWithFlags -- variant of the variant... With + Flags

So I saw the following conventions to add suffixes (oldest to the most recent):

• Add Flags suffix
• Add Ex suffix
• Add WithSomething suffix
• Add Object suffix
• Add Ref suffix

The funny part is that some added functions get multiple suffixes like PyRun_AnyFileExFlags() (Ex + Flags).

[markshannon]

Given this for new API functions why not have a new prefix to clearly distinguish new API functions from old?

E.g.

PyAPI_DictGetItem()
PyCAPI_DictGetItem()
Python_DictGetItem()
PyStable_DictGetItem() /* To distinguish from the unstable API */

[vstinner]

A side question of this issue is: How to shrink the size of the C API? See the issue #53 for that.

[vstinner]

Given this for new API functions why not have a new prefix to clearly distinguish new API functions from old?

Over time, some functions accumulated many variants (versions):

• Version 1: PyRun_AnyFile()
• Version 2: PyRun_AnyFileFlags() and PyRun_AnyFileEx() -- yes, two for the price of one!
• Version 3: PyRun_AnyFileExFlags() (sic)
• Version 4 (private): _PyRun_AnyFileObject()

Total: 5 flavors of PyRun_AnyFile(). All of them are still available in Python 3.13!

Adding a prefix or a suffix doesn't make a big difference: at the end, we will always need to add a new version in future to fix yet another issue. Someone will come up with a new use case which requires to change an existing API which means adding yet another flavor.

So do we prefer accumulating suffixes, prefixes, or both? So far, the trend was to always keep the name short Py prefix (PyXXX or Py_XXX names) and only add suffixes.

[markshannon]

PyRun_XXX is one case where we do want a struct in the API.
Structs can be used as input to general, extensible functions, allowing named arguments in C.
E.g.

struct PythonRunConfig {
    int version; /* Necessary, so the API function doesn't read off the end */
    FILE *fp;
    const char *filename; 
    int closeit,
    PyCompilerFlags *flags; 
    /* Extra fields go here */
};

PyAPI_FUNC(int) Python_Run(const PythonRunConfig *config);

Used as follows:

PythonRunConfig config =
(PythonRunConfig) {
    .fp = the_file,
    .closeit = 1
};

int err = Python_Run(&config);

[vstinner]

IMO changing the suffix should be a one time event and the whole legacy API would still be supported forever anyway: #54 Changing the suffix is an opportunity to reuse existing names for the "sane" behavior.

Even if the suffix is changed once, we will still have to add new variants tomorrow. Do you have to agree on a suffix?

[vstinner]

@markshannon @erlend-aasland: are you ok with the Ref suffix convention for new functions returning a strong reference rather than a borrowed reference?

cc @serhiy-storchaka

[vstinner]

PyRun_XXX is one case where we do want a struct in the API.

The struct API sounds appealing. But that would be a 6th version of the same function and so a new name should be found: completely different or add yet another suffix.

[erlend-aasland]

Data point: SQLite simply use a version suffix, increasing the version number for each API revision. Example:

• sqlite3_prepare
• sqlite3_prepare_v2
• sqlite3_prepare_v3

[erlend-aasland]

@markshannon @erlend-aasland: are you ok with the Ref suffix convention for new functions returning a strong reference rather than a borrowed reference?

cc @serhiy-storchaka

What if we have to amend one of the *Ref() functions say in five years time from now. What do we call the new, amended API? *RefEx()? :)

I kind of like the version convention of SQLite; you never know how many variants of an API you'll need, and naming them similar only creates more confusion (in my opinion). For example PySomething_Fetch, PySomething_Get vs. PySomething_GetRef. It is not obvious which one is the newest? If they are named PySomething_Get, PySomething_Get_v2, and PySomething_Get_v3 it would be obvious which one is newest.¨

If _GetItem is the perfect name for an API that need amending, why not simply attach _v2 to it for the amended variant?

[erlend-aasland]

I think this issue is drifting to a discussion about possible solutions; suggesting to create an issue in the devguide, and continue discussing solutions (that is: guidelines) over there.

[erlend-aasland]

I created python/devguide#1126 for discussion solutions.

[vstinner]

What if we have to amend one of the *Ref() functions say in five years time from now. What do we call the new, amended API? *RefEx()? :)

My experience says that yes, for sure, we will have to add a variant of at least one Ref function.

In that case, I suggest coming with a new suffix and replace Ref with that suffix.