gh-92651: Remove the Include/token.h header file

[vstinner]

Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

• PyToken_OneChar() => _PyToken_OneChar()
• PyToken_TwoChars() => _PyToken_TwoChars()
• PyToken_ThreeChars() => _PyToken_ThreeChars()

[vstinner]

@pablogsal @lysnikolaou @iritkatriel: Do you see any reason to expose token.h as part of the public C API? Or are you fine with removing this header file from the public C API?

Issue: #92651

[vstinner]

These functions and _PyParser_TokenNames symbols are no longer exported.

I tried to no longer export these symbols, but it broke test_peg_generator. I reverted this change.

[serhiy-storchaka]

Update also the default value in make_h().

Should PyAPI_FUNC and PyAPI_DATA be used in declarations?

[vstinner]

Update also the default value in make_h().

Oh right, I updated my PR.

Should PyAPI_FUNC and PyAPI_DATA be used in declarations?

As I wrote in my previous comment, first, I removed PyAPI_FUNC() and PyAPI_DATA() but it broke test_peg_generator. This test builds C extensions which are linked dynamically.

I added // Symbols exported for test_peg_generator comment to future readers.

[pablogsal]

Do you see any reason to expose token.h as part of the public C API?

No, and is a bad idea to have it as part of the public C API, so if we can move it to the internal heafers folder, that would be great :)

[vstinner]

This PR doesn't respect PEP 387 deprecation process because:

• token.h is not included by Python.h and so is part of "the Python C API"
• it was never possible to get tokens from any public Python C API function: this header file is basically useless

In Python 3.10, Parser/tokenizer.h defines these functions:

extern struct tok_state *PyTokenizer_FromString(const char *, int);
extern struct tok_state *PyTokenizer_FromUTF8(const char *, int);
extern struct tok_state *PyTokenizer_FromFile(FILE *, const char*,
                                              const char *, const char *);
extern void PyTokenizer_Free(struct tok_state *);
extern int PyTokenizer_Get(struct tok_state *, const char **, const char **);

Python is now built with -fvisibility=hidden: functions which are not declared with PyAPI_FUNC() are not exported. So these functions are not exported.

In Python 3.11, these functions were renamed to get a "_Py" prefix. For example, PyTokenizer_FromString() was renamed to _PyTokenizer_FromString().

These functions are not exported and are private since Python 3.11 (_Py prefix).

[vstinner]

Merged. Thanks for the reviews!