[C API] Deprecate Global Configuration Variables like Py_IgnoreEnvironment

[vstinner]

PEP 587 "Python Initialization Configuration" got implemented in Python 3.8: https://peps.python.org/pep-0587/ It adds a new PyConfig API which is more complete and reliable than the legacy API to initialize Python: Py_Initialize() with a scattered collection of global variables and some "Set" functions like Py_SetPath().

In Python 3.11, I deprecated functions like Py_SetPath() and Py_SetStandardStreamEncoding(): https://docs.python.org/dev/c-api/init.html#c.Py_SetPath

I now propose to deprecate global configuration variables like Py_IgnoreEnvironment: PyConfig members should be used instead.

PR: #93943 merged as commit 0ff626f

[malemburg]

@vstinner: I don't know where this deprecation process has been discussed, but I didn't see it.

While it does make sense, given that we have the PyConfig structure now, the PRs you put in place literally skip the deprecation process: e.g. code setting one of the Py_XYZFlag variables will no longer work after the PyConfig struct has been set up, since the PRs redirect the lookup to PyConfig and completely ignore the global flags.

A better way would be to query both values and use the global var in case it is set. The gobal vars could be set to -1 during init to make this possible.

The global vars could then then be removed in a later release.

[vstinner]

skip the deprecation process

It seems like there is misunderstanding. This issue is only about marking global configuration variables as deprecated, it should not change the behavior of Python. This issue is not about scheduling the removal of these variables. That will be discussed later.

I don't know where this deprecation process has been discussed

It's discussed here.

When I implemented PEP 587 PyConfig, I spent a significant time on writing unit tests (Lib/test/test_embed.py and Programs/_testembed.c) to test that using the old way (like global config vars) still works. You might want to look at Python/preconfig.c and Python/initconfig.c to see the details.

A better way would be to query both values and use the global var in case it is set.

That's basically how it works. PyConfig_InitPythonConfig() is backward compatible with Python 3.7, whereas PyConfig_InitIsolatedConfig() ignores global config vars.

[vstinner]

test_init_global_config is a good example of unit test for the feature.

[malemburg]

I'll have a look, thanks.

Note that these flags can be adjusted after the interpreter has been initialized, e.g. to turn on optimization or increase verbosity, so I doubt that the logic as-is will work by just having PyConfig_InitPython*Config() pick up the current global flag values.

[malemburg]

Here's the analysis from reading the code:

Python/initconfig.c:

- config_get_global_vars() copies the global vars into the PyConfig struct, but only if config->_config_init != _PyConfig_INIT_COMPAT
  - this appears to be the only connection between the global vars and PyConfig
- it is called by _PyConfig_Read(), which is called by PyConfig_Read()

Programs/_bootstrap_python.c:

- only needed for building deep frozen modules
- calls PyConfig_Read() as part of main() during interpreter startup
  - this appears to duplicate the config read done later by PyConfig_Read()
- calls Py_InitializeFromConfig() as part of main() during interpreter startup

Modules/main.c:

- calls Py_InitializeFromConfig() as part of pymain_init()
- pymain_init() is (eventually) called by main() in Programs/python.c

Python/frozenmain.c:

- only needed for building frozen apps
- calls Py_InitializeFromConfig() as part of Py_FrozenMain() during interpreter startup of frozen apps

Python/pylifecycle.c:

- calls _PyConfig_Read() as part of _PyInterpreterState_SetConfig()
  - _PyInterpreterState_SetConfig() is only used by tests
- calls _PyConfig_Read() as part of pyinit_core()
- pyinit_core() is part of Py_InitializeFromConfig()

Python/pystate.c:

- _Py_GetConfig() is the helper used to access the PyConfig struct
  - this does not recheck the global vars

I may have missed some logic, but from the above, it seems that PyConfig is indeed only initialized from the global vars during interpreter startup.

Subsequent changes to the global flags are not taken into account by the various calls to _Py_GetConfig(), so any changes after interpreter init are ignored. This effectively causes the flags to be disabled. The only part which probably still works is when embedding the interpreter and setting up these global vars before initializing the interpreter.

To me, this looks like the deprecation has been skipped :-) and since this is not intended, I would suggest that the global flags are not replaced everywhere with calls to _Py_GetConfig(), but instead queried to check whether they have been altered first and only then have _Py_GetConfig() override them in case they have not been set.

This can be had by initializing the global vars to -1 (they are currently set to 0). If they are -1, _Py_GetConfig() is used. Otherwise the global var value is used (and perhaps a deprecation warning raised).

[malemburg]

Related to this, the other way around also doesn't appear to work, i.e. setting PyConfig.parser_debug doesn't result in Py_DebugFlag to be set.

In this particular case, the PyConfig member also appears to be misnamed. The debug flag happens to only be used in the parser (at the moment), but its global flag name certainly has a more general meaning than "parser_debug" and can be used for more than just the Python parser, e.g. to enable debugging code in Python extensions.