Install a static installation description file as part of the Python installation

[FFY00]

Feature or enhancement

Ship a static file that describes various aspects of the installation as part of the Python installation.

UPDATE: Proposed as PEP 739, which has been accepted.

Pitch

Shipping such a file would make it much easier to introspect a Python installation without having to run the interpreter. There are many use-cases for this, but some key would be for eg. Python launchers, and cross-compilation tooling.

Information we could provide:

• Python language version
• Python implementation (inc. version)
• Interpreter path
• Standard library path
• Interpreter architecture
• Python ABI
  • Tag (PEP 3149 on Posix)
  • Build flags (debug, trace refs)
• Static libpython (available?)
  • Name
  • Location
• Shared libpython (available?)
  • Name (standard library)
  • Major-only pin library name (eg. libpython3.so)
  • Location
• C API
  • Hex version
  • Include paths
  • Required C flags
• Supported native module file extensions
  • Interpreter-specific
  • Stable ABI

(incomplete table, just an initial proposal)

Note: This issue specifically targets a descriptor file for the Python installation, not a Python environment, so paths are out of scope.

Previous discussion

https://discuss.python.org/t/what-information-is-useful-to-know-statically-about-an-interpreter/25563.

(cc @brettcannon)

Linked PRs

• GH-107956: install a static build description file #108483
• GH-107956: install build-details.json (PEP 739) #130069

[brettcannon]

To be clear, this is for a CPython installation, not a general Python installation based on what's included (and assuming it's all required as nothing is specified as optional).

[FFY00]

I'd like to make the a general Python installation thing, by putting the CPython specific details in their own section.

[brandonardenwalli]

Also to be clarify:

Note: This issue specifically targets a descriptor file for the Python installation, not a Python environment, so paths are out of scope.

Is this to mean that the description file would be the same across all virtual environments made from a specific installation? For example, let us say I install python version 3.11 and then make 3 virtual environments: one, two, and three. All three of this environments to have the same description file, no?

[FFY00]

Yes, as the description file would be located in the Python installation paths themselves, instead of virtual environment.

Btw, standard Python virtual environment already have such a file, pyvenv.cfg, though it has it's own issues.

[brettcannon]

My thoughts on this general topic are written down in brettcannon/python-launcher#168 . While that proposal does not make sense in view of a static file tied to an interpreter, below is probably what would make sense (if you created this file for virtual environments as well):

    {
        // An array specifying what is required to execute the interpreter.
        // The expectation is to append args to the end of the array before
        // execution.
        // E.g. for conda environments:
        // ```
        // ["/path/to/conda", "run", "--path",
        //  "/home/brett/.conda/envs/conda-env", "--no-capture-output"]
        // ```.
        "run": [
            "/home/brett/my-venvs/my-venv/bin/python3.10"
        ],
        "python_version": {
            // `sys.version_info`
            "major": 3, // Optional
            "minor": 10, // Optional
            "micro": 1, // Optional
            "releaselevel": "final", // Optional
            "serial": 0 // Optional
        },
        "implementation": { // Optional
            // `sys.implementation`
            "name": "cpython",
            "version": { // Has the same structure as `python_version` above.
                "major": 3, // Optional
                "minor": 10, // Optional
                "micro": 1, // Optional
                "releaselevel": "final", // Optional
                "serial": 0 // Optional
            }
        },
        "executable": {
            "bits": 64, // Optional; `math.log2(sys.maxsize+1) + 1`
            "architecture": "x86-64"  // Optional; platform.machine()
        },
        "environment": { // Optional
            // What type of environment, e.g. "virtual", "conda", etc.
            "type": "virtual",
            "name": "my-venv"
        }
}

[FFY00]

@brettcannon thanks! Your use-case is a bit different than mine (improving cross-compilation support), so it's definitely .

I have a working prototype locally, which I am planing to propose soon, and there are a couple things I'd like to finalize before that, if you have any thoughts:

• Format
  • In my prototype I opted for TOML as that is what is being most widely used lately in modern Python-related tooling. The other main alternative to consider would be JSON. Personally, I have a biased preference towards TOML, but JSON would also be a good choice.
• Location
  • I think what would make the most sense would probably be placing the static description file in the stdlib directory, like the EXTERNALY-MANAGED file, but I think there would also be a lot of value in having standardized-ish location (or, more realistically, set of locations) to enable easier discovery. I am very dubious, though, on how easier that'd actually make discovery, but nothing actually prevents us from doing both (placing the file next to stdlib and in this, easier to discover, location) so unless there is any argument against placing it in the stdlib path, I think that's probably the way to go.

This initial implementation proposal will only include the main key details (I think probably the version, executable path, and stdlib path), and the rest of the details can then be proposed later in smaller PRs/proposals. My key goal here is to get the static description file itself sorted out.

if you created this file for virtual environments as well

That was specifically left out of scope ¹_ because it's tricky to do properly, the main thing that jumps out to me being the Python installation updating and leaving incorrect virtual environment static description files in place.

Some thoughts on possible approaches to tackle this issue:

• Have multiple config files that complement each-other (eg. the virtual environment static description file would reference the installation static description file)
• Have centralized virtual environment management (eg. when a virtual environment is created, it would be registered in a central database of sorts, and then a command to ensure virtual environments are up-to-date could be provided)

But ultimately, this is a significantly harder problem, and considering this is not incompatible with the current proposal and that the current proposal would already be a huge improvement over the current status quo, I think it makes sense to pursue it with the current scope.

Footnotes

1. The file proposed here is tied to the installation, not the environment or interpreter path, excluding virtual environments. ↩

[brettcannon]

• Format

I think the question is whether you expect people to ever read or write this file? If the answer is in general "no", then I think JSON is the easier format for people to ingest tooling-wise. But if you expect a human being to ever interact with the file I can see TOML making sense.

• Location

I think the way EXTERNALLY-MANAGED is stored makes sense from a CPython perspective. I think the question is if you expect other implementations or builds of Python to use this file? For instance, what if the binary is fully self-contained and thus there is no stdlib directory to look at since it doesn't exist?

the main thing that jumps out to me being the Python installation updating and leaving incorrect virtual environment static description files in place.

Fair enough. I always have to remember Unix is just symlinks and so it can change underneath itself while Windows is a hard copy and so you can't mess that detail up. But that might actually be an argument for the file when copying the python binary into the virtual environment.

[zooba]

This seems to me like it should just be a sysconfig.json file in sys.prefix, which ought to adequately communicate to anyone who's looking for it whether it's their thing or not (and it's specifically not going to be a PEP 514 substitute).

Hopefully between a couple of substitutes (i.e. allow {sys.<attr>} substitutions to be embedded in the otherwise static file) and some better alignment between platforms, we can more-or-less replace sysconfig.get_config_vars() with json.load, and also anyone outside of our runtime can also get everything that way too.

I don't see any reason to separate it from sysconfig though. The semantics already exist and people understand them, so any proposal will get through quicker if it's clearly based on it.

[FFY00]

I think the question is whether you expect people to ever read or write this file? If the answer is in general "no", then I think JSON is the easier format for people to ingest tooling-wise. But if you expect a human being to ever interact with the file I can see TOML making sense.

Yes, I expect people to read, and in some situations, write this file, but not too frequently. Some use-cases:

• Manually introspecting an installation without running the interpreter, which is the most relevant when working with cross-builds, but certainly not limited to it
• This plays a bit with my outer goal of trying to make it possible to build most extensions, etc. without having to run the interpreter, where in certain scenarios, I think users might need to write the file themselves (eg. when it's not available — I expand into this below)

I think the way EXTERNALLY-MANAGED is stored makes sense from a CPython perspective. I think the question is if you expect other implementations or builds of Python to use this file? For instance, what if the binary is fully self-contained and thus there is no stdlib directory to look at since it doesn't exist?

Yes, I expect other implementations to use this file, though not necessarily ship it in the same way.

My plan was for the format to be shared between everyone, but if and how the static description file is provided, to be up to each implementation.

Of course, I'd like the way the static description file is provided to be consistent, but as you point out, this is something that can be highly dependent on the implementation, build format, target platform, etc., so I don't think we can define a single way to do it. We could define how this should work in different scenarios, but I don't think we (CPython) should be the responsible for telling other implementations how they should implement this, especially in scenarios where we might not be experts.

For consistency, I would expect implementations that use an installation scheme similar to CPython to place the file in the same/equivalent place, but they wouldn't be required to do so. I think trying to get this to happen is something we should do as a community, by engaging in discussions with the other parties.

Finally, as I mentioned above, deciding if the file should be provided at all, is something I also think should be up to each implementation, and is something that could depend on the target, build format, etc.

This seems to me like it should just be a sysconfig.json file in sys.prefix, which ought to adequately communicate to anyone who's looking for it whether it's their thing or not (and it's specifically not going to be a PEP 514 substitute).

Regarding the name, I think using "sysconfig" in the name would imply a tie to the sysconfig module, which isn't really true for the proposal as-is (its relationship with sysconfig is something you mention later in your reply, so I elaborate this in the response to that). The purpose of the file is also a bit different, as it is not related to the system, but rather the Python installation specifically.

Regarding the location, while I guess using sys.prefix would be fine for Windows (I'm assuming, I'm not a Windows expert and I haven't looked into that), but it wouldn't on POSIX environments, where the prefix is /usr, /usr/local, etc., which are paths where we shouldn't be installing data like this.
Additionally, it is currently possible to install multiple versions of Python to the same prefix, resulting in the static description files to conflict. To fix this, we would need to change the file name, which would need to include all the things that might change between Python installations that can be installed to the same prefix, and since we don't really have limitations regarding that, I think this introduces a lot of avoidable complexity.
Thus, IMO the static description file should be installed to an installation-specific directory. That said, do note that nothing prevents us from also installing the files (probably by symlinking them when possible) to another directory too, where it could be more discoverable (this is something I already went into in my reply above).

Hopefully between a couple of substitutes (i.e. allow {sys.<attr>} substitutions to be embedded in the otherwise static file) and some better alignment between platforms, we can more-or-less replace sysconfig.get_config_vars() with json.load, and also anyone outside of our runtime can also get everything that way too.

I don't quite agree here 😅. This goes into the bigger picture question of the direction we want to push sysconfig and friends. I wanted to try to move users away from using sysconfig.get_config_vars in the way they are right now.

IMO sysconfig.get_config_vars is very problematic. I wrote a bit about this in the 1st point of GH-103480, sysconfig.get_config_vars exports very specific low-level information that was never meant to be public API. The issue is that a lot of information necessary for building extensions, and other similar use-cases, is only available this way.

For this reason, instead of building further on top of sysconfig.get_config_vars, I think a better direction would be to provide a lot of the information currently only available there in a proper stable public API — this is the main goal of the new sysconfig API proposal.

That said, sysconfig.get_config_vars is still a valuable mechanism and I don't think we should get rid of it or try to replace it. IMO it should stay as an unstable-ish feature that provides access to internal details of the build, similarly to the dis module. A good first step for this would be to improve the documentation to more clearly communicate the stability of its data.
Right now, we need to be careful when making changes to the Makefile, because a lot of user code depends on the names and values of its variables, and while I think we should keep making an effort to avoid breaking things unnecessarily, I think we should move towards the goal of minimizing the impact of this kind of changes.

Regarding being able to simply replace sysconfig.get_config_vars with json.load in code, I am not necessarily against the idea of also making the sysconfig.get_config_vars data more easily available statically, but if we go with the approach I described above, I think that should be separate from this proposal.

@zooba does this make sense to you? Like I said, it depends a lot on our higher level goals regarding the direction want to move sysconfig and friends towards.