Embedded Python 3.11 on windows warns about <exec_prefix>

[davidhewitt]

Bug report

On Windows, with Python 3.11, an embedded Python interpreter prints a new warning at startup Could not find platform dependent libraries <exec_prefix>.

3.10 and older versions do not exhibit this.

Sample program (I'm using Rust / PyO3 because it's what I know, however I'm fairly sure that it's not relevant to the repro. If you've got a recommended way to configure a windows C toolchain for building an embedded Python I'd be happy to demonstrate the same issue in C):

fn main() {
    unsafe { 
        pyo3::ffi::Py_Initialize();
        dbg!(pyo3::ffi::Py_IsInitialized());
    }
}

Running outputs:

Could not find platform dependent libraries <exec_prefix>
[src\main.rs:4] pyo3::ffi::Py_IsInitialized() = 1

Execution then proceeds without issue. (We observe this in PyO3's test suite, which passes all 3.11 tests despite emitting this warning.)

I have a couple of thoughts:

• Documentation for exec_prefix suggests it is irrelevant on Windows. So is this warning actually useful?
• I suspect this is because of the reimplementation of getpath into Python? (Rewrite getpath.c in Python #89745)

Your environment

• CPython versions tested on: 3.11.0
• Operating system and architecture: Windows x64

• PR: gh-98790: Disable pathconfig warnings except when verbose mode is enabled #98933

• PR: gh-98790: When DLLs directory is missing on Windows, assume executable_dir contains PYD files instead #98936

• PR: [3.11] gh-98790: When DLLs directory is missing on Windows, assume executable_dir contains PYD files instead (GH-98936) #99019

[kevinf100]

I can confirm this isn't a PyO issue. This happening in Visual Studio running C++ code.
Calling Py_Initialize(); gets the exact same error. No matter if x86, x64, debug or release. Rebuild and changing the Include and library Directories back to 3.10 and the error goes away.

[zooba]

All you probably need to do is create an empty DLLs folder, or use a ._pth file. Either will suppress the warning.

Incidentally, for an embedded Python, you really want to set paths yourself. Your app is very open to attack if it's allowed to pick up libraries from anywhere on disk, and the only ways to prevent this (Py_SetPath or a ._pth file) will prevent this warning. (The warning, of course, does not mention this, because it was unintentional 😉 )

Now, for the case where we are expecting a "standard" layout and don't find it, we actually only run because Windows has always implicitly included the executable's directory in sys.path. We decided a while back that this was incorrect, but unfixable. So perhaps we ought to encode the extra path as a search for an expected .pyd file? (Which will make it even more important to override the paths when embedding, but would allow for manually rearranging into a flat layout.)

[zooba]

Alternatively, you can set PyConfig.pathconfig_warnings to 0 in the initialisation config.

Changing this default might be the best fix here anyway. Right now, it's based on the Py_FrozenFlag flag, which is documented as internal, but is only ever used by tests. And all it does is disable logging, rather than enabling it, which means if we default to disabled it just becomes a pointless flag. Instead, we could use Py_VerboseFlag (the -v option) to enable logging, and use that as a recommended debugging step when initialisation fails.

Not sure who has the most thoughts about this vs. changing platlib search (which may be unnecessary if we can rely on it always being relative to the stdlib directory)? Maybe @vstinner or @FFY00?

[kevinf100]

All you probably need to do is create an empty DLLs folder, or use a ._pth file. Either will suppress the warning.

Incidentally, for an embedded Python, you really want to set paths yourself. Your app is very open to attack if it's allowed to pick up libraries from anywhere on disk, and the only ways to prevent this (Py_SetPath or a ._pth file) will prevent this warning. (The warning, of course, does not mention this, because it was unintentional 😉 )

Now, for the case where we are expecting a "standard" layout and don't find it, we actually only run because Windows has always implicitly included the executable's directory in sys.path. We decided a while back that this was incorrect, but unfixable. So perhaps we ought to encode the extra path as a search for an expected .pyd file? (Which will make it even more important to override the paths when embedding, but would allow for manually rearranging into a flat layout.)

Py_SetPath no longer works in Python 3.11.
What is the right way to do this in Python 3.11? The following way didn't work for me
config.module_search_paths_set = 1; PyWideStringList_Append(&config.module_search_paths, L"C:\\Users\\Kevinf100\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\;C:\\Users\\Kevinf100\\source\\repos\\PythonTest\\x64\\Release");
Also Python no longer looks in the Working Directory of the .exe anymore it seems. Having the .py file next to the .exe found the .py file in 3.10, but now it no longer does in 3.11.

[zooba]

Py_SetPath should still work. If it's not, we need more test cases that cover your scenario (see _testembed.c for our current tests). And it certainly includes the executable path in sys.path by default, though there are some non-obvious ways to affect that once you're embedding.

I've posted two PRs - one which I think is more correct (suppressing debug messages by default), and one that just adds a Windows-specific hack to never warn about not having found DLLs. Neither is good, but I'm trying to stick to things appropriate for 3.11 at this stage.

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 😉 ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

[kevinf100]

Py_SetPath should still work. If it's not, we need more test cases that cover your scenario (see _testembed.c for our current tests). And it certainly includes the executable path in sys.path by default, though there are some non-obvious ways to affect that once you're embedding.

I've posted two PRs - one which I think is more correct (suppressing debug messages by default), and one that just adds a Windows-specific hack to never warn about not having found DLLs. Neither is good, but I'm trying to stick to things appropriate for 3.11 at this stage.

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 😉 ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

Looks like your right. Py_SetPath still does work. VS 2022 just doesn't compile without #pragma warning(suppress : 4996) in 3.11 it seems. Thanks. Didn't think of that without.

[FFY00]

I still think getpath.py could do with a revamp, but I don't have enough context to do it for all platforms (might be a project for @FFY00 wink ). The aim is to have more consistency, rather than less, which is why the OS-specific hacks are all marked and are less preferred.

Touching that is very tricky, and ought to break things, so I think we probably should only do it with a PEP, but yeah, I think making it more consistent across platforms would be really nice. For now, I think we should try to mimic the old initialization as best-effort, so I am in favor of the PR changing the behavior back to close of what it was.

[zooba]

This is resolved in 3.12 and backporting to 3.11, by changing the rule slightly on how DLLs is found on Windows. The visible changes will be:

• DLLs is only found adjacent to the executable (consistent with 3.10)
• if it's not found, the executable dir is added instead (in 3.10 the missing DLLs dir would still be added, and exe dir would be later)

[gpshead]

FYI this caused a regression in 3.11.1 that prevents some users from using it at all. #100171