getpath miscalculates sys.path from second initialization with PYTHONHOME on Windows

[neonene]

When I ran the code below in a python build directory:

#include <Python.h>
int
main(int argc, char *argv[])
{
    PyConfig config;
    PyConfig_InitPythonConfig(&config);
    for (int i=0; i < 3; i++) {
        printf("\n%d\n", i);
        Py_InitializeFromConfig(&config);
        PyRun_SimpleString("import sys; list(map(print, sys.path))");
        Py_Finalize();
    }
    PyConfig_Clear(&config);
    return 0;
}

• Expected results without PYTHONHOME:

0
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages

1
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages

2
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages

• set PYTHONHOME=C:\cpython-main:

0
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\PCbuild\amd64
C:\cpython-main
C:\cpython-main\Lib\site-packages

1
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\DLLs  # <<<<<<<<<< unexpected
C:\cpython-main
C:\cpython-main\Lib\site-packages

2
C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\DLLs  # <<<<<<<<<< unexpected
C:\cpython-main
C:\cpython-main\Lib\site-packages

Currently, test_embed fails due to this. (#32313)

[neonene]

First initialization can also cause the issue running Py_SetPythonHome() on a build directory:

#include <Python.h>
int
main(int argc, char *argv[])
{
    Py_SetPythonHome(L"C:\\cpython-main");
    Py_Initialize();
    PyRun_SimpleString("import sys; list(map(print, sys.path))");
    Py_Finalize();
    return 0;
}

C:\cpython-main\PCbuild\amd64\python311.zip
C:\cpython-main\Lib
C:\cpython-main\DLLs  # <<<<<<<<<< unexpected
C:\cpython-main
C:\cpython-main\Lib\site-packages

EDIT: If Py_SetPythonHome() is expected to be static unlike PYTHONHOME, this is correct behavior.

[neonene]

On 3.10 and earlier with or without PYTHONHOME, an executable in a build directory has sys.path as below:

0
C:\cpython-3.10\PCbuild\amd64\python310.zip
C:\cpython-3.10\DLLs
C:\cpython-3.10\lib
C:\cpython-3.10\PCbuild\amd64
C:\cpython-3.10
C:\cpython-3.10\lib\site-packages

1
C:\cpython-3.10\PCbuild\amd64\python310.zip
C:\cpython-3.10\DLLs
C:\cpython-3.10\lib
C:\cpython-3.10\PCbuild\amd64
C:\cpython-3.10
C:\cpython-3.10\lib\site-packages

2
C:\cpython-3.10\PCbuild\amd64\python310.zip
C:\cpython-3.10\DLLs
C:\cpython-3.10\lib
C:\cpython-3.10\PCbuild\amd64
C:\cpython-3.10
C:\cpython-3.10\lib\site-packages

[zooba]

Okay, so the problem is that home is set on the second call, which means getpath skips checking for build directory landmarks.

IIRC, other tests were broken if we check against real_executable for the build landmarks and ignore PYTHONHOME (presumably because they used symlinks to a new directory for tests?), so this probably just needs new logic.

However, the fix in #92980 might also affect this, by not recalculating sys.path on subsequent calls. You might want to test again before changing too much.

[neonene]

IIRC, other tests were broken if we check against real_executable for the build landmarks and ignore PYTHONHOME (presumably because they used symlinks to a new directory for tests?), so this probably just needs new logic.

test_getpath passes the following test cases:

• test_symlink_buildtree_win32
• test_buildtree_pythonhome_win32

I can also start python in their file layouts, but test_embed were broken. Neither PYTHONHOME nor home worked around. New logic may be needed for this. Do I understand what you mean?

[zooba]

Had a quick look at your changes, but honestly am going to need a lot more time to figure out that the logic is sound. That also concerns me, because I really want to avoid putting any logic for this stuff back in C - moving to Python was to fix that ;-)

Another idea that might work: what if we just add a config field for "running in-tree build"? It can start as "unspecified" and then be updated if we detect landmarks. Then even if home is already set but we know we're in tree, we can go down the path that overrides platlibdir. What do you think?