regression with latest Python 3.11.1 and _socket not found

[endmarsfr]

Hello,

I had an error with latest Python 3.11.1 on windows 10 and Windows 11.
module _socket not found

it concerns socket.py file line 52

• Everything worked well with previous version 3.11.0
• So i moved back to 3.11.0 version
• it worked again with 3.11.0

I hope, that could be solved easily

Regards

[ramvikrams]

Could you please share the code where you got this error

[pyscripter]

Same issue here. I am using python embedded into another application (the PyScripter IDE). It worked fine with 3.11.0 and all previous versions of python.
It is not just _socket. Python is not able to load any of the dlls in the DLLs directory.

I am using Python on Windows. I have tried adding the DLLs subdirectory to the system path or using os.add_dll_directory with no luck.

partial traceback

  File "c:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\__init__.py", line 44, in <module>
  File "c:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\__init__.py", line 1, in <module>
  File "c:\Program Files\PyScripter\Lib\rpyc.zip\rpyc\core\stream.py", line 7, in <module>
  File "C:\Python\Python311\Lib\socket.py", line 51, in <module>
    import _socket
ModuleNotFoundError: No module named '_socket'

It works fine from python.exe.

importlib.import_module also fails:

>>> importlib.import_module("_socket")
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked
ModuleNotFoundError: No module named '_socket'

[pyscripter]

Ok I found the source of this issue and a workaround:
os.path.join(sys.prefix, 'DLLs') is missing from the python path.

The following works.

    dllpath = os.path.join(sys.base_prefix, 'DLLs')
    sys.path.insert(2, dllpath)
    import _socket

Now the question is why os.path.join(sys.base_prefix, 'DLLs') is missing...

[pyscripter]

3d889dc is probably the culprit.

Assumes that a missing ``DLLs`` directory means that standard extension
modules are in the executable's directory.

[endmarsfr]

Nice job :)

[zooba]

I expect this is the same issue as fixed in #100947, since it looks like PyScripter relies on having a separate install of Python. It's not obvious that it's running the executable separately from the original install, but given the timing for the issue appearing I assume it must.

[pyscripter]

@zooba Indeed this is the case. Python is embedded in an executable that it is located in a different directory.

How can I test whether this resolves the issue? Where is getpath.py located and how is it used by python?

[zooba]

getpath.py is part of the source code. It gets compiled when we build and merged into the DLL so that we can run it before the compiler is ready to run - no easy way to change it without recompiling Python, I'm afraid.

If you set PYTHONHOME environment variable to Python's actual install location, it should work.

I think a more accurate workaround is to modify your registry - find the 3.11/PythonPath key and add two new keys into it. In the (Default) value for each, put the path to the Lib folder in one and the path to DLLs in the other. (One issue is that we are only looking at subkeys of this right now, and not the main one - that'll be fixed for 3.11.2 - so putting values in individual subkeys should help them be found. It will mess with your normal install a bit though, probably not badly, but it's an example of why we prefer to have the library in the same location as the executable rather than relying on registry keys.)

[pyscripter]

If you set PYTHONHOME environment variable to Python's actual install location, it should work.

I do that for unregistered versions. For registered versions it used to work fine without it before 3.11.1. Will it work as before in 3.11.2?

[zooba]

It should be fixed for 3.11.2, but I would still recommend setting the variable if you know what it should be set to. The PythonPath registry values are mostly there for back-compat, and newer installers may not have them (e.g. Windows Store installs do not, Nuget installs do not, etc.)

[pyscripter]

I can confirm that calling Py_SetPythonHome before initializing a registered python version works OK with 3.11.1.

[zooba]

Should be fixed in #100947