Python errors related to failures loading DLL's lack information

[never-eat-yellow-snow]

BPO	25655
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @The-Compiler, @pombredanne, @miss-islington
PRs	bpo-25655: Improve Win DLL loading failures doc #22372 [3.9] bpo-25655: Improve Win DLL loading failures doc (GH-22372) #22894 [3.8] bpo-25655: Improve Win DLL loading failures doc (GH-22372) #22895

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-11-18.08:37:03.188>
labels = ['3.10', 'type-feature', '3.8', '3.9', 'OS-windows']
title = "Python errors related to failures loading DLL's lack information"
updated_at = <Date 2020-10-22.16:41:05.236>
user = 'https://bugs.python.org/never-eat-yellow-snow'

bugs.python.org fields:

activity = <Date 2020-10-22.16:41:05.236>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2015-11-18.08:37:03.188>
creator = 'never-eat-yellow-snow'
dependencies = []
files = []
hgrepos = []
issue_num = 25655
keywords = ['patch']
message_count = 16.0
messages = ['254838', '254849', '377326', '377327', '377335', '377351', '377360', '377400', '377416', '377418', '379301', '379303', '379306', '379307', '379308', '379315']
nosy_count = 9.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'The Compiler', 'pombredanne', 'never-eat-yellow-snow', 'miss-islington']
pr_nums = ['22372', '22894', '22895']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25655'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[never-eat-yellow-snow]

Currently you get errors like this:

ImportError: DLL load failed: The specified procedure could not be found.
ImportError: DLL load failed: The specified module could not be found.

It would be nice to include more information, at least the name of the dll which could not be loaded. Maybe also the name of the (dependent) dll which could not be found.

Currently, I use ProcessMonitor to debug which dll could not be found, because the error message is lacking important information.

Note: I tagged the two versions I use, but probably all python versions are affected by this issue.

[zooba]

I don't know that we can necessarily provide correct information for those errors as it depends where they actually fail. If we're simply passing on an error from the loader, then there's very little we can do.

We may be able to make an educated guess based on our context, but there's a fairly high chance we'll guess wrong in some cases and make them more difficult to debug (unless you know that the error is only a guess, in which case you're back to procmon or Dependency Walker).

You're right that all versions are affected, however I think improving these diagnostics is only within scope for 3.5 and 3.6 at this stage.

[pombredanne]

From https://bugs.python.org/issue41836 closed as a dupe of this:

When the dependency of a DLL is missing (at least on Windows) the error " OSError: [WinError 126] The specified module could not be found" is raised when calling ctypes.CDLL(dll_path) even when this "dll_path" exists... because the error comes from another DLL.

These errors are really hard to diagnose because the path of the missing DLL is not returned in the exception message. Returning it would help fixing these kind of errors quickly.

Researching errors such as this one aboutcode-org/scancode-toolkit#2236 wastes quite a bit of time and would be made a non issue if we had the path in the error message.

and this reply from Eric Smith: https://bugs.python.org/msg377324

Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-09-22 14:13

My understanding is that Windows doesn't tell you which DLL is missing. I think the best we could do is append something to the error message saying "or one its dependencies".

[pombredanne]

Eric Smith, you wrote:

My understanding is that Windows doesn't tell you which DLL is missing. I think the best we could do is append something to the error message saying "or one its dependencies".

If we have such an error message, this means the main DLL exists: the original path passed to ctypes exists and is a valid DLL otherwise the message would be different.

So I think that this is always a direct or indirect dependency of that primary DLL that would be missing and we could be explicit in the error message.

We could also provide some hints in the error message on how to research the issue may be?

[eryksun]

" OSError: [WinError 126] The specified module could not be found" is
raised when calling ctypes.CDLL(dll_path) even when this "dll_path"
exists... because the error comes from another DLL.

That's the old error. bpo-36085 changed it to FileNotFoundError, with the message "Could not find module '%.500S'. Try using the full path with constructor syntax." bpo-39393 modified the message to "Could not find module '%.500S' (or one of its dependencies). Try using the full path with constructor syntax."

IMO, the most direct way to resolve the problem is by enabling "loader snaps" for python.exe via gflags and attaching a native debugger to the process. The loader outputs debug strings that show the computed DLL search path (from LdrpComputeLazyDllPath), each attempt to resolve the dependent DLL to a directory in the search path (via LdrpResolveDllName), and the final result from loader's work queue (from LdrpProcessWork), which includes the dependent DLL that caused loading to fail and the parent module (DLL or EXE) that depends on it.

[zooba]

IMO, the most direct way to resolve the problem is by enabling "loader snaps" for python.exe via gflags and attaching a native debugger to the process ...

This is indeed the best way to go about solving it, so you can see why we don't put it in an error message or take responsibility for documenting the process. It's not for the faint-hearted :)

Also, the recommended releases of WinDBG (from the Microsoft Store) no longer include gflags, though I believe once you're in the debugger it will just break at the point where the DLL can't be loaded and it's "simple" to get its expected name.

I wouldn't refuse a docs PR to add a short section pointing to this page and explaining its relevance: https://docs.microsoft.com/cpp/build/reference/dependents

I *would* stop short of writing a whole tutorial on how to do it. That's a great topic for someone's blog, and will likely get better SEO and social attention from not being in the docs.

[pombredanne]

I wouldn't refuse a docs PR to add a short section pointing to
this page and explaining its relevance:
https://docs.microsoft.com/cpp/build/reference/dependents

Steve,
would you see this as a note in https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries

What about something like this?

class ctypes.CDLL
.....

Note: On Windows a call to CDLL(name) may fail even if the DLL name exists when a dependent DLL of this DLL is found. This will lead to an OSErrror error with the message "[WinError 126] The specified module could not be found".

This error message does not contains the name of the missing DLL because the Windows API does not return this information making this error hard to diagnose.

To resolve this error and determine which DLL is missing, you need to find the list of dependent DLLs using Windows debugging and tracing tools.

See https://docs.microsoft.com/cpp/build/reference/dependents for some explanations.

[zooba]

would you see this as a note in
https://docs.python.org/3/library/ctypes.html?highlight=ctypes#loading-shared-libraries

Haven't looked at the PR, but it probably needs to be somewhere in the
import docs as well, to do with native extension modules. That's where
most people run into this. And in general the solution is either going
to involve moving/renaming files or calling os.add_dll_directory, so a
link to the latter may also be useful.

I think we've got a Sphinx tag for platform-specific information? If we
do, it should use that. (Unless I'm just thinking of the "API
availability" tag rather than a "Note" style box.)

[pombredanne]

So the other locations to add docs would be petinetially

• https://docs.python.org/3/library/os.html?#os.add_dll_directory
• https://docs.python.org/3/extending/windows.html
• https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll
• https://docs.python.org/3/using/windows.html#finding-modules

Which ones would be the best?

Also AFAIK there is no Windows Sphinx tag beyond .. availability::

[zooba]

Thanks for doing the search :)

> -
https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll

Probably not here.

> - https://docs.python.org/3/using/windows.html#finding-modules

Perhaps it is best to put a new section here like what you posted above
(but more generic for ctypes and imports), and then link to it from the
other places?

> - https://docs.python.org/3/library/os.html?#os.add_dll_directory

e.g. "This function may be used to work around <module not found> errors"

> - https://docs.python.org/3/extending/windows.html

e.g. "If your extension module relies on any DLLs other than those
included with Windows or CPython, you will need to include them or else
users may receive 'module not found' errors. See <this page> for more
details."

(Some of that text may already be there, been a while since I read that
one.)

> Also AFAIK there is no Windows Sphinx tag beyond .. availability::

Yeah, I think I was thinking of a different project. But if it's all in
Windows-specific sections anyway, and pointing towards the Windows doc
page, then it won't matter.

[zooba]

New changeset b6f2fc9 by Philippe Ombredanne in branch 'master':
bpo-25655: Improve Win DLL loading failures doc (GH-22372)
b6f2fc9

[zooba]

Thanks for the PR!