WindowsError: [Error 0] The operation completed successfully for exe() after upgrading to 5.5.0

[jablonskim]

Tested with Python 2.7.13 and 3.7.1, Windows 10 1809 (10.0.17763.0).
psutil version: 5.5.0

It fails on 'Secure System' process, only when Python was run as an Administrator:

Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> [p for p in psutil.process_iter()]
[psutil.Process(pid=0, name='System Idle Process', started='2018-11-06 04:18:12'), psutil.Process(pid=4, name='System', started='2018-11-06 04:18:12'), Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\psutil\__init__.py", line 395, in __str__
    info["name"] = self.name()
  File "C:\Python27\lib\site-packages\psutil\__init__.py", line 609, in name
    name = self._proc.name()
  File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 635, in wrapper
    return fun(self, *args, **kwargs)
  File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 687, in name
    return py2_strencode(os.path.basename(self.exe()))
  File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 635, in wrapper
    return fun(self, *args, **kwargs)
  File "C:\Python27\lib\site-packages\psutil\_pswindows.py", line 701, in exe
    return py2_strencode(convert_dos_path(cext.proc_exe(self.pid)))
WindowsError: [Error 0] The operation completed successfully

[giampaolo]

Mmm not good, and I can't reproduce it. Do you have VS? Could you re-compile psutil from sources and see where exactly the error originates from?

[jablonskim]

Yes, sure. I will try to provide more details till end of the day or next Monday.

About reproducing it: just running it from Administrator account is not enough. Try starting cmd or powershell using right click and 'Run as an Administrator' and then run Python from it. Without running as an Administrator it runs perfectly fine.

[jablonskim]

After some investigation, I realized that the issue is related to Virtual Secure Mode in Windows (http://woshub.com/virtual-secure-mode-vsm-in-windows-10-enterprise/). When I disabled Hyper-V using bcdedit /set hypervisorlaunchtype off, process_iter() works correctly because there is no 'Secure System' at all.

The problem is in this line: https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c#L767
The OpenProcess function returns valid handle for 'Secure System' process, GetProcessImageFileNameW also ends with success, so the GetLastError() returns 0. Unfortunatelly, the name returned from GetProcessImageFileNameW is empty, so the return value, which is the length of the name, is zero. According to the documentation it should be zero only in case of failure.
I tried to change the GetProcessImageFileNameW to QueryFullProcessImageNameW (with PROCESS_NAME_NATIVE flag) which returns BOOL and it works fine then, except that the process names for some secure processes are empty.

So the partial output looks like this:

In [2]: [p for p in psutil.process_iter()]
Out[2]:
[psutil.Process(pid=0, name='System Idle Process', started='13:53:01'),
 psutil.Process(pid=4, name='System', started='13:53:01'),
 psutil.Process(pid=56, name='', started='13:53:01'),
 psutil.Process(pid=72, name='svchost.exe', started='13:53:11'),
 psutil.Process(pid=104, name='', started='13:53:01'),
 ...]

[jablonskim]

I additionally tried to raise AccessDenied if returned length is zero, and then it's working like for other privileged processes:

[psutil.Process(pid=0, name='System Idle Process', started='13:53:01'),
 psutil.Process(pid=4, name='System', started='13:53:01'),
 psutil.Process(pid=56, name='Secure System', started='13:53:01'),
 psutil.Process(pid=72, name='svchost.exe', started='13:53:11'),
 psutil.Process(pid=104, name='Registry', started='13:53:01'),
 ...]

[giampaolo]

Thanks for investigating this. It looks like the culprit is QueryFullProcessImageNameW which returns 0 (failure). The subsequent call to GetLastError() returns 0 but it should have been ERROR_ACCESS_DENIED instead (so that is the proper fix).

[giampaolo]

OK, I should have fixed this in 6f4a622. To check everything's fine could you please run the following script in "Virtual Secure Mode"? I want to make sure there are no other APIs returning errcode 0 instead of ERROR_ACCESS_DENIED.

import psutil
for p in psutil.process_iter():
    p.as_dict()

[jablonskim]

Yes, the as_dict() works.

For now, there is a non-empty name for 'Secure System' process, but (for me) there are 2 more 'secure' processes and the name is empty for them:

[psutil.Process(pid=0, name='System Idle Process', started='13:53:01'),
 psutil.Process(pid=4, name='System', started='13:53:01'),
 psutil.Process(pid=56, name='Secure System', started='13:53:01'),
 psutil.Process(pid=72, name='svchost.exe', started='13:53:11'),
 psutil.Process(pid=104, name='', started='13:53:01'),
 ...]

I will try to check the GetLastError() value for them too.

For the previous comment: The QueryFullProcessImageNameW actually returns True (success) and copies a string to the name parameter, but the string is empty. The GetProcessImageFileNameW (currently used in the code) returns the length, which is zero - docs says that the return value is the length, or zero in case of failure, but in our case the length is also zero - the same as for failure.
But yes, in this situation both functions should probably fail and the GetLastError should return ERROR_ACCESS_DENIED.

[jablonskim]

The GetProcessImageFileNameW and QueryFullProcessImageNameW are somehow broken in this case. They should return full path like \Device\HarddiskVolume2\Windows\System32\svchost.exe but for process with PID 104 it returns now just Registry string, so the convert_dos_path cannot parse it properly and returns empty string.

Additionally, I rebooted my PC and now the returned string for 'Registry' process is empty and everything works fine... But I tested it also on Windows Server and the returned string is Registry. So generally, there is no exception but sometimes the name for some processes is empty.

EDIT: This was actually not true. I accidentally run my test compilation for the second test. My mistake.

[jablonskim]

Ok, to sum up:

1. For PID 75, GetProcessImageFileNameW returns 52 (length), QueryFullProcessImageNameW returns TRUE. The returned name is for both '\Device\HarddiskVolume2\Windows\System32\svchost.exe'.
2. For PID 4, GetProcessImageFileNameW returns 0 (length), QueryFullProcessImageNameW returns TRUE. The returned name is for both '' (empty string).
3. For PID 104, GetProcessImageFileNameW returns 8 (length), QueryFullProcessImageNameW returns TRUE. The returned name is for both 'Registry'.

In my opinion, the solution is to change GetProcessImageFileNameW to QueryFullProcessImageNameW (so it is possible to check the success/failure return value), remove the check for if (GetLastError() == 0) and check if returned strings starts with device path form: \Device\. What do you think about this idea?

Another thing to check: QueryFullProcessImageNameW with flags set to 0 according to the docs should return 'Win32 path format' so there is no need to convert it back to Windows version. I do not have access to my setup now, I can check it on Monday.

[giampaolo]

Avoiding to convert the DOS/win32 path format would be a clear win. Before proceeding though I would investigate how both APIs behave and if there are differences in terms of returned paths and (above all) permissions. QueryFullProcessImageNameW may work better but we should first make sure it won’t result in more AccessDenied exceptions.