Notebook does not work on Windows, "ImportError: DLL load failed while importing shell: The specified procedure could not be found."

[phil-blain]

What happened: I tried to open the "Calculating ENSO with Xarray" tutorial on my Windows laptop. It does not work because of some error with package pywin32.

What you expected to happen: It works.

Minimal Complete Verifiable Example:

1. Install Miniconda (well, Miniforge)
2. Follow https://foundations.projectpythia.org/preamble/how-to-use.html:

conda env create --force -f https://raw.githubusercontent.com/ProjectPythia/pythia-foundations/main/environment.yml
conda activate pythia-book-dev
git clone git@github.com:ProjectPythia/pythia-foundations.git
cd pythia-foundations
jupyter notebook

then open "enso-xarray" Notebook and execute the first cell. Result:

ImportError: DLL load failed while importing shell: The specified procedure could not be found.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_31200/1210338269.py in <module>
      2 import matplotlib.pyplot as plt
      3 import xarray as xr
----> 4 from pythia_datasets import DATASETS

~\AppData\Local\miniforge3\envs\pythia-book-dev\lib\site-packages\pythia_datasets\__init__.py in <module>
      4 from pkg_resources import DistributionNotFound, get_distribution
      5 
----> 6 from .datasets import DATASETS, locate
      7 
      8 try:

~\AppData\Local\miniforge3\envs\pythia-book-dev\lib\site-packages\pythia_datasets\datasets.py in <module>
      3 
      4 DATASETS = pooch.create(
----> 5     path=pooch.os_cache('pythia-datasets'),
      6     base_url='https://github.com/ProjectPythia/pythia-datasets/raw/main/data/',
      7     env='PYTHIA_DATASETS_DIR',

~\AppData\Local\miniforge3\envs\pythia-book-dev\lib\site-packages\pooch\utils.py in os_cache(project)
     97 
     98     """
---> 99     return Path(appdirs.user_cache_dir(project))
    100 
    101 

~\AppData\Local\miniforge3\envs\pythia-book-dev\lib\site-packages\appdirs.py in user_cache_dir(appname, appauthor, version, opinion)
    291         if appauthor is None:
    292             appauthor = appname
--> 293         path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))
    294         if appname:
    295             if appauthor is not False:

~\AppData\Local\miniforge3\envs\pythia-book-dev\lib\site-packages\appdirs.py in _get_win_folder_with_pywin32(csidl_name)
    478 
    479 def _get_win_folder_with_pywin32(csidl_name):
--> 480     from win32com.shell import shellcon, shell
    481     dir = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)
    482     # Try to make this a unicode path because SHGetFolderPath does

ImportError: DLL load failed while importing shell: The specified procedure could not be found.

Anything else we need to know?: I read https://github.com/mhammond/pywin32#the-specified-procedure-could-not-be-found--entry-point-not-found-errors but I'm not administrator on my computer...

Environment:

• Python version: 3.8.13
• Operating System: Windows 10 1909
• Install method (conda, pip, source): conda

[phil-blain]

I did try pywin32_postinstall.py -install after activating the environment and before running jupyter notebook, and the same error happens.

Output:

 PS C:\Users\me> pywin32_postinstall.py -install
Parsed arguments are: Namespace(install=True, remove=False, wait=None, silent=False, quiet=False, destination='C:\\Users\\me\\AppData\\Local\\miniforge3\\Lib\\site-packages')
Copied pythoncom39.dll to C:\Users\me\AppData\Local\miniforge3\pythoncom39.dll
Copied pywintypes39.dll to C:\Users\me\AppData\Local\miniforge3\pywintypes39.dll
You do not have the permissions to install COM objects.
The sample COM objects were not registered.
NOTE: PyWin32.chm can not be located, so has not been registered
Registered help file
Pythonwin has been registered in context menu
Creating directory C:\Users\me\AppData\Local\miniforge3\Lib\site-packages\win32com\gen_py
Can't install shortcuts - 'C:\\Users\\me\\AppData\\Roaming\\Microsoft\\Windows\\Start Menu\\Programs\\Python 3.9' is not a folder
The pywin32 extensions were successfully installed

[phil-blain]

I also read https://jupyter-notebook.readthedocs.io/en/latest/troubleshooting.html#resolving-pywin32-issues and tried using:
python.exe $env:CONDA_PREFIX/Scripts/pywin32_postinstall.py -install:

Parsed arguments are: Namespace(destination='C:\\Users\\me\\AppData\\Local\\miniforge3\\envs\\pythia-book-dev\\Lib\\site-packages', install=True, quiet=False, remove=False, silent=False, wait=None)        
Copied pythoncom38.dll to C:\Users\me\AppData\Local\miniforge3\envs\pythia-book-dev\pythoncom38.dll
  Copied pywintypes38.dll to C:\Users\me\AppData\Local\miniforge3\envs\pythia-book-dev\pywintypes38.dll
You do not have the permissions to install COM objects.
  The sample COM objects were not registered.
      NOTE: PyWin32.chm can not be located, so has not been registered 
 Registered help file
     Failed to register pythonwin as editor
   Traceback (most recent call last):
 File "C:\Users\me\AppData\Local\miniforge3\envs\pythia-book-dev/Scripts/pywin32_postinstall.py", line 524, in install
  RegisterPythonwin(True, lib_dir)
       File "C:\Users\me\AppData\Local\miniforge3\envs\pythia-book-dev/Scripts/pywin32_postinstall.py", line 328, in RegisterPythonwin
from win32com.shell import shell, shellcon
   ImportError: DLL load failed while importing shell: La procédure spécifiée est introuvable.
      Creating directory C:\Users\me\AppData\Local\miniforge3\envs\pythia-book-dev\Lib\site-packages\win32com\gen_py
       DLL load failed while importing shell: La procédure spécifiée est introuvable.
   The pywin32 extensions were successfully installed.

Again, same error when executing the first cell.

[phil-blain]

A short and faster reproducer:

mamba env create -f https://raw.githubusercontent.com/ProjectPythia/pythia-foundations/main/environment.yml
conda activate pythia-book-dev
python -c 'import pythia_datasets'

[brian-rose]

@phil-blain thanks for this concise reproducer!

It doesn't sound like the error has anything to do with Pythia content per se, but maybe someone on our team has some knowledge about pywin32 and Jupyter notebooks.

[phil-blain]

Yes I agree. I just wanted to report since it's the environment description you ship with the repo that creates this user experience on Windows...

[phil-blain]

OK, even smaller env that reproduces:

mamba create -n test python=3.8 pythia-datasets pywin32
conda activate test
 python -c 'import pythia_datasets'

Without =3.8 it installs Python 3.10, and this works. So something in the rest of the packages listed in https://github.com/ProjectPythia/pythia-foundations/main/environment.yml seems to constrain Python to 3.8, in which apparently pywin32 interacts badly ...

[phil-blain]

Ok, wrote too fast, the environment.yml itself constrains Python to 3.8 ...

[phil-blain]

And removing =3.8 from the environment YAML file does result in a full working environment.

I could submit a PR to do that.

[brian-rose]

Interesting, I can't remember if there was a good reason that the pythia-book-dev environment was pinned to Python 3.8.

I suggest submitting a PR that removes that, and we'll see if all the tests pass.

[phil-blain]

PR is here: #279