typeshed not being found on windows user install

[Summertime]

PS D:\testing> git clone https://github.com/python/mypy.git
# no errors

PS D:\testing> cd .\mypy\

PS D:\testing\mypy> git submodule update --init typeshed
# worked, checked out '55dbb967ad006d5835590c53ab023774dc5a1341'

PS D:\testing\mypy> pip install --user --upgrade .
Processing D:\testing\mypy
Requirement already up-to-date: typed-ast<1.2.0,>=1.1.0 in c:\users\[redacted!]\appdata\roaming\python\python36\site-packages (from mypy===0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca)
Installing collected packages: mypy
  Running setup.py install for mypy ... done
Successfully installed mypy-0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca

PS D:\testing\mypy> mypy setup.py  # to test
Could not resolve typeshed subdirectories. If you are using MyPyfrom source, you need to run "git submodule --init update".Otherwise your MyPy install is broken.

The data directory (after searching for typeshed) appears to be %AppData%\Python\lib\mypy

full path: C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy

PS D:\testing\mypy> ls $Env:AppData\Python\lib\mypy

typeshed
xml

using this directory leads to more expected results

PS D:\testing\mypy> mypy --custom-typeshed-dir $Env:AppData\Python\lib\mypy\typeshed setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable

[gvanrossum]

It would seem that mypy's search for typeshed needs to be augmented with a case for this type of configuration. Mypy itself is at $HOME\appdata\roaming\python\python36\site-packages\mypy and typeshed is at $HOME\AppData\Python\lib\mypy\typeshed. Could you take a debugger and step through the default_data_dir() function (or insert some print statements) to find out where it goes wrong? I'm guessing the reason might be that it's looking for a component of the path named lib, but in your case there isn't one.

How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

[emmatyping]

I can debug this.

How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

For me the default 3.x install directory for the newer installers (3.5+) point to %USERPROFILE%\AppData\Local\Programs\Python\PythonXY instead of C:\PythonXY.

[Summertime]

sorry for a slightly-too-late response, the fix doesn't seem to work for me :(

https://gist.github.com/Summertime/403ce90d067253669796326212becce0 probably-overkill install logs are here for the exact paths of everything (installing from the latest merge)

I can give a shot at trying to work out the issue in a couple days, however the problem might be obvious from the install logs already, I'm not too sure.

[emmatyping]

Hm okay. Based on the output of the log it seems that you are using a custom compiled Python. Is that true? Sorry that fix didn't work for you.

[Summertime]

no custom compiling! I reinstalled python just to make sure, clicked the main 3.x.x download link on python.org, did the following, then ran pip install . --user in the same folder as before, same result!

[emmatyping]

Could you tell me what where mypy gives you now that you re-installed Python? Thanks!

[gvanrossum]

I wonder if (at least on Windows) the search code should apply os.path.normpath() before comparing?

…

On Sun, Sep 24, 2017 at 3:13 PM, Ethan Smith ***@***.***> wrote: Could you tell me what where mypy gives you now that you re-installed Python? Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3988 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMkCtUmPLkNqTdu6tANJwr6ik7HrBks5sltP0gaJpZM4PheqA> .

-- --Guido van Rossum (python.org/~guido)

[emmatyping]

Could you point out the comparison that you are thinking of? The part of the code in question is independent of the OS path separator to my knowledge. I'm thinking of here in particular

mypy/mypy/build.py

Line 231 in 6c409b4

if os.path.basename(lib) in ('lib', 'lib32', 'lib64') \

[gvanrossum]

I wasn't thinking of os.sep but of different cases for lib or python (on the next line). In particular the OP has Python (%AppData%\Python\lib\mypy and C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy).

[emmatyping]

Confusingly, %APPDATA% expands to %USERPROFILE%\AppData\Roaming or C:\Users\<username>\AppData\Roaming, so essentially the two are the same.

[gvanrossum]

I'm working on the hypothesis that ...\appdata\roaming\python\python36\site-packages (as quoted by OP) might be using capitalization everywhere, i.e. ...\AppData\Roaming\Python\Python36\site-packages. The only time this path is quoted by the OP it's clearly already lowercased by pip, so we have no hard evidence against this hypothesis. And if that's true, the algorithm (which seems to be searching for the python36 segment of the path) would fail since it's doing a case-sensitive comparison.

I have no idea what installer was used, so I don't want to exclude this hypothesis yet -- it seems to be using capitalization in earlier segments of the path (at least ...\AppData\Roaming\Python are quoted by the OP). Also, I found this about AppData\Roaming: https://askleo.com/whats-the-appdata-roaming-folder/