Symlinks don't work right on Windows

[gvanrossum]

As noted in #1173, git does not support symlinks on Windows. This means that our use of symlinks in typeshed (admittedly pretty limited) is entirely broken on Windows, since a symlink just shows up as a plain text file on Windows whose contents is the name of the file to which it points.

See http://stackoverflow.com/questions/5917249/git-symlinks-in-windows for more info.

AFAIK there are two ways out:

1. Get rid of all symlinks in favor of copies -- but then there's the risk of divergence (e.g. between stdlib/3.4/enum.pyi and third_party/3/enum.pi).

2. Write a post-checkout script that runs on Windows and detects symlinks and replaces them with the contents of the referenced file (see stackoverflow above for more details).

Note that while this problem doesn't appear in current tests, it has caused AppVeyor tests to fail (e.g. https://ci.appveyor.com/project/gvanrossum/mypy/build/1.0.1171/job/eyt2lvjrs2sq8njv) and there are several files in the Windows-specific wheel files on PyPI that are indeed just little text files like described above.

[matthiaskramm]

Strongly in favor of (2). Symlinks are a really useful concept in typeshed.

How urgently do we need a fix/workaround for this?

[gvanrossum]

Well, 0.501 was released March 2nd, so I think it's about time for the next mypy release. Then again, this isn't a regression. So if we don't have a solution we can live with it.

[matthiaskramm]

There's also the rather hacky option 3,

3. Teach mypy and pytype to treat files that only contain a path (and nothing else) as a link to the real file.

[gvanrossum]

That should be easy enough for mypy. I would do this only on Windows to limit the risk of misfires.

@JukkaL, @ddfisher, thoughts?

[JukkaL]

That sounds acceptable if we only use it on Windows. The only issues I'm a little worried about are contributor confusion, since the link files probably can't have comments in them, and the ability to create symlinks on Windows. Code duplication is probably worse though. I think that having a Windows-specific section in the typeshed readme where we discuss this would be sufficient.

[ddfisher]

I'm opposed to 3. It's hacky, and I'd expect it to cause problems in the long run. 2 feels like the correct solution to me -- this is a build-time issue and should be fixed with a build-time fix.

[matthiaskramm]

Ok, it seems that we at least all agree that we want 2 or 3, and not 1. Do we want to do 3 (pathline hack) until one of us gets around to implementing 2 (git post-checkout hook)? Or leave this problem alone, for the time being?

[ddfisher]

I think if we implement 3 as a temporary measure it will live forever. Let's do the right thing and implement 2 -- it doesn't seem particularly hard.

[JukkaL]

it doesn't seem particularly hard.

Famous last words :-) My experience suggests that Windows-specific things tend to be harder than you'd expect.

After some more thought, I no longer like 3, but I'm also strongly opposed to 2 -- I don't like the idea of having Windows-specific workflows and Windows-specific scripts that potentially nobody in the core teams uses regularly, and that may be hard to get working on everybody's current and future Windows environments. I also generally don't like post-checkout scripts and other git magic, since when things go wrong, most people don't know how they work and don't expect them.

I have two additional proposals that wouldn't require any Windows-specific code and would avoid the problems with 1:

4. Use duplicate files but have a test that verifies that certain files always have the same contents. These files can be described in a simple text file so adding new "implicitly linked" files would be easy. The test would be trivial to implement and no changes are needed to type checkers.
5. Implement custom symlinks (e.g. foo.pyi.link) and update tools to look for them in addition to .pyi files. The files itself would just contain the link target path and perhaps an optional comment. Basically, we'd use 3 on every platform for consistency. Though fairly simple, this would cause extra work for everybody who wants to use typeshed and there are some corner cases that could cause incompatibility between tools.

[gvanrossum]

I like (4). I agree with all Jukka's arguments.

[JelleZijlstra]

I suggested (4) in #1173 before. I can implement it, hopefully this weekend.

[gvanrossum]

Great!

[ddfisher]

Hmm. Supposedly recent versions of git support symlinks on Windows if you clone the repository with core.symlinks=true. Is that an option here?

[gvanrossum]

If that works it would be brilliant. Does anyone know how to set that flag to true in AppVeyor? Though from the docs at https://git-scm.com/docs/git-config I cannot tell whether setting it to True on Windows would have the desired effect. Searching http://stackoverflow.com/questions/5917249/git-symlinks-in-windows for "core.symlinks" suggests that it works if you can run a recent git version in Administrator mode like this: git clone -c core.symlinks=true <URL>