Look into MSys2

[dscho]

I am looking into using MSys2 for future Git for Windows development, for a couple of reasons:

• MSys2 packages are much more up-to-date than MSys ones
• MSys2 offers a nice, stable package management system, pacman (ported from Arch Linux)
• MSys2 already ships with mintty, a powerful alternative to the cmd.exe command-line window
• MSys2 has 64-bit support
• MSys2 allows us to (re-)build Perl with Subversion bindings easily (at least that is the impression I got)

This ticket mainly serves as a notepad listing quirks and issues I find out while working on getting Git for Windows to build with MSys2.

It is quite easy to get started with MSys2:

1. download and install MSys2 as mentioned on https://msys2.github.io/
2. run the MINGW shell (not the MSys2 shell) from the Start Menu
3. run pacman -Sy git mingw-w64-x86_64-gcc to install Git and GCC (-S means "synchronize", i.e. install, and -y does not mean "yes" but "refresh the package list before installing the newest versions of the specified packages")
4. clone https://github.com/git-for-windows/MINGW-packages, say, into /usr/src/
5. in MINGW-packages/mingw-w64-git/, run makepkg-mingw -s (the -s says: install all dependencies, including build time dependencies, via pacman)
6. Play with the unpacked sources (they are in the src/build-x86_64/ or the src/build-i686/ subdirectory)

So it looks that it is pretty easy to get started, as well as to rebuild the MSys2 installer (which we might want to do in order to continue offering a net installer to contributors that avoids requiring manual steps).

Having said that, there are a few bumps on the road to heave: The following issues need to be solved before we can start bundling up even so much as a portable application:

• tab completion in the MSys2 shell seems to be flakey (it sometimes does not find anything, and pressing the Tab key repeatedly will not do anything in that case, but often it helps to delete the last character, retype it, and then press the Tab key again). This did not happen with the MinGW shell, funnily enough, so it might not be a huge problem for us; However, it could indicate a more problematic, deeper bug
• MSys2's openssh refuses to heed the HOME variable and always looks into /home/.ssh/
• MinGW Git built with makepkg-mingw hardcodes the absolute exec path (e.g. C:/msys64/mingw64/git-core)
• it seems that git ls-remote origin, run from a freshly-built and installed MinGW Git fails to be able to output anything, and git ls-remote origin | cat (a trick learned from working with old MSys'/MinGW's quirks) only says: output is not a tty (the exit code is 127, suggesting that some executable was not found, but it is very difficult to say which one because not even debug print statements to stderr are shown; It seems that in case of a crash or of a die(), stderr is not flushed)
• we #define some S_* flags in mingw.h that mingw-w64's headers already define. This results in tons of compile warnings that might hide valid problems
• MSys2's build recipe (called PKGBUILD; very similar to msysGit's home-brewn release.sh scripts) lists a dozen patches; We need to determine whether to reject them or whether to integrate them in our Git fork proper)
• MSys2 builds Git with ./configure while our strategy has always been to enhance the configuration in config.mak.uname; We need to decide whether we want to follow the ./configure strategy or rather continue with our tried and tested approach
• the mingw-w64-x86_64-gdb package is a 64-bit executable, but handles only 32-bit executables, it seems. This will make debugging substantially harder
• we will also need to add support for the -f flag to MSys2's kill command (Note: MSys2's kill already knows the -f flag, but there is a Bash builtin that takes precedence when calling kill without an explicit path, and that kill does not know the -f flag)

I will keep updating this report as I find out more.

[dscho]

Update: to rebuild MinGW curl, I had to import Daniel Sternberg's (i.e. the main cURL developer's) key into gpg, not into pacman via pacman-key.

In other words, I had to call gpg --keyserver hkp://pgp.mit.edu --recv-keys 78E11C6B279D5C91 before running makepkg-mingw -s in MINGW-packages' mingw-w64-curl/ directory. The symptom that this is needed:

==> Verifying source file signatures with gpg...
    curl-7.40.0.tar.bz2 ... FAILED (unknown public key 78E11C6B279D5C91)
==> ERROR: One or more PGP signatures could not be verified!

[dscho]

Another update, this time really dramatic:

git merge origin/msys2
merge: origin\msys2 - not something we can merge

In other words, MSys2 has an even worse path translation than MSys... 😦

[dscho]

Good news: the funny issue with git ls-remote goes away when recompiling myself. It seems that there was some problem with curl, maybe even caused by a version skew.

[dscho]

Sadly, there are also bad news: t0001 already fails with a Segmentation fault (!!!) when running git aliasedinit... It keeps being a challenging project!

[oysteinkrog]

I just want to let you know that I appreciate your work:)
I use MSYS2 myself and I agree that it is much better than old msys, the main problem for me is that the cygwin-based git is so slow.
If you get mingw-git working it would be awesome.
I tried to look at it myself but it is really very hard, especially if one is not all that experienced with native code and buildsystems.

[dscho]

merge: origin\msys2 - not something we can merge

This issue was probably caused by mingw_startup() not being called in the Git built from MSys2's MinGW Git build script. I will not investigate the issue more because it cannot be reproduced with the Git I built from scratch.

Given this insight, I decided that it would be better to stick with our config.mak.uname-based approach to configure the build (as opposed to the ./configure-based method proposed by MSys2's mingw-w64-git).

Sidenote: I do plan to offer a PR to MSys2's MINGW-packages repository once I completed my work successfully.

The hard crash I saw in t0001 (i.e. when running git aliasedinit) was caused by unsetenv()'s not knowing about our UTF-8 environment: when looking up an environment variable, our binary search would try to compare a string against NULL. I added a commit that errors out explicitly when NO_UNSETENV is defined and the mingw_startup() code is active at the same time.

[dscho]

Another set of updates: the two most crucial problems I addressed today are:

1. The TZ environment variable is stripped out by the MSys2 runtime when executing non-MSys2 programs, causing regressions in the test suite. While the rationale is sound (Win32 API treats TZ differently than POSIX), making the information unavailable is not a proper solution. My proposed fix renames the environment variable to MSYS2_TZ before passing it on to non-MSys2 programs (including Git).
2. The MSys2 POSIX <-> Windows conversion (p2w) is a complete rewrite of MSys1' p2w code, unfortunately re-introducing some bugs such as converting ~/file to ~C:\msys64\file. My current version of a fix most likely will evolve, but for the moment, we simply skip the conversion when a tilde, a quote or wildcards were found before the first slash or white space character.

The current version of my fixes can be seen here: https://github.com/dscho/Cygwin/branches

[dscho]

Update: as of last night, the following tests still fail: t0060, t1504, t1506, t1508, t1513, t3300, t3301, t3400, t3404, t3600, t3707, t3902, t4016, t4056, t4135, t4208, t4211, t5000, t5500, t5516, t5570, t5601, t5602, t5801, t6006, t6132, t7201, t7400, t7800, t9001, t9010, t9350, t9700, t9903 and t9300 waits forever.

While this is an impressive list of failures, the number of root causes is likely much smaller. I expect most of the issues to go away with a more careful p2w handling.

[dscho]

Progress: after skipping filenames-with-tabs tests explicitly (MSys2 allows you to create such files even if the Win32 API cannot handle them), and after working quite a bit on the posix-to-windows path mangling as well as the environment mangling of MSys2, the list of failing tests shrunk to t3703, t4056, t5000, t5500, t5516, t5570, t5601, t5602, t5801, t7400, t7800, t9001, t9010, t9350, t9700 and t9300 still has five test cases that wait for input on stdin by mistake.

The current patches can be seen here: https://github.com/dscho/Cygwin/compare/git-for-windows and dscho/git@git-for-windows...msys2 (to be cleaned up)

[dscho]

A ton of work later, t7800 and 9700 are the only tests that expose problems.

[hvoigt]

Wow, @dscho I just have to say: Your work is awesome!

[dscho]

t7800 down, t9700 to go.

[dscho]

Done. While at it, I also documented packaging a bit more, as well as started describing how to run and debug the regression tests.