Fix Windows builds

[flavorjones]

RubyInstaller 2.4 and 2.5 depend on MSYS2, and some of MSYS2's packages are broken at the moment. The part of the build that's failing is libiconv generating a Makefile, because gawk isn't working.

I'm tracking a couple of related upstream bugs:

• gawk 4.2 libraries are broken msys2/MSYS2-packages#1127
• mc (Midnight Commander) exits with code 127 on startup msys2/MSYS2-packages#1172

Not sure what to do here. @larskanis MSYS2 seems to be really, really brittle; do you have any creative ideas on how to stabilize our dependency? This is the third time the Windows builds have broken due to upstream MSYS2 issues.

[larskanis]

All 3 MSYS2 tools (awk, mc, gdb) are running fine on my local system. I verified that my system is up-to-date (pacman -Syu twice) and that I'm using the MSYS2 version of the tools (and not as MINGW). Also Appveyor is running fine as well as building libiconv on my local system.

Could it be, that the Windows CI environment has changed? I've seen strange errors by some security tools like this.

[flavorjones]

@larskanis The windows environment used is bootstrapped from a clean windows image, and is built by this bosh packaging script.

The failures we're seeing are totally reproducible with this script. If I'm doing it wrong, it would be great to have some help to understand how to do an unattended installation of RI+MSYS2. It's not obvious.

If necessary we can ssh into the VM and pair on trying to fix the installation. I've run a wide variety of "pacman -Syuu ..." commands, and everything is telling me it's all fine.

[larskanis]

@flavorjones I tried the build task in some more environments, but I still can't reproduce your error. I used a Windows-10 (10.0.16299.309) and a Windows-Server-2016. The Windows-10 box has Defender activated and is mostly on default settings. On both boxes gawk and mc work as expected and nokogiri's rake compile or rake test run.

Which Windows-OS do you use? Which anti virus? Any addition security options (ASLR, etc.)?

In the meantime I released RubyInstaller with builtin MSYS2-Devkit. This probably makes installation on Concourse easier, especially in unattended mode.

[flavorjones]

@larskanis The setup is a bit opaque, so I apologize for not documenting it better.

The windows image I'm using is a bosh stemcell for Windows 2012R2 on GCP, version 1200.16:

https://bosh.io/stemcells/bosh-google-kvm-windows2012R2-go_agent

Not sure if there's anything odd in there, I'm tagging @mhoran who helps maintain that image in case he has some insight. MSYS2 used to build fine, it's only the past few months that there have been issues.

The script I'm using to bootstrap it is here:

https://github.com/flavorjones/windows-ruby-dev-tools-release/blob/master/packages/ruby-2.5/packaging#L58-L72

I'll check out the release with built-in msys2-devkit this week.

[flavorjones]

You can actually see the history of the installation process I've been using here a little bit better:

https://github.com/flavorjones/windows-ruby-dev-tools-release/commits/master/packages/ruby-2.4/packaging

Notably between September and January everything was fine. I'm pretty sure been on the 1200.latest stemcell series the whole time.

[larskanis]

Windows still has some weird limitations like a MAX_PATH of 260 characters. I had to shorten the path for RubyInstaller build, lately. So, I tested with the nokogiri repository in the same long path as on the Concourse CI node, but it worked on my system. However I didn't test with MSYS2 or Ruby within a path with near to or more than MAX_PATH characters.

[mhoran]

Interesting. The MAX_PATH issue could certainly be contributing to the issue. I wonder if the paths are of variable length, hence the flakiness?

@flavorjones do you know if the issues coincided with a stemcell bump of any sort? Windows Defender should be disabled, so I don't expect that'd be interfering in any way.

[flavorjones]

Sorry for not responding. This is not an issue with rubyinstaller, the underlying issue is that gawk will not run and always exits with 127 and raises an exception of c0000139 (you can read more at msys2/MSYS2-packages#1127 and msys2/MSYS2-packages#1172).

[flavorjones]

Note for future self: some useful concourse windows worker configs are here: https://gist.github.com/esauser/d8e937568e93dce902b0984abcab8690

[flavorjones]

Closing this; Appveyor has been fine and I'm planning to move to Github Actions, anyway.