[ci] add CMake + R 3.6 test back (fixes #3469)

[jameslamb]

I think this PR fixes #3469 😬

My theory

Using https://github.com/nelsonjchen/reverse-rdp-windows-github-actions, (mentioned in #3469 (comment)), I RDP'd into a GitHub Actions runner running the windows-latest image and try some experiments. I discovered something confusing and scary that I think could be the root cause of CI instability mentioned in #3469.

I manually copied the content of test_r_package_windows.ps1 into a Powershell shell running in that runner, a few lines at a time.

Immediately after installing Rtools, the directory C:\Rtools was visible in Windows Explorer but C:\Rtools\mingw_64 didn't exist yet! Even though, in powershell, the command had exited successfully and returned control to the shell.

After around 20 seconds, the mingw_64 directory finally popped up.

And when I say "it popped up", I don't think that's just the effect of buffering or a delay in showing the files. I hit "refresh" in Windows Explorer several times and did not see mingw_64 show up until after 20 seconds or so had passed. So I think the install returned control to powershell before all the files were actually written, somehow 😱 .

This type of problem could explain what we've seen in #3469, where the R 3.6 + CMake job fails but not always. Recall that the failures look like this:

  The C compiler

    "C:/Rtools/mingw_64/bin/gcc.exe"

  is not able to compile a simple test program.

...

    C:/Rtools/mingw_64/bin/../lib/gcc/x86_64-w64-mingw32/4.9.3/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lshell32

It makes sense to me that that could happen if all of the contents of C:\Rtools\mingw_64 (which include a lot of things used by ld.exe) aren't available yet when CMake tries to compile LightGBM!

Proposed Changes

• just sleeping for 1 minute after installing Rtools
• removing possibly-conflicting files pre-installed by GitHub from PATH ([R-package] [ci] mingw gcc often fails on CMake builds #3469 (comment))

I know fixing a problem by just introducing a sleep isn't ideal, but I think it would be difficult to figure out the full list of files needed by ld.exe when compiling LightGBM. I found that with this fix, I got two consecutive runs of the R 3.6 + CMake + Windows job (5 builds on each run) to complete successfully on my fork.

• https://github.com/jameslamb/LightGBM/actions/runs/631289073
• https://github.com/jameslamb/LightGBM/actions/runs/631256411

[jameslamb]

these paths all had some msys2 or mingw components in them.

I dumped the full PATH below so you can see what's there, including why I needed both chocolatey and Chocolatey 😂

$env:PATH.split(";")

full contents of PATH before removals

C:\Program Files\PowerShell\7
C:\Users\runneradmin\.dotnet\tools
C:\Program Files\MongoDB\Server\4.4\bin
C:\aliyun-cli
C:\vcpkg
C:\cf-cli
C:\Program Files (x86)\NSIS\
C:\Program Files\Mercurial\
C:\hostedtoolcache\windows\stack\2.5.1\x64
C:\ProgramData\chocolatey\lib\ghc.8.10.3\tools\ghc-8.10.3\bin
C:\Program Files\dotnet
C:\mysql-5.7.21-winx64\bin
C:\Program Files\R\R-4.0.4\bin\x64
C:\SeleniumWebDrivers\GeckoDriver
C:\Program Files (x86)\sbt\bin
C:\Rust\.cargo\bin
C:\Program Files (x86)\GitHub CLI
C:\Program Files\Git\bin
C:\Program Files (x86)\pipx_bin
C:\hostedtoolcache\windows\go\1.15.8\x64\bin
C:\hostedtoolcache\windows\Python\3.7.9\x64\Scripts
C:\hostedtoolcache\windows\Python\3.7.9\x64
C:\hostedtoolcache\windows\Ruby\2.5.8\x64\bin
C:\Program Files\Java\jdk8u282-b08\bin
C:\npm\prefix
C:\Program Files\Microsoft SDKs\Azure\Azure Dev Spaces CLI
C:\Program Files\Microsoft SDKs\Azure\Azure Dev Spaces CLI\
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin
C:\ProgramData\kind
C:\windows\system32
C:\windows
C:\windows\System32\Wbem
C:\windows\System32\WindowsPowerShell\v1.0\
C:\windows\System32\OpenSSH\
C:\ProgramData\Chocolatey\bin
C:\Program Files\Microsoft\Web Platform Installer\
C:\Program Files\Docker
C:\Program Files\PowerShell\7\
C:\Program Files\dotnet\
C:\Program Files\Microsoft SQL Server\130\Tools\Binn\
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\
C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\
C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\
C:\Program Files\nodejs\
C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin
C:\ProgramData\chocolatey\lib\maven\apache-maven-3.6.3\bin
C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code
C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager
C:\Program Files\OpenSSL\bin
C:\Strawberry\c\bin
C:\Strawberry\perl\site\bin
C:\Strawberry\perl\bin
C:\Program Files\Git\cmd
C:\Program Files\Git\mingw64\bin
C:\Program Files\Git\usr\bin
c:\tools\php
C:\Program Files (x86)\sbt\bin
C:\Program Files\TortoiseSVN\bin
C:\SeleniumWebDrivers\ChromeDriver\
C:\SeleniumWebDrivers\EdgeDriver\
C:\Program Files\CMake\bin
C:\Program Files\Amazon\AWSCLIV2\
C:\Program Files\Amazon\SessionManagerPlugin\bin\
C:\Program Files\Amazon\AWSSAMCLI\bin\
C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin
C:\Program Files (x86)\Microsoft BizTalk Server\

[jameslamb]

I had to look this up, but Strawberry is a perl environment for Windows: https://strawberryperl.com/. It's Program Files contain some mingw tools used to compile C programs

[StrikerRUS]

Nice idea and implementation of PATH cleaning! Thank you very much! Please take a look at some my comments below.

[StrikerRUS]

Maybe we can use Start-Process -FilePath Rtools.exe -NoNewWindow -Wait -ArgumentList ... above to not wait manually?

[StrikerRUS]

Oops! Seems I missed that:

I hit "refresh" in Windows Explorer several times and did not see mingw_64 show up until after 20 seconds or so had passed. So I think the install returned control to powershell before all the files were actually written, somehow 😱 .

Indeed, very interesting observation! 👍

Given that, please ignore my comment above about -Wait argument (it will not help). Instead, I believe we should do something like

while True:
    sleep(1)
    if exists("$RTOOLS_INSTALL_PATH/mingw_64"):
        break

instead of hardcoded sleep(60).

[jameslamb]

😱 I didn't know about -Wait! I don't write Powershell code anywhere else except this one script in this project, so I'm not very knowledgeable about it. Yes that seems a lot better than a sleep! Let me try that.

[StrikerRUS]

Or you think that writing all files into mingw_64 takes some significant time and this is not just about delay in mingw_64 folder creation?

[jameslamb]

ha we posted our responses at about the same time, I didn't see your second comment.

I don't think this while loop approach is exactly correct a reliable fix. The directory mingw_64 could exist without all of the other files and folders in it being populated yet. This is what I meant by this explanation in the PR description

know fixing a problem by just introducing a sleep isn't ideal, but I think it would be difficult to figure out the full list of files needed by ld.exe when compiling LightGBM

[StrikerRUS]

OK, I see... Could then please try -Wait workaround in RDP before merging this? I'm not an expert in PowerShell too, so maybe some magic will happen 😄

[jameslamb]

sure! I won't be able to test it until later tonight though, maybe 6-8 hours from now

[StrikerRUS]

No problem!
Maybe this is indeed a "recommended" official way of installation:
https://github.com/r-windows/rtools-base/blob/e60c36de70de7dedce89577c11d2fbe6008b2df4/scripts.ps1#L39

[jameslamb]

oh nice, I forgot about r-windows/r-tools. I vaguely remember reading a blog post about it once. If that's how Jeroen does it, it's definitely a good sign.

Happy to say that switching to a -Wait did work! I could see that the Rtools install command didn't return control to powershell until both ming_32 and mingw_64 directories were created.

[StrikerRUS]

WDYT about to remove this particular directory physically?
Looks like there could be conflicts during our own installation (but for R 4.x job)

LightGBM/.ci/test_r_package_windows.ps1

Line 45 in 3a5e3c0

$RTOOLS_INSTALL_PATH = "C:\rtools40"

[jameslamb]

I originally had something like Remove-Item "C:\rtools40" -Recurse", but it was failing with an error saying something like "insufficient permission to remove this, must be an administrator or this may be a system file" (don't remember the exact error message).

Although, I was just looking at your work in RGF...maybe we could borrow your solution from there to get past that error? 😀

https://github.com/RGF-team/rgf/blob/ab328a01d3af809c0c1fd4122d8b70b9fb2e9757/.ci/r_tests_windows.ps1#L20

Remove-Item C:\rtools40 -Force -Recurse -ErrorAction Ignore

[StrikerRUS]

I think so! This command takes some time (several seconds) but I think it won't be excess and can protect us from some other problems.

[StrikerRUS]

BTW, I don't know why, but I had to manually install qpdf for Windows R jobs (I chose pacman package manager because it's bundled with Rtools and one-line solution) to pass --as-cran checks for vignettes.
Maybe this info will help you to save some time on other projects or even here in LightGBM.
https://github.com/RGF-team/rgf/blob/ab328a01d3af809c0c1fd4122d8b70b9fb2e9757/.ci/r_tests_windows.ps1#L42

[jameslamb]

thanks! Maybe we'll have to worry about qpdf on #3946 , actually, so that is timely advicce

[StrikerRUS]

Thank you very much! Hope this PR will really fix that issue this time!

[jameslamb]

As soon as #4055 builds successfully, I'll merge that, then merge master into this branch, then merge this.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.