Get Windows playbooks production ready

[sxa]

General issue to cover work being done to make the Windows playbooks "production ready". Should also result in some setup doc.

Info on setting up the communication to windows boxes (since it uses WinRM rather than ssh generally, and the pywinrm package to communicate with it) are at https://docs.ansible.com/ansible/latest/user_guide/windows.html

See also #300 which has been pending for a while.

[sxa]

FYI @CJKwork

[sxa]

Related: #378 #367

[karianna]

Build machines should always have rsync installed via Cygwin. I'm not sure if/how Ansible can cover that.

[sxa]

Yep we can cover by ansible - it's just another dependency we need to call out explicitly :-)

[jdekonin]

@cwesMills added #566, #563, #560 to further get windows playbooks functional. @cwesMills were there other changes required that you came across on your 2012 install?

[sxa]

@cwesMills Has your work/testing been based on PR300 as a starting point?

[cwesMills]

I did not use PR300 as a reference, but the changes I had to make were:

• Install nasm by downloading https://www.nasm.us/pub/nasm/releasebuilds/2.13.03/win64/nasm-2.13.03-win64.zip and extracting it to C:\openjdk\nasm-2.13.03
• Install clang based on steps provided in this PR
• Run ssh-host-config based on these steps. With the exception, when prompted Enter the value of CYGWIN for the daemon: the value should be nothing.
• The jdk I used didn't have the proper cacerts (known issue), so I had to copy it from an existing machine

[sxa]

What do we need an ssh server config for?

[cwesMills]

It is needed to setup cygwin ssh, so the jenkins master can ssh into the machine.

[cwesMills]

I've just added #589 so windows playbooks will install nasm.

[sxa]

It is needed to setup cygwin ssh, so the jenkins master can ssh into the machine.

At the moment the adoptopenjdk machines start the jenkins slave as a service, not via ssh so I'm not sure at the moment that I want another port exposed on the adoptopenjdk windows servers.

[jdekonin]

Are you wanting cygwin removed from the playbook, because its already present. My suggestion would be to leave it and modify such that the role is run when requested. In that way other projects that don't have the option of jenkins connection method can enable and use.

EDIT: The cygwin roll does not configure an ssh server in its current form. This is an action that would need to be performed after the playbook is run. I don't believe there is any issue here.

[sxa]

cygwin 100% has to stay there and must not be removed - it's needed for the build scripts we use and a lot of the test infrastructure - my comment was purely about sshd :-)

[sxa]

@cwesMills What's left to get the Windows playbooks into a fulling working state?

[cwesMills]

I've fixed a couple of the issues I found with the playbooks, but I don't know if more changes are needed. Currently I have other work that has my priority.

[sxa]

For reference, next steps when you can get back to it:

1. Try running playbooks on a new machine and see if they run to completion
2. See if the machine will be able to git clone https://github.com/AdoptOpenJDK/openjdk-build and then
3. From that repository, CONFIGURE_ARGS_FOR_ANY_PLATFORM="--with-target-bits=32 --target=x86 --with-freetype-include=/cygdrive/c/openjdk/freetype/include --with-freetype-lib=/cygdrive/c/openjdk/freetype/lib32 --with-toolchain-version=2010" ./makejdk-any-platform.sh jdk8u
4. And then try another build with a fresh clone: CONFIGURE_ARGS_FOR_ANY_PLATFORM="-with-freemarker-jar=/cygdrive/c/openjdk/freemarker.jar --with-freetype-include=/cygdrive/c/openjdk/freetype/include --with-freetype-lib=/cygdrive/c/openjdk/freetype/lib64 --with-toolchain-version=2013" ./makejdk-any-platform.sh --build-variant openj9 --jdk-boot-dir /cygdrive/c/openjdk/jdk10 jdk11u

I think those commands should be correct and adequate for a decent amount of testing to understand if the playbooks are production ready, but if they don't appear to work let me know.

Any questions give me a shout here or in the #infrastructure channel on the adoptopenjdk slack :-)

[cwesMills]

I don't have a functional AWX server currently. Hopefully I will be able to begin testing toward the end of next week.

[sxa]

That should matter as you can run the playbooks from the ansible-playbook command line

[cwesMills]

The MSVS_2017 role hangs endlessly when run as part of the playbook. I believe this is because even though the /Silent option is used, popups will still appear on the machine that require input (tested manually).

[cwesMills]

@sxa555 I've tried running the command you suggested that builds jdk8, but I get the following error

checking for FREETYPE... yes
configure: Found freetype using pkg-config, but ignoring since we can not bundle that
configure: error: Could not find freetype!
The freetype library can now be build during the configure process.
Download the freetype sources and unpack them into an arbitrary directory:

wget http://download.savannah.gnu.org/releases/freetype/freetype-2.5.3.tar.gz
tar -xzf freetype-2.5.3.tar.gz

Then run configure with '--with-freetype-src=<freetype_src>'. This will
automatically build the freetype library into '<freetype_src>/lib64' for 64-bit
builds or into '<freetype_src>/lib32' for 32-bit builds.
Afterwards you can always use '--with-freetype-include=<freetype_src>/include'
and '--with-freetype-lib=<freetype_src>/lib32|64' for other builds.
configure exiting with result code 1

This was run after the freetype role was updated in #633.

[HusainYusufali]

The most recent application of the playbooks and completing the post playbook steps allows all the builds complete successfully. The OS's tested were Win 10, Server 2012, Server 2016, and Server 2019.

For Server 2012, powershell needs to updated to version 5.1 in order for the playbook to work correctly. Some roles like Java installation hang when using an older powershell version. This MSI will update powershell to 5.1

These are the commands I used for each build:

### _JDK8-32_
git clone https://github.com/AdoptOpenJDK/openjdk-build
cd openjdk-build/build-farm
export TARGET_OS=windows
export ARCHITECTURE=x86-32
export JAVA_TO_BUILD=jdk8u
export VARIANT=openj9
export JDK_BOOT_DIR=/cygdrive/c/openjdk/jdk-8
./make-adopt-build-farm.sh

### _JDK8-64_
git clone https://github.com/AdoptOpenJDK/openjdk-build
cd openjdk-build/build-farm
export TARGET_OS=windows
export ARCHITECTURE=x64
export JAVA_TO_BUILD=jdk8u
export VARIANT=openj9
export JDK_BOOT_DIR=/cygdrive/c/openjdk/jdk-8
./make-adopt-build-farm.sh

### _JDK11-64_
git clone https://github.com/AdoptOpenJDK/openjdk-build
cd openjdk-build/build-farm
export TARGET_OS=windows
export ARCHITECTURE=x64
export JAVA_TO_BUILD=jdk11u
export VARIANT=openj9
export JDK_BOOT_DIR=/cygdrive/c/openjdk/jdk-11
./make-adopt-build-farm.sh

These are the POST PLAYBOOK steps that need to be completed:

Windows Post Ansible Playbook Configuration for OpenJ9 OpenJDK8

The following steps assume that the Windows machine has been already configure using the Windows Ansible Playbook.

Windows Debugging Tools

You can get the Debugging Tools as part of the Windows 7 SDK. Download and install the Microsoft Windows SDK for Windows 7. The default installation options will install the Debugging Tools for Windows (check the Debugging Tools for Windows checkbox on the Installation Options screen, under Common Utilities).

Set Environement Variables

To add a new system variable or update an existing variable,

go to Control Panel\System and Security\System\Advanced system settings, click on the Environment Variables... button and add:

• C:\Program Files\Debugging Tools for Windows (x64)\sdk\inc to INCLUDE
• C:\Program Files\Debugging Tools for Windows (x64)\sdk\lib to LIB

or run in Command Prompt:

setx INCLUDE "C:\Program Files\Debugging Tools for Windows (x64)\sdk\inc";%INCLUDE% /M
setx LIB "C:\Program Files\Debugging Tools for Windows (x64)\sdk\lib";%LIB% /M

Microsoft Visual Studio 2010 SP1

Download and install the Visual Studio 2010 SP1 from the official Microsoft download page.
Download and install the Security Update for Microsoft Visual Studio 2010 Service Pack 1 (KB2565057).

Restart your computer after installations.

[sxa]

Is there no way to install the Debugging SDK and VS2010SP1 automatically?
Also, what is the reason for the jenkins user to be granted shutdown permission?

[HusainYusufali]

Silent mode is not supported for the Debugging SDK and VS2010SP1 requires a Microsoft account to download the installer. The installer also has a warning message that hangs the silent install. The jenkins permission doesn't need to be there. I didn't catch it

[sxa]

Hmmm annoying ref VS2010SP1, even more so if it won't silently install.
As far as the Debugging SDK is concerned I suppose to could still download it to the temp directory during the playbook and leave it there for someone to install.

[karianna]

We have moved almost all of hotspot to VS2017 - can the same be done here?

[jdekonin]

JDK8 Hotspot compile jobs [1] are using 2013, JDK8 OpenJ9 compile jobs [2] are using VS2010. Oracle supported toolchain [3] is VS2010 atm. I am not sure as to the current status on VS2017 for JDK8 compiles but when I attempted a few months ago for use with OpenJ9 it wasn't working.

1. https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-hotspot/254/consoleFull
2. https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-openj9/343/consoleFull
3. https://wiki.openjdk.java.net/display/Build/Supported+Build+Platforms

[sxa]

@HusainYusufali Can the powershell update be automated?

[HusainYusufali]

It can be automated. I will add that to the playbook

[adamfarley]

Some issues running the Windows playbook locally have been documented here: #791

[sxa]

When run multiple times one of the PATH additions fails - this needs to be idempotent:

TASK [cygwin : Add c:\cygwin64\bin to front of %PATH%] *************************
fatal: [40.121.164.56]: FAILED! => {"changed": true, "cmd": "setx /M PATH \"C:\\cygwin64\\bin;%PATH%\"", "delta": "0:00:00.125003", "end": "2019-05-28 03:32:41.392060", "msg": "non-zero return code", "rc": 1, "start": "2019-05-28 03:32:41.267057", "stderr": "ERROR: Invalid syntax. Default option is not allowed more than '2' time(s).\r\nType \"SETX /?\" for usage.\r\n", "stderr_lines": ["ERROR: Invalid syntax. Default option is not allowed more than '2' time(s).", "Type \"SETX /?\" for usage."], "stdout": "", "stdout_lines": []}
	to retry, use: --limit @/root/openjdk-infrastructure/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/main.retry

PLAY RECAP *********************************************************************

[sxa]

This operation appears to be hanging on test-azure-win2012r2-x64-2 :

Get Java 8 Folder name to create symlink

Likewise for the equivalent opersion in java 9, 10 and 11. The code that it is executing is as follows:

  win_shell: powershell ([Environment]::SetEnvironmentVariable("Java8_Folder", (Get-ChildItem -Path 'C:\Program Files\Java\jdk8*' | Select-String -Pattern 'C*'), "Machine"))

Ref: https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/6a1a62d46ec71471d3f1c6dd57ffb81d426db206/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/Java8/tasks/main.yml#L38

[HusainYusufali]

This operation appears to be hanging on test-azure-win2012r2-x64-2 :

Get Java 8 Folder name to create symlink

Likewise for the equivalent opersion in java 9, 10 and 11. The code that it is executing is as follows:

  win_shell: powershell ([Environment]::SetEnvironmentVariable("Java8_Folder", (Get-ChildItem -Path 'C:\Program Files\Java\jdk8*' | Select-String -Pattern 'C*'), "Machine"))

Ref:

https://github.com/AdoptOpenJDK/openjdk-infrastructure/blob/6a1a62d46ec71471d3f1c6dd57ffb81d426db206/ansible/playbooks/AdoptOpenJDK_Windows_Playbook/roles/Java8/tasks/main.yml#L38

This is due to the older version of powershell installed on the machine

[sxa]

On the assumption that the later PowerShell is only required for this operation I have submitted #795

[sxa]

The issue with the SETX of the cygwin PATH is because there is a " mark in the PATH - one of the entries is C:\Program Files\Microft SQL Server\100\DTS\Binn"

[sxa]

Modified the SETX so that quote marks are removed from the PATH before it is processed under #796

[sxa]

Note that the symptoms if you don't have VS2010SP1 installed you run into this issue: eclipse-openj9/openj9#938 with the _xgetvb symbol not being found