Repository navigation
Reprovision the linuxone s390x machines #2080
Description
Activity
All five machine have been ansibled and a build+test passes.
Next steps:
- Add machines to Jenkins
- Test builds pass on the release and test machines
- Begin to switch over the jobs to these new machines
(will need new jobs I think) - Deprovision the old s390x machines as they will be turned off by the end of the year
- Begin to switch over the jobs to these new machines (will need new jobs I think)
We shouldn't need new jobs.
Note: need to change the /data reference to:
/home/iojs/git/io.js.reference
in the main job when I switch the labels.
- 14.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/6/ PASS
- 13.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/7/ PASS
- 12.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/8/ FAIL
Failed:
10:03:04 Started by upstream project "node-test-commit-linuxone-sam-github" build number 8 10:03:04 originally caused by: 10:03:04 Started by user Sam Roberts 10:03:04 Running as SYSTEM 10:03:04 [EnvInject] - Loading node environment variables. 10:03:04 Building remotely on test-ibm-rhel7-s390x-4 (7.7 s390x-RedHatEnterprise RedHatEnterprise-7.7 rhel7-s390x s390x-RedHatEnterprise-7.7 RedHatEnterprise s390x) in workspace /home/iojs/build/workspace/node-test-commit-linuxone-sam-github/nodes/rhel7-s390x 10:03:04 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 148.100.86.94/148.100.86.94:38882 10:03:04 java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy11 10:03:04 Caused: java.io.IOException: Remote call on JNLP4-connect connection from 148.100.86.94/148.100.86.94:38882 failedTried again, I think https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-4 is broken, taking it offline.
- 12.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/10/ PASS
- 14.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/11/ PASS
- 8.x: https://ci.nodejs.org/view/All/job/node-test-commit-linuxone-sam-github/12/ PASS?
not ok 2109 sequential/test-timers-throw-reschedule # TODO : Fix flaky test
ci-release:
If builds are PASS on all the machines, I will change the build label in node-test-commit-linuxone from
rhel72-s390xtorhel7-s390x. Once that's green for a while, we'll remove the old machines.If builds are PASS on all the machines, I will change the build label in node-test-commit-linuxone from
rhel72-s390xtorhel7-s390x. Once that's green for a while, we'll remove the old machines.We will need to update the other jobs that build on Linux One (e.g. CITGM, libuv, V8, node-addons-api) before removing the old machines. I can start updating some of those in a few hours time.
Thanks for the reminder, and that'd be great, thanks.
Switch node-test-commit-linuxone to the new machines:
- https://ci.nodejs.org/job/node-test-commit-v8-linux/
I added av8testlabel to https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-1/ otherwise the build wouldn't schedule onto any nodes.
Test build: https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=rhel7-s390x,v8test=v8test/2713/ (❌ appears to have socket time outs)
I've added back the old machine label so the job currently has both new and old machines for now.
- https://ci.nodejs.org/job/node-test-commit-v8-linux/
41 remaining items
Took the old rhel s390x machine offline in ci-release.
The last step I need someone from @nodejs/releasers to check: is the release ssh key setup correctly?
Please try:
ssh release-ibm-rhel7-s390x-1I'm not on releasers, so I don't have the private key, so I can't confirm. I can ssh in use the IBM mgmt key, so I copied the authorized ssh key from the AIX machines.
I think that's all the release specific setup needed.
Releasers don't ssh into release machines. I'm not aware of having access to any of the existing ones
I looked more closely, it looks like the only people with ssh access to all the release machines are:
% ls .gpg 'bugs@bergstroem.nu' 'reis@janeasystems.com' 'michael_dawson@ca.ibm.com' 'rod@vagg.org'@jbergstroem @joaocgreis @rvagg @mhdawson --- can one of you confirm that
ssh release-ibm-rhel7-s390x-1works for you with thenodejs_build_releaseprivate key?- added a commit that references this issue
on Dec 31, 2019 @sam-github it does not work for me (other release servers work though).
@joaocgreis do you think you could fix that? I'm not a member of the releasers team, so I don't have access to the secrets, so I'm not in a great position to put those secrets on the machine. If my copy of the .ssh authorized keys across didn't work, I'm not sure what else I can do.
@joaocgreis contacted me offline, and helped get the correct releasers key onto the release box, so any with release infrastructure access should be able to ssh in.
Reacted by João Reis@sam-github ssh-d into the the release rhel7 with the nodejs_build_release key.
https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-1/ -- does anybody know where the labels come from?
rhel7-s390xis explicitly configured in https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-1/configure, and it is used, all the others are unused (they have no jobs that use them). I was going to clean up the unused lables, but I don't see how. Its messy and confusing, but it doesn't cause any operational problems.Nothing left to do, the label cleanup is unrelated tidying.
https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-1/ -- does anybody know where the labels come from?
rhel7-s390xis explicitly configured in https://ci.nodejs.org/computer/test-ibm-rhel7-s390x-1/configure, and it is used, all the others are unused (they have no jobs that use them). I was going to clean up the unused lables, but I don't see how. Its messy and confusing, but it doesn't cause any operational problems.I think they're coming from this plugin: https://github.com/jenkinsci/platformlabeler-plugin
@richardlau OK, that makes sense. I'll assume we are using that.
@richardlau OK, that makes sense. I'll assume we are using that.
We are. When I first posted I was searching through the list of plugins installed in our Jenkins for the word
labeland picked out the most plausible sounding one (after reading the description on the linked page). I've subsequently found the configuration page where it's enabled and posted the screenshot as an edit to my previous comment.
The current linuxone machines
*-rhel72-s390x-*will be switched off in the new year and we are being migrated to a new data (EDIT(sam): "new datacentre's") machinesWe have been given access to the five replacement machines and work is being undertaken to ansible them and set them up for ci