WIP initial auto jenkins provisioning #192

gdams · 2018-02-09T00:11:07Z

This probably shouldn't be run until we have standardised our hostnames as it could create a load of rogue jenkins nodes that we probably wouldn't want

gdams · 2018-02-09T00:33:15Z

FYI https://ci.adoptopenjdk.net/computer/build-cloudcone-ubuntu1604-x64-1/ is an example of a node that configured using this playbook:

labels: linux ubuntu1604 x64 cloudcone x86_64 build ubuntu ubuntu16

TODO:

Add JDK7_BOOT_DIR, JDK8_BOOT_DIR, JDK9_BOOT_DIR.

I feel that this can be merged without the JAVA dirs for now and they can be manually added, I will work on automating this at a later date

bblondin · 2018-02-09T13:23:43Z

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Jenkins_Master/tasks/main.yml

+- name: SSH into the Jenkins and excute the jenkins-create-node.sh script
+  become: false
+  # TODO fetch a var instead of hard coding linux
+  local_action: command ssh -o StrictHostKeyChecking=no jenkins@ci.adoptopenjdk.net "/home/jenkins/.jenkins/jenkins-create-node.sh {{ inventory_hostname }} {{ ansible_host }} /home/jenkins linux,{{os}},{{arch}},{{provider}},{{ansible_architecture}},{{type}},{{ansible_distribution|lower}},{{ansible_distribution|lower}}{{ansible_distribution_major_version}}"


These should be vars

jenkins@ci.adoptopenjdk.net --> {{ Jenkins_Username }}@{{ Jenkins_Server }} /home/jenkins (x2) --> /home/{{ Jenkins_Username }}

bblondin · 2018-02-09T13:24:06Z

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Jenkins_Master/tasks/main.yml

+---
+- name: Check if machine exists in jenkins
+  uri:
+    url: "https://ci.adoptopenjdk.net/computer/{{ ansible_hostname }}/api/json"


This should be a var

url: "https://ci.adoptopenjdk.net/computer/{{ ansible_hostname }}/api/json" --> "https://{{ Jenkins_Server }}/computer/{{ ansible_hostname }}/api/json"

bblondin · 2018-02-09T13:25:02Z

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Jenkins_Master/tasks/main.yml

@@ -0,0 +1,15 @@
+---


################## # Jenkins_Master # ##################

bblondin · 2018-02-09T13:27:03Z

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Nagios_Master_Config/tasks/main.yml

@@ -17,7 +17,7 @@
 # Script must exist on Nagios Master
 #
 - name: SSH into the Nagios Master and excute the Nagios_Ansible_Config_tool.sh script
-  local_action: command ssh -o StrictHostKeyChecking=no root@{{ Nagios_Master_IP }} "/usr/local/nagios/Nagios_Ansible_Config_tool/Nagios_Ansible_Config_tool.sh  {{ ansible_distribution }} {{ ansible_architecture }} {{ inventory_hostname }} {{ ansible_default_ipv4.address }} "
+  local_action: command ssh -o StrictHostKeyChecking=no root@{{ Nagios_Master_IP }} "/usr/local/nagios/Nagios_Ansible_Config_tool/Nagios_Ansible_Config_tool.sh  {{ ansible_distribution }} {{ ansible_architecture }} {{ inventory_hostname }} {{ ansible_host }} "


Nagios will require the IP address of the client machine
Which out it, Nagios will not be able to find the client machine by unresolvable hostnames
abc123.adoptopenjdk.net

ansible_host is the IP address that is set in the yml inventory :)

bblondin · 2018-02-09T13:28:54Z

This is great work, fully automating the Jeknins end of things

bblondin

Looks good to go

karianna · 2018-02-12T09:34:31Z

@gdams Have the hostnames been standardised so that this PR is safe to merge?

gdams · 2018-02-12T10:59:15Z

I'm going to leave this open until I've worked out how to set the java paths as vars

…e into jenkins_auto

gdams · 2018-02-12T23:38:56Z

I've now added java paths for Ubuntu ONLY.

gdams · 2018-03-03T20:45:14Z

I'm going to bring @smlambert into this conversation because I want to make sure that the testing team is happy with the labels that are generated by this PR. I know that there was an issue that Shelley raised about standardising them but I'm not sure if this schema will work.

An example test machine would have the following labels:
linux ubuntu1604 x64 cloudcone x86_64 test ubuntu ubuntu16

smlambert · 2018-03-04T15:38:09Z

Yes, we have started using some of the labels discussed in #93, particularly our Jenkins scripts look for sw.os.xxx, and hw.arch.xxx. Thanks @gdams for noticing that I had added those labels manually to the previous 2 x86-64_linux machines we have for test.

The reason I need/want the standard naming is so that downstream CI servers and these AdoptOpenJDK servers (ci.adopt and ci-jck.adopt) align in a labelling strategy, and we can directly reuse the Jenkins collateral from AdoptOpenJDK. The more alignment we have, the more people we can bring to the AdoptOpenJDK project, as the work here is then directly reusable.

gdams · 2018-03-04T18:14:46Z

thanks for that @smlambert. Is the above schema that I mentioned going to be accpetable for you or am I going to need to change it? Basically I am going to either have to modify all of the test jobs or all of the build jobs

smlambert · 2018-03-04T18:33:24Z

I think we should get final agreement on #93 and have all machines following the agreed upon schema, I can help update what ever build jobs would need it.

I also am suggesting we rethink how we are using the machines, and whether we should consider having them configured for dual purpose, see #199 (comment)

If the requirement is as @karianna says "each type of build does need to run on an identical host for consistency and auditability", then build machines can be statically allocated, but to avoid them sitting idle, all but one of each platform of build machines could be labelled as ci.role.test and ci.role.build, so that when they are idle, they go into the pool of machines available to test jobs (keeping 1 as exclusively ci.role.build, is the cheap way of ensuring builds never wait for tests to finish, though there are also Jenkins plugins that will prioritize certain jobs over others, that could be combined for a hybrid usage story).

karianna · 2018-03-05T10:39:26Z

I think we can loosen the requirement of identical host as long as the libraries (native or otherwise) that we build with are identical. If we deviate from that, then we need to make that clear to the end user (again that might be acceptable.

jdekonin · 2018-03-22T22:50:47Z

Can we please have this as a role that is enabled via option and not on as a default? There are users that would like to consume these scripts and not have to add to a jenkins server.

karianna · 2018-06-05T15:12:25Z

@gdams there's conflicts.

gdams · 2018-06-06T09:47:40Z

I think that this was being closed in favour of https://github.com/AdoptOpenJDK/openjdk-jenkins-helper but I'll let someone confirm that before closing

sxa · 2018-09-03T16:20:16Z

Hmmm hadn't seen that repo (catching up on old PRs just now) @karianna @smlambert Can we do something with this, possibly in light of George's comment from about 3 months ago?

smlambert · 2018-09-03T18:10:50Z

I think it is worth having an infra call about this, so we can discuss requirements & options. There are varying approaches and opinions, and It will be good to converge.

karianna · 2018-09-03T20:44:11Z

@smlambert Yep - I'm meeting George and John online tomorrow at 1900 UK time if maybe after that suits?

sxa · 2020-01-07T19:33:02Z

@karianna What was the outcome of that meeting in September 2018? ;-)

karianna · 2020-02-18T00:22:13Z

@karianna What was the outcome of that meeting in September 2018? ;-)

Hmmm, yes well. Let's have a new call between @gdams and @smlambert on this one.

@gdams - If it's trivial you might as well solve the merge conflict and address @jdekonin's comment as well.

gdams requested a review from bblondin February 9, 2018 00:11

gdams changed the title ~~WIP: initial auto jenkins provisioning~~ initial auto jenkins provisioning Feb 9, 2018

gdams added the enhancement label Feb 9, 2018

gdams force-pushed the jenkins_auto branch 2 times, most recently from be81c6f to df6316e Compare February 9, 2018 03:51

bblondin reviewed Feb 9, 2018

View reviewed changes

gdams force-pushed the jenkins_auto branch from df6316e to 07b2bcb Compare February 9, 2018 13:26

bblondin reviewed Feb 9, 2018

View reviewed changes

gdams force-pushed the jenkins_auto branch from 07b2bcb to 0b36209 Compare February 9, 2018 13:27

bblondin approved these changes Feb 9, 2018

View reviewed changes

gdams force-pushed the jenkins_auto branch 2 times, most recently from 6a1550a to 5939bfb Compare February 9, 2018 20:32

initial auto jenkins provisioning

4cbe0a6

gdams force-pushed the jenkins_auto branch from 5939bfb to 4cbe0a6 Compare February 9, 2018 20:36

George Adams added 4 commits February 9, 2018 20:38

Merge branch 'master' into jenkins_auto

c6b7c34

initial auto jenkins provisioning

adb7335

fixup

1e42f24

fixup 2

fffe45d

George Adams added 3 commits February 12, 2018 00:59

initial auto jenkins provisioning

b9f9254

initial auto jenkins provisioning

9764790

fixup

5000e5f

gdams changed the title ~~initial auto jenkins provisioning~~ WIP initial auto jenkins provisioning Feb 12, 2018

gdams added the DO NOT MERGE label Feb 12, 2018

George Adams added 3 commits February 12, 2018 22:25

Merge branch 'jenkins_auto' of github.com:gdams/openjdk-infrastructur…

27bfedf

…e into jenkins_auto

add java paths for ubuntu

5cda446

Merge branch 'master' into jenkins_auto

9a76ba6

gdams added the ansible label Feb 25, 2018

gdams mentioned this pull request Mar 5, 2018

Add Jenkins shared library #259

Closed

sxa marked this pull request as draft June 1, 2020 19:30

gdams closed this Aug 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP initial auto jenkins provisioning #192

WIP initial auto jenkins provisioning #192

gdams commented Feb 9, 2018 •

edited

Loading

gdams commented Feb 9, 2018 •

edited

Loading

bblondin Feb 9, 2018

bblondin Feb 9, 2018

bblondin Feb 9, 2018

bblondin Feb 9, 2018

gdams Feb 9, 2018

bblondin commented Feb 9, 2018

bblondin left a comment

karianna commented Feb 12, 2018

gdams commented Feb 12, 2018

gdams commented Feb 12, 2018

gdams commented Mar 3, 2018

smlambert commented Mar 4, 2018

gdams commented Mar 4, 2018

smlambert commented Mar 4, 2018

karianna commented Mar 5, 2018

jdekonin commented Mar 22, 2018

karianna commented Jun 5, 2018

gdams commented Jun 6, 2018

sxa commented Sep 3, 2018

smlambert commented Sep 3, 2018

karianna commented Sep 3, 2018

sxa commented Jan 7, 2020

karianna commented Feb 18, 2020

WIP initial auto jenkins provisioning #192

WIP initial auto jenkins provisioning #192

Conversation

gdams commented Feb 9, 2018 • edited Loading

gdams commented Feb 9, 2018 • edited Loading

bblondin Feb 9, 2018

Choose a reason for hiding this comment

bblondin Feb 9, 2018

Choose a reason for hiding this comment

bblondin Feb 9, 2018

Choose a reason for hiding this comment

bblondin Feb 9, 2018

Choose a reason for hiding this comment

gdams Feb 9, 2018

Choose a reason for hiding this comment

bblondin commented Feb 9, 2018

bblondin left a comment

Choose a reason for hiding this comment

karianna commented Feb 12, 2018

gdams commented Feb 12, 2018

gdams commented Feb 12, 2018

gdams commented Mar 3, 2018

smlambert commented Mar 4, 2018

gdams commented Mar 4, 2018

smlambert commented Mar 4, 2018

karianna commented Mar 5, 2018

jdekonin commented Mar 22, 2018

karianna commented Jun 5, 2018

gdams commented Jun 6, 2018

sxa commented Sep 3, 2018

smlambert commented Sep 3, 2018

karianna commented Sep 3, 2018

sxa commented Jan 7, 2020

karianna commented Feb 18, 2020

gdams commented Feb 9, 2018 •

edited

Loading

gdams commented Feb 9, 2018 •

edited

Loading