Skip to content

WIP - RAC-5956 baremetal updates - DON'T MERGE ME #774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hohene wants to merge 3 commits into from
Closed

WIP - RAC-5956 baremetal updates - DON'T MERGE ME #774

hohene wants to merge 3 commits into from

Conversation

@hohene
Copy link
Contributor

@hohene hohene commented Sep 14, 2017

Added updates into os-install worfklows for ESXi, CentOS and OpenSUSE to help debug install issues we are seeing in the bare-metal regression stack.
The os install workflows fail to 'physical' quanta nodes.

Paired with Manfred on this.

@hohene
Added retries around the callback in post test to address issues we a…
d16b630 
…re hitting when running against quanta bare metal nodes.
@hohene
removed the network option to address issues being hit in bare metal …
c775da5 
…nodes installations.

the network port is not always coming up in time to allow selection of which interface to install kickstart on
@hohene
Adding retries around callback routine to avoid timeouts during insta…
278dd8f 
…ll on bare metal nodes
@hohene
Copy link
Contributor Author

hohene commented Sep 14, 2017

@keedya @manfrednde

@JenkinsRHD
Copy link
Contributor

JenkinsRHD commented Sep 14, 2017

BUILD on-http #527 : FAILURE

@hohene
Copy link
Contributor Author

hohene commented Sep 14, 2017

test this please

@JenkinsRHD
Copy link
Contributor

JenkinsRHD commented Sep 14, 2017

BUILD on-http #528 : FAILURE

BUILD on-http #528 Error Logs ▼Test Name: test03_api20_bootstrap_esxi6 Error Details: ESXi6.0 failed. -------------------- >> begin captured stdout << --------------------- restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41 restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41
restful: Status code = 200

Node info {
"workflow": "Graph.InstallESXi",
"kvm": false,
"nodeID": "59bacffb427ac94601f3a83e",
"version": "6.0",
"StartTime": 1505415470.257062,
"EndTime": 1505417005.615193,
"id": "7ab13025-f065-499c-a748-27e985153b41"
}

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
infra.run: DEBUG: relaying set_test((None,)) to all trackers {}
test.run: DEBUG: handle_after_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7efe06c95bd0> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)
test.run: DEBUG: handle_after_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7efe06cdaf10> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)
root: INFO: +1.03 - STARTING TEST: [test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)]
infra.run: DEBUG: relaying set_test((Test(<pr_gate_os_install.api20_bootstrap_base testMethod=test03_api20_bootstrap_esxi6>),)) to all trackers {}
test.run: DEBUG: handle_start_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7efe06c95bd0> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)
test.run: DEBUG: handle_start_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7efe06cdaf10> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: ERROR: Graph.InstallESXi Workflow failed: {"node":"59bacffb427ac94601f3a83e","status":"failed","context":{"graphId":"7ab13025-f065-499c-a748-27e985153b41","graphName":"Graph.InstallESXi","nodeIp":"172.31.128.4","repoOptions":{"tbootFile":"tboot.b00","moduleFiles":"b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- elxnet.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lpfc.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- misc_cni.v00 --- misc_dri.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_vmxn.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- uhci_usb.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz","mbootFile":"mboot.c32","repo":"http://172.31.128.250:9080/repo/esxi/6.0"},"target":"59bacffb427ac94601f3a83e","users":[{"password":"REDACTED","name":"rackhduser","uid":1010,"plainPassword":"REDACTED","encryptedPassword":"REDACTED"},{"name":"root","password":"REDACTED"}]},"definition":"/api/2.0/workflows/graphs/Graph.InstallESXi","domain":"default","id":"59bad12e499cc37ce24885e3","injectableName":"Graph.InstallESXi","instanceId":"7ab13025-f065-499c-a748-27e985153b41","logContext":{"graphInstance":"7ab13025-f065-499c-a748-27e985153b41","graphName":"Install ESXi","id":"59bacffb427ac94601f3a83e"},"name":"Install ESXi","serviceGraph":"","tasks":[{"label":"installed-callback-notification-wait","instanceId":"05379789-9560-463f-9f2a-05f21e11d6d6","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"failed","terminalOnStates":["succeeded","timeout","cancelled","failed"],"waitingOn":{"a0e5fb71-6832-434a-b314-d3a11cfaa334":"succeeded"}},{"label":"reboot","instanceId":"3d801dbf-727f-4085-8f7e-d014d55725e1","options":{"action":"reboot","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"succeeded","taskStartTime":"2017-09-14T18:58:21.996Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"7ec21fe1-9a5d-458c-87c9-b3db41e591ea":["finished"]}},{"label":"install-os","instanceId":"4f1915fb-5166-4d08-a27b-fbb9ff0d6da5","options":{"osType":"esx","profile":"install-esx.ipxe","installScript":"esx-ks","installScriptUri":"http://172.31.128.250:9080/api/current/templates/esx-ks?nodeId=59bacffb427ac94601f3a83e","rackhdCallbackScript":"esx.rackhdcallback","esxBootConfigTemplate":"esx-boot-cfg","esxBootConfigTemplateUri":"http://172.31.128.250:9080/api/current/templates/esx-boot-cfg?nodeId=59bacffb427ac94601f3a83e","comport":"com1","comportaddress":"0x3f8","gdbPort":"default","logPort":"com1","debugLogToSerial":"1","repo":"http://172.31.128.250:9080/repo/esxi/6.0","hostname":"rackhdnode","rootPassword":"REDACTED","kargs":{},"progressMilestones":{"requestProfile":{"value":1,"description":"Enter ipxe and request OS installation profile"},"enterProfile":{"value":2,"description":"Enter profile, start to download installer"},"startInstaller":{"value":3,"description":"Start installer and prepare installation"},"preConfig":{"value":4,"description":"Enter Pre OS configuration"},"postConfig":{"value":5,"description":"Enter Post OS configuration"},"completed":{"value":6,"description":"Finished OS installation"}},"version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"taskTimeout":3600000},"runJob":"Job.Os.Install","state":"succeeded","taskStartTime":"2017-09-14T18:58:23.942Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"3d801dbf-727f-4085-8f7e-d014d55725e1":"succeeded"}},{"label":"set-boot-pxe","instanceId":"7ec21fe1-9a5d-458c-87c9-b3db41e591ea","error":"Error: Command failed: /usr/bin/ipmitool -I lanplus -U rackhd -P 666c1a37 -H 172.31.128.2 raw 0x00 0x08 0x03 0x08\nUnable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8 rsp=0xcc): Invalid data field in request\n\n at ChildProcess.exithandler (child_process.js:200:12)\n at emitTwo (events.js:92:20)\n at ChildProcess.emit (events.js:172:7)\n at maybeClose (internal/child_process.js:862:16)\n at Process.ChildProcess.handle.onexit (internal/child_process.js:222:5)","options":{"action":"setBootPxe","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"failed","taskStartTime":"2017-09-14T18:57:50.340Z","terminalOnStates":[],"waitingOn":{"aa9968d8-e664-4c91-b144-7b41c17d73c0":"succeeded"}},{"label":"firstboot-callback-notification-wait","instanceId":"a0e5fb71-6832-434a-b314-d3a11cfaa334","error":"TaskTimeoutError: Task did not complete within 1200000ms\n at null. (/RackHD/on-taskgraph/node_modules/on-tasks/lib/task.js:366:29)\n at Timer.listOnTimeout (timers.js:92:15)","options":{"taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"timeout","taskStartTime":"2017-09-14T19:03:25.171Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"4f1915fb-5166-4d08-a27b-fbb9ff0d6da5":"succeeded"}},{"label":"analyze-repo","instanceId":"aa9968d8-e664-4c91-b144-7b41c17d73c0","options":{"osName":"ESXi","repo":"http://172.31.128.250:9080/repo/esxi/6.0","version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","taskTimeout":86400000},"runJob":"Job.Os.Analyze.Repo","state":"succeeded","taskStartTime":"2017-09-14T18:57:50.260Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{}}]}
test.run: INFO: Node: {
"workflow": "Graph.InstallESXi",
"kvm": false,
"nodeID": "59bacffb427ac94601f3a83e",
"version": "6.0",
"StartTime": 1505415470.257062,
"EndTime": 1505417005.615193,
"id": "7ab13025-f065-499c-a748-27e985153b41"
}
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/home/jenkins/workspace/on-http/RackHD/test/tests/bootstrap/pr_gate_os_install.py", line 173, in test03_api20_bootstrap_esxi6
self.assertTrue(wf_status, "ESXi6.0 failed.")
File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
raise self.failureException(msg)
'ESXi6.0 failed.\n-------------------- >> begin captured stdout << ---------------------\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/7ab13025-f065-499c-a748-27e985153b41\nrestful: Status code = 200 \n\nNode info {\n "workflow": "Graph.InstallESXi", \n "kvm": false, \n "nodeID": "59bacffb427ac94601f3a83e", \n "version": "6.0", \n "StartTime": 1505415470.257062, \n "EndTime": 1505417005.615193, \n "id": "7ab13025-f065-499c-a748-27e985153b41"\n}\n\n--------------------- >> end captured stdout << ----------------------\n-------------------- >> begin captured logging << --------------------\ninfra.run: DEBUG: relaying set_test((None,)) to all trackers {}\ntest.run: DEBUG: handle_after_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7efe06c95bd0> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: DEBUG: handle_after_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7efe06cdaf10> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)\nroot: INFO: +1.03 - STARTING TEST: [test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)]\ninfra.run: DEBUG: relaying set_test((Test(<pr_gate_os_install.api20_bootstrap_base testMethod=test03_api20_bootstrap_esxi6>),)) to all trackers {}\ntest.run: DEBUG: handle_start_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7efe06c95bd0> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: DEBUG: handle_start_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7efe06cdaf10> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: ERROR: Graph.InstallESXi Workflow failed: {"node":"59bacffb427ac94601f3a83e","status":"failed","context":{"graphId":"7ab13025-f065-499c-a748-27e985153b41","graphName":"Graph.InstallESXi","nodeIp":"172.31.128.4","repoOptions":{"tbootFile":"tboot.b00","moduleFiles":"b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- elxnet.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lpfc.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi.v00 --- lsu_lsi.v01 --- lsu_lsi.v02 --- lsu_lsi.v03 --- lsu_lsi.v04 --- misc_cni.v00 --- misc_dri.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_vmxn.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- uhci_usb.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz","mbootFile":"mboot.c32","repo":"http://172.31.128.250:9080/repo/esxi/6.0"},"target":"59bacffb427ac94601f3a83e","users":[{"password":"REDACTED","name":"rackhduser","uid":1010,"plainPassword":"REDACTED","encryptedPassword":"REDACTED"},{"name":"root","password":"REDACTED"}]},"definition":"/api/2.0/workflows/graphs/Graph.InstallESXi","domain":"default","id":"59bad12e499cc37ce24885e3","injectableName":"Graph.InstallESXi","instanceId":"7ab13025-f065-499c-a748-27e985153b41","logContext":{"graphInstance":"7ab13025-f065-499c-a748-27e985153b41","graphName":"Install ESXi","id":"59bacffb427ac94601f3a83e"},"name":"Install ESXi","serviceGraph":"","tasks":[{"label":"installed-callback-notification-wait","instanceId":"05379789-9560-463f-9f2a-05f21e11d6d6","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"failed","terminalOnStates":["succeeded","timeout","cancelled","failed"],"waitingOn":{"a0e5fb71-6832-434a-b314-d3a11cfaa334":"succeeded"}},{"label":"reboot","instanceId":"3d801dbf-727f-4085-8f7e-d014d55725e1","options":{"action":"reboot","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"succeeded","taskStartTime":"2017-09-14T18:58:21.996Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"7ec21fe1-9a5d-458c-87c9-b3db41e591ea":["finished"]}},{"label":"install-os","instanceId":"4f1915fb-5166-4d08-a27b-fbb9ff0d6da5","options":{"osType":"esx","profile":"install-esx.ipxe","installScript":"esx-ks","installScriptUri":"http://172.31.128.250:9080/api/current/templates/esx-ks?nodeId=59bacffb427ac94601f3a83e","rackhdCallbackScript":"esx.rackhdcallback","esxBootConfigTemplate":"esx-boot-cfg","esxBootConfigTemplateUri":"http://172.31.128.250:9080/api/current/templates/esx-boot-cfg?nodeId=59bacffb427ac94601f3a83e","comport":"com1","comportaddress":"0x3f8","gdbPort":"default","logPort":"com1","debugLogToSerial":"1","repo":"http://172.31.128.250:9080/repo/esxi/6.0","hostname":"rackhdnode","rootPassword":"REDACTED","kargs":{},"progressMilestones":{"requestProfile":{"value":1,"description":"Enter ipxe and request OS installation profile"},"enterProfile":{"value":2,"description":"Enter profile, start to download installer"},"startInstaller":{"value":3,"description":"Start installer and prepare installation"},"preConfig":{"value":4,"description":"Enter Pre OS configuration"},"postConfig":{"value":5,"description":"Enter Post OS configuration"},"completed":{"value":6,"description":"Finished OS installation"}},"version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"_taskTimeout":3600000},"runJob":"Job.Os.Install","state":"succeeded","taskStartTime":"2017-09-14T18:58:23.942Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"3d801dbf-727f-4085-8f7e-d014d55725e1":"succeeded"}},{"label":"set-boot-pxe","instanceId":"7ec21fe1-9a5d-458c-87c9-b3db41e591ea","error":"Error: Command failed: /usr/bin/ipmitool -I lanplus -U rackhd -P 666c1a37 -H 172.31.128.2 raw 0x00 0x08 0x03 0x08\nUnable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8 rsp=0xcc): Invalid data field in request\n\n at ChildProcess.exithandler (child_process.js:200:12)\n at emitTwo (events.js:92:20)\n at ChildProcess.emit (events.js:172:7)\n at maybeClose (internal/child_process.js:862:16)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:222:5)","options":{"action":"setBootPxe","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"failed","taskStartTime":"2017-09-14T18:57:50.340Z","terminalOnStates":[],"waitingOn":{"aa9968d8-e664-4c91-b144-7b41c17d73c0":"succeeded"}},{"label":"firstboot-callback-notification-wait","instanceId":"a0e5fb71-6832-434a-b314-d3a11cfaa334","error":"TaskTimeoutError: Task did not complete within 1200000ms\n at null. (/RackHD/on-taskgraph/node_modules/on-tasks/lib/task.js:366:29)\n at Timer.listOnTimeout (timers.js:92:15)","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"timeout","taskStartTime":"2017-09-14T19:03:25.171Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"4f1915fb-5166-4d08-a27b-fbb9ff0d6da5":"succeeded"}},{"label":"analyze-repo","instanceId":"aa9968d8-e664-4c91-b144-7b41c17d73c0","options":{"osName":"ESXi","repo":"http://172.31.128.250:9080/repo/esxi/6.0","version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Os.Analyze.Repo","state":"succeeded","taskStartTime":"2017-09-14T18:57:50.260Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{}}]}\ntest.run: INFO: Node: {\n "workflow": "Graph.InstallESXi", \n "kvm": false, \n "nodeID": "59bacffb427ac94601f3a83e", \n "version": "6.0", \n "StartTime": 1505415470.257062, \n "EndTime": 1505417005.615193, \n "id": "7ab13025-f065-499c-a748-27e985153b41"\n}\n--------------------- >> end captured logging << ---------------------'

@hohene
Copy link
Contributor Author

hohene commented Sep 14, 2017

test this please

@JenkinsRHD
Copy link
Contributor

JenkinsRHD commented Sep 14, 2017

BUILD on-http #530 : FAILURE

BUILD on-http #530 Error Logs ▼Test Name: test03_api20_bootstrap_esxi6 Error Details: ESXi6.0 failed. -------------------- >> begin captured stdout << --------------------- restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

restful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee
restful: Status code = 200

Node info {
"workflow": "Graph.InstallESXi",
"kvm": false,
"nodeID": "59baee31cbec8b4601cf2a30",
"version": "6.0",
"StartTime": 1505423198.791515,
"EndTime": 1505424734.164256,
"id": "afedef49-a8bc-4c7d-a0d7-47476b49cfee"
}

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
infra.run: DEBUG: relaying set_test((None,)) to all trackers {}
test.run: DEBUG: handle_after_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7f96f4ae8bd0> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)
test.run: DEBUG: handle_after_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7f96f4b2df10> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)
root: INFO: +1.03 - STARTING TEST: [test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)]
infra.run: DEBUG: relaying set_test((Test(<pr_gate_os_install.api20_bootstrap_base testMethod=test03_api20_bootstrap_esxi6>),)) to all trackers {}
test.run: DEBUG: handle_start_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7f96f4ae8bd0> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)
test.run: DEBUG: handle_start_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7f96f4b2df10> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: INFO_5: Graph.InstallESXi Workflow status: running
test.run: ERROR: Graph.InstallESXi Workflow failed: {"node":"59baee31cbec8b4601cf2a30","status":"failed","context":{"graphId":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","graphName":"Graph.InstallESXi","nodeIp":"172.31.128.5","repoOptions":{"tbootFile":"tboot.b00","moduleFiles":"b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- elxnet.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lpfc.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi_.v00 --- lsu_lsi_.v01 --- lsu_lsi_.v02 --- lsu_lsi_.v03 --- lsu_lsi_.v04 --- misc_cni.v00 --- misc_dri.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_vmxn.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- uhci_usb.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz","mbootFile":"mboot.c32","repo":"http://172.31.128.250:9080/repo/esxi/6.0"},"target":"59baee31cbec8b4601cf2a30","users":[{"password":"REDACTED","name":"rackhduser","uid":1010,"plainPassword":"REDACTED","encryptedPassword":"REDACTED"},{"name":"root","password":"REDACTED"}]},"definition":"/api/2.0/workflows/graphs/Graph.InstallESXi","domain":"default","id":"59baef5e01c9276028941abf","injectableName":"Graph.InstallESXi","instanceId":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","logContext":{"graphInstance":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","graphName":"Install ESXi","id":"59baee31cbec8b4601cf2a30"},"name":"Install ESXi","serviceGraph":"","tasks":[{"label":"analyze-repo","instanceId":"51eb50e8-cfea-43ed-a658-7aa03554e537","options":{"osName":"ESXi","repo":"http://172.31.128.250:9080/repo/esxi/6.0","version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Os.Analyze.Repo","state":"succeeded","taskStartTime":"2017-09-14T21:06:38.796Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{}},{"label":"installed-callback-notification-wait","instanceId":"5267d4b3-40a6-4a5f-af87-9452e5a33670","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"failed","terminalOnStates":["succeeded","timeout","cancelled","failed"],"waitingOn":{"8135d1de-1906-408f-9e29-c55f271d701f":"succeeded"}},{"label":"firstboot-callback-notification-wait","instanceId":"8135d1de-1906-408f-9e29-c55f271d701f","error":"TaskTimeoutError: Task did not complete within 1200000ms\n at null. (/RackHD/on-taskgraph/node_modules/on-tasks/lib/task.js:366:29)\n at Timer.listOnTimeout (timers.js:92:15)","options":{"taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"timeout","taskStartTime":"2017-09-14T21:12:11.429Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"fa5ae937-45c2-4c6a-88b4-1da26f2ff7e4":"succeeded"}},{"label":"set-boot-pxe","instanceId":"90c1aaa8-b2a5-435d-b56b-ce82ea783a00","error":"Error: Command failed: /usr/bin/ipmitool -I lanplus -U rackhd -P 77f2d8c4 -H 172.31.128.2 raw 0x00 0x08 0x03 0x08\nUnable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8 rsp=0xcc): Invalid data field in request\n\n at ChildProcess.exithandler (child_process.js:200:12)\n at emitTwo (events.js:92:20)\n at ChildProcess.emit (events.js:172:7)\n at maybeClose (internal/child_process.js:862:16)\n at Process.ChildProcess.handle.onexit (internal/child_process.js:222:5)","options":{"action":"setBootPxe","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"failed","taskStartTime":"2017-09-14T21:06:38.878Z","terminalOnStates":[],"waitingOn":{"51eb50e8-cfea-43ed-a658-7aa03554e537":"succeeded"}},{"label":"reboot","instanceId":"a8dd205d-ba69-45f6-be07-0afcbb7a5b47","options":{"action":"reboot","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"succeeded","taskStartTime":"2017-09-14T21:07:10.602Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"90c1aaa8-b2a5-435d-b56b-ce82ea783a00":["finished"]}},{"label":"install-os","instanceId":"fa5ae937-45c2-4c6a-88b4-1da26f2ff7e4","options":{"osType":"esx","profile":"install-esx.ipxe","installScript":"esx-ks","installScriptUri":"http://172.31.128.250:9080/api/current/templates/esx-ks?nodeId=59baee31cbec8b4601cf2a30","rackhdCallbackScript":"esx.rackhdcallback","esxBootConfigTemplate":"esx-boot-cfg","esxBootConfigTemplateUri":"http://172.31.128.250:9080/api/current/templates/esx-boot-cfg?nodeId=59baee31cbec8b4601cf2a30","comport":"com1","comportaddress":"0x3f8","gdbPort":"default","logPort":"com1","debugLogToSerial":"1","repo":"http://172.31.128.250:9080/repo/esxi/6.0","hostname":"rackhdnode","rootPassword":"REDACTED","kargs":{},"progressMilestones":{"requestProfile":{"value":1,"description":"Enter ipxe and request OS installation profile"},"enterProfile":{"value":2,"description":"Enter profile, start to download installer"},"startInstaller":{"value":3,"description":"Start installer and prepare installation"},"preConfig":{"value":4,"description":"Enter Pre OS configuration"},"postConfig":{"value":5,"description":"Enter Post OS configuration"},"completed":{"value":6,"description":"Finished OS installation"}},"version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"taskTimeout":3600000},"runJob":"Job.Os.Install","state":"succeeded","taskStartTime":"2017-09-14T21:07:12.570Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"a8dd205d-ba69-45f6-be07-0afcbb7a5b47":"succeeded"}}]}
test.run: INFO: Node: {
"workflow": "Graph.InstallESXi",
"kvm": false,
"nodeID": "59baee31cbec8b4601cf2a30",
"version": "6.0",
"StartTime": 1505423198.791515,
"EndTime": 1505424734.164256,
"id": "afedef49-a8bc-4c7d-a0d7-47476b49cfee"
}
--------------------- >> end captured logging << ---------------------
Stack Trace: File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/home/jenkins/workspace/on-http/RackHD/test/tests/bootstrap/pr_gate_os_install.py", line 173, in test03_api20_bootstrap_esxi6
self.assertTrue(wf_status, "ESXi6.0 failed.")
File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
raise self.failureException(msg)
'ESXi6.0 failed.\n-------------------- >> begin captured stdout << ---------------------\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nrestful: Action = get , URL = http://localhost:9090/api/2.0/workflows/afedef49-a8bc-4c7d-a0d7-47476b49cfee\nrestful: Status code = 200 \n\nNode info {\n "workflow": "Graph.InstallESXi", \n "kvm": false, \n "nodeID": "59baee31cbec8b4601cf2a30", \n "version": "6.0", \n "StartTime": 1505423198.791515, \n "EndTime": 1505424734.164256, \n "id": "afedef49-a8bc-4c7d-a0d7-47476b49cfee"\n}\n\n--------------------- >> end captured stdout << ----------------------\n-------------------- >> begin captured logging << --------------------\ninfra.run: DEBUG: relaying set_test((None,)) to all trackers {}\ntest.run: DEBUG: handle_after_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7f96f4ae8bd0> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: DEBUG: handle_after_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7f96f4b2df10> test=test02_api20_bootstrap_centos65 (pr_gate_os_install.api20_bootstrap_base)\nroot: INFO: +1.03 - STARTING TEST: [test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)]\ninfra.run: DEBUG: relaying set_test((Test(<pr_gate_os_install.api20_bootstrap_base testMethod=test03_api20_bootstrap_esxi6>),)) to all trackers {}\ntest.run: DEBUG: handle_start_test pluggin progression for <stream_sources.amqp_source.AMQPStreamMonitor object at 0x7f96f4ae8bd0> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: DEBUG: handle_start_test pluggin progression for <stream_sources.self_test_source.SelfTestStreamMonitor object at 0x7f96f4b2df10> test=test03_api20_bootstrap_esxi6 (pr_gate_os_install.api20_bootstrap_base)\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: INFO_5: Graph.InstallESXi Workflow status: running\ntest.run: ERROR: Graph.InstallESXi Workflow failed: {"node":"59baee31cbec8b4601cf2a30","status":"failed","context":{"graphId":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","graphName":"Graph.InstallESXi","nodeIp":"172.31.128.5","repoOptions":{"tbootFile":"tboot.b00","moduleFiles":"b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- uc_intel.b00 --- uc_amd.b00 --- sb.v00 --- s.v00 --- mtip32xx.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- elxnet.v00 --- emulex_e.v00 --- weaselin.t00 --- esx_dvfi.v00 --- esx_ui.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lpfc.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- lsu_hp_h.v00 --- lsu_lsi.v00 --- lsu_lsi.v01 --- lsu_lsi.v02 --- lsu_lsi.v03 --- lsu_lsi.v04 --- misc_cni.v00 --- misc_dri.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_vmxn.v00 --- nmlx4_co.v00 --- nmlx4_en.v00 --- nmlx4_rd.v00 --- nvme.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- uhci_usb.v00 --- vsan.v00 --- vsanheal.v00 --- vsanmgmt.v00 --- xhci_xhc.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz","mbootFile":"mboot.c32","repo":"http://172.31.128.250:9080/repo/esxi/6.0"},"target":"59baee31cbec8b4601cf2a30","users":[{"password":"REDACTED","name":"rackhduser","uid":1010,"plainPassword":"REDACTED","encryptedPassword":"REDACTED"},{"name":"root","password":"REDACTED"}]},"definition":"/api/2.0/workflows/graphs/Graph.InstallESXi","domain":"default","id":"59baef5e01c9276028941abf","injectableName":"Graph.InstallESXi","instanceId":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","logContext":{"graphInstance":"afedef49-a8bc-4c7d-a0d7-47476b49cfee","graphName":"Install ESXi","id":"59baee31cbec8b4601cf2a30"},"name":"Install ESXi","serviceGraph":"","tasks":[{"label":"analyze-repo","instanceId":"51eb50e8-cfea-43ed-a658-7aa03554e537","options":{"osName":"ESXi","repo":"http://172.31.128.250:9080/repo/esxi/6.0","version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Os.Analyze.Repo","state":"succeeded","taskStartTime":"2017-09-14T21:06:38.796Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{}},{"label":"installed-callback-notification-wait","instanceId":"5267d4b3-40a6-4a5f-af87-9452e5a33670","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"failed","terminalOnStates":["succeeded","timeout","cancelled","failed"],"waitingOn":{"8135d1de-1906-408f-9e29-c55f271d701f":"succeeded"}},{"label":"firstboot-callback-notification-wait","instanceId":"8135d1de-1906-408f-9e29-c55f271d701f","error":"TaskTimeoutError: Task did not complete within 1200000ms\n at null. (/RackHD/on-taskgraph/node_modules/on-tasks/lib/task.js:366:29)\n at Timer.listOnTimeout (timers.js:92:15)","options":{"_taskTimeout":1200000},"runJob":"Job.Wait.Notification","state":"timeout","taskStartTime":"2017-09-14T21:12:11.429Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"fa5ae937-45c2-4c6a-88b4-1da26f2ff7e4":"succeeded"}},{"label":"set-boot-pxe","instanceId":"90c1aaa8-b2a5-435d-b56b-ce82ea783a00","error":"Error: Command failed: /usr/bin/ipmitool -I lanplus -U rackhd -P 77f2d8c4 -H 172.31.128.2 raw 0x00 0x08 0x03 0x08\nUnable to send RAW command (channel=0x0 netfn=0x0 lun=0x0 cmd=0x8 rsp=0xcc): Invalid data field in request\n\n at ChildProcess.exithandler (child_process.js:200:12)\n at emitTwo (events.js:92:20)\n at ChildProcess.emit (events.js:172:7)\n at maybeClose (internal/child_process.js:862:16)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:222:5)","options":{"action":"setBootPxe","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"failed","taskStartTime":"2017-09-14T21:06:38.878Z","terminalOnStates":[],"waitingOn":{"51eb50e8-cfea-43ed-a658-7aa03554e537":"succeeded"}},{"label":"reboot","instanceId":"a8dd205d-ba69-45f6-be07-0afcbb7a5b47","options":{"action":"reboot","version":"6.0","repo":"http://172.31.128.250:9080/repo/esxi/6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"hostname":"rackhdnode","rootPassword":"REDACTED","_taskTimeout":86400000},"runJob":"Job.Obm.Node","state":"succeeded","taskStartTime":"2017-09-14T21:07:10.602Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"90c1aaa8-b2a5-435d-b56b-ce82ea783a00":["finished"]}},{"label":"install-os","instanceId":"fa5ae937-45c2-4c6a-88b4-1da26f2ff7e4","options":{"osType":"esx","profile":"install-esx.ipxe","installScript":"esx-ks","installScriptUri":"http://172.31.128.250:9080/api/current/templates/esx-ks?nodeId=59baee31cbec8b4601cf2a30","rackhdCallbackScript":"esx.rackhdcallback","esxBootConfigTemplate":"esx-boot-cfg","esxBootConfigTemplateUri":"http://172.31.128.250:9080/api/current/templates/esx-boot-cfg?nodeId=59baee31cbec8b4601cf2a30","comport":"com1","comportaddress":"0x3f8","gdbPort":"default","logPort":"com1","debugLogToSerial":"1","repo":"http://172.31.128.250:9080/repo/esxi/6.0","hostname":"rackhdnode","rootPassword":"REDACTED","kargs":{},"progressMilestones":{"requestProfile":{"value":1,"description":"Enter ipxe and request OS installation profile"},"enterProfile":{"value":2,"description":"Enter profile, start to download installer"},"startInstaller":{"value":3,"description":"Start installer and prepare installation"},"preConfig":{"value":4,"description":"Enter Pre OS configuration"},"postConfig":{"value":5,"description":"Enter Post OS configuration"},"completed":{"value":6,"description":"Finished OS installation"}},"version":"6.0","users":[{"password":"REDACTED","name":"rackhduser","uid":1010}],"_taskTimeout":3600000},"runJob":"Job.Os.Install","state":"succeeded","taskStartTime":"2017-09-14T21:07:12.570Z","terminalOnStates":["timeout","cancelled","failed"],"waitingOn":{"a8dd205d-ba69-45f6-be07-0afcbb7a5b47":"succeeded"}}]}\ntest.run: INFO: Node: {\n "workflow": "Graph.InstallESXi", \n "kvm": false, \n "nodeID": "59baee31cbec8b4601cf2a30", \n "version": "6.0", \n "StartTime": 1505423198.791515, \n "EndTime": 1505424734.164256, \n "id": "afedef49-a8bc-4c7d-a0d7-47476b49cfee"\n}\n--------------------- >> end captured logging << ---------------------'

@hohene hohene changed the title WIP - RAC-5956 baremetal updates WIP - RAC-5956 baremetal updates - DON'T MERGE ME Sep 27, 2017
@keedya
Copy link

keedya commented Oct 13, 2017

The Copyright Check has failed!
The files listed below do not have a valid copyright statement:

  • lib/services/ws-error.js - lib/services/ws-resources.js - lib/services/ws-service.js - package.json - spec/.jshintrc - spec/lib/api/index-spec.js - spec/lib/services/ws-service-spec.js

@keedya
Copy link

keedya commented Oct 14, 2017

The Copyright Check has failed!
The files listed below do not have a valid copyright statement:

  • data/templates/centos-ks - data/templates/esx-ks - data/templates/suse-autoinst.xml

@JenkinsRHD
Copy link
Contributor

JenkinsRHD commented Oct 26, 2017

BUILD on-http #639 : FAILURE

@jimturnquist
Copy link
Contributor

jimturnquist commented Jan 5, 2018

I checked with @hohene. This can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@tannoa2 tannoa2 tannoa2 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

run-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hohene @JenkinsRHD @keedya @jimturnquist @tannoa2