cloud-early-config detects unknown hypervisor type "xen-domU"

[MartinEmrich]

ISSUE TYPE

• Bug Report

COMPONENT NAME

systemVM image 4.11 for XenServer

CLOUDSTACK VERSION

4.11.0 as well as 4.11.1 git from today.
SystemVM: 4.11.0 Sun Jan 14 15:37:28 UTC 2018

CONFIGURATION

XenServer 6.5 SP1

OS / ENVIRONMENT

Management Server on CentOS 7.
SystemVMs running as "Debian 9.3" in HVM mode.

SUMMARY

Newly created system VMs are not configured, cloud.log reports "systemvm type=" (nothing else).

the /opt/cloud/bin/setup/cloud-early-config script tries to detect the hypervisor type, there are cases for "xen-pv" and "xen-hvm". But it uses "virt-what", which returns the unknown type "xen-domU".

After applying these fixes...

• add "xen-domU" to the "xen-hvm" case
• changing the path to xenstore-read from /usr/sbin to /usr/bin

and rebooting, the SystemVM comes up correctly.

[yadvr]

@MartinEmrich the 4.11.0.0 systemvmtemplate cannot be used with 4.11.1.0-SNAPSHOT. There was a commit from @khos2ow that changed the xenserver pv vs hvm detection, this happens in cloud-early-config script that cannot be simply patched. I'll update 4.11.1.0 systemvmtemplate in next few weeks.

@khos2ow can you comment on the /usr/bin, /usr/sbin path change?

[MartinEmrich]

@rhtyd Thanks for the hint, a new systemvm image is building, too, right now (No KVM in VirtualBox, so I removed "accelerator":"kvm" from the packer config and it will probably take ages...

[MartinEmrich]

I built the new systemvm, but I have issues installing it:
During a previous update from 4.9.2.0 to 4.11.0, I installed the new systemvm via UI as "systemvm-xenserver-4.11". This maps to /secondary/templates/tmpl/2/204.
Using cloud-install-sys-tmplt installs it into /secondary/templates/1/204.
Newly created system vms use "systemvm-xenserver-4.11" (still the old one), which cannot be deleted. What's the safe way to replace a current systemvm template with the same version?

[yadvr]

@MartinEmrich You register a template like you normally do, give it a name and change bunch of global settings, such as router.template.xenserver to the new template's name, in the db in the vm_template details set type=SYSTEM. Btw I've the new 4.11 templates here: https://lab.yadav.cloud/systemvmtemplates/4.11/

[rafaelweingartner]

@rhtyd does it work for SSVM and console proxy? I was under the impression that this would only change the template for routers.

There is this PR to enable customizing system VMs template here: #2275

[khos2ow]

@rhtyd this specific change doesn't require a new systemvm template and will be deployed on systemvms using systemvm.iso.

@MartinEmrich the file /opt/cloud/bin/setup/cloud-early-config is the source file and you need to check if it's actually correctly updated in /etc/init.d/cloud-early-config.

I'll have to look to see what's going on with /usr/bin and /usr/sbin.

[khos2ow]

@MartinEmrich afaik you should never use cloud-install-sys-tmplt after the first installation was done, it will mess up ssh key inside systemvm template and result in management server cannot talk to system vms.

@rafaelweingartner yes it does, SSVM, CPVM work as is (with the process @rhtyd mentioned above) and VR gets to use the same new template through router.template.xenserver setting.

[rafaelweingartner]

I am pretty sure it does not.... they get the systemvm.iso injected (if this is the only thing that matter to you), but I think the execution flow is different when it comes on selecting the template.

[khos2ow]

Well, that's how we upgrade our systemvm templates when required.

• add new template from ui
• change its type to SYSTEM (in db)
• update current running SSVM and CPVM to use this new template (in db)
• change router.template.xenserver to use this new template (in db)
• restart mgmt server and destroy VMs to be spun up with new template

[rafaelweingartner]

@khos2ow ah... that might work because you change the type to system in the database. However, that might not be a sure thing. If you take a look into com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(long) and org.apache.cloudstack.secondarystorage.SecondaryStorageManagerImpl.startNew(long, Role). You will see that they do not look into that router parameter.

Those methods simply use the method com.cloud.storage.dao.VMTemplateDaoImpl.findSystemVMReadyTemplate(long, HypervisorType) , which only looks for the first template it find.

That is one of the reason for PR: #2275

[yadvr]

@rafaelweingartner yes, as others has pointed out we use the router.xxx to specify the systemvmtemplate name which gets used for all the systemvms (cpvm, ssvm, routers etc)

[khos2ow]

I've found a rather interesting, potential problem, on this. Hypervisor detection has been changed in bb96564, virt-what executed prior to check from /proc/xen or in my change at 535e615 to read from dmesg.

I just tested executing virt-what on both PV and HVM vms on my environment (Xen 7.1.1 and 7.1.0) and I'm getting conflicting results from virt-what (needs more investigation) but I believe we need to read from /proc/xen and dmesg prior to virt-what.

[MartinEmrich]

@rhtyd Thanks for the updated systemvm templates, I'd like to try them out.

But @khos2ow pointed out that cloud-install-sys-tmplt is not a good idea. But I cannot download it via UI either, as there's no working SSVM now.
So there's a chicken-and-egg problem...

I did the ugly thing, as this is just a test system: disabled all hosts and the zone, deleted all system VMs, removed all references from vm_instance, template_store_ref, template_spool_ref and the entry from vm_template.

I then installed your template via cloud-install-sys-tmplt, it installed to /1/1 this time.

after restarting and reenabling everything, two CPVM/SSVM pairs came up, only one was referenced in the DB. I deleted the orphaned pair from XenCenter.
The remaining pair is Debian 7, ACS 4.6.0 (the original installed during the first ACS setup with 4.9.2.0).

Looks like it was cached somewhere?