Issue while migrating window or ubuntu VM from VMware vcenter

[verendrakalsi]

followed all the required steps mentioned in this document regarding the ​prerequisites for migrating VMs from "Import-Export Insatnce"section in cloudstackUI.

Also since it was windows VM we had also install following tools on the KVM host.

wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.noarch.rpm

install “alien” which can convert rpms to debs

apt -y install alien

the conversion, can take a while

alien -d virtio-win.noarch.rpm

install the resulting deb

dpkg -i virtio-win*.deb

In addition to this, we need to install the below package as well to avoid the error “virt-v2v: error: One of rhsrvany.exe or pvvxsvc.exe is missing in /usr/share/virt-tools“.

wget -nd -O srvany.rpm https://kojipkgs.fedoraproject.org//packages/mingw-srvany/1.1/4.fc38/noarch/mingw32-srvany-1.1-4.fc38.noarch.rpm

alien -d srvany.rpm
dpkg -i srvany.deb

When installing virt-v2v on Ubuntu KVM hosts, the “nbdkit” package (which is required to convert VMware Instances) is not installed by default. To install it, execute the following command:

apt install virt-v2v
apt install nbdkit

The OVF tool (ovftool) must be installed on the KVM hosts if the hosts should export VM files (OVF) from vCenter. If not, the management server exports them using internal java lib code.

Download the ovftool from https://developer.broadcom.com/tools/open-virtualization-format-ovf-tool/latest

unzip VMware-ovftool-4.6.3-24031167-lin.x86_64.zip -d /usr/local/

#create a soft link

ln -s /usr/local/ovftool/ovftool /usr/local/bin/ovftool

Error

However we keep getting this error now.

These are the agent.log outputs

Windows VM.

2025-03-18 16:36:23,028 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:1c3c47aa) Attempting to convert the instance Win-Sr-2022-Std-CloudStack from VMware to KVM
2025-03-18 16:36:23,409 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-5:[]) (logid:) (export ovf) Opening VI source: vi://t989190@10.63.8.5:443/25%20York/vm/Win-Sr-2022-Std-CloudStack
2025-03-18 16:36:23,668 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-5:[]) (logid:) (export ovf) Error: Connection error: Failed to connect to host esx42.pp.emr.local due to Couldn't resolve host name, please see the logs for more details
2025-03-18 16:36:23,676 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-5:[]) (logid:) (export ovf) Completed with errors
2025-03-18 16:36:23,680 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:1c3c47aa) Execution of process [167579] for command [ovftool --noSSLVerify vi://t989190:D0ntW0rryBH%40ppy%4024@10.63.8.5/25 York/vm/Win-Sr-2022-Std-CloudStack /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/33b17ec1-ccd7-4f15-89a9-bcbe6ee662db/ ] failed.
2025-03-18 16:36:23,680 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:1c3c47aa) Exception [Stream closed] occurred when attempting to run command [ovftool --noSSLVerify vi://t989190:D0ntW0rryBH%40ppy%4024@10.63.8.5/25 York/vm/Win-Sr-2022-Std-CloudStack /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/33b17ec1-ccd7-4f15-89a9-bcbe6ee662db/ ]. java.io.IOException: Stream closed
at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
at com.cloud.utils.script.Script.execute(Script.java:314)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.exportOVAFromVMOnVcenter(LibvirtConvertInstanceCommandWrapper.java:391)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:117)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:61)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

Ubuntu VM

025-03-18 16:18:44,042 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:[]) (logid:81a64e31) Attempting to convert the instance ubuntutest from VMware to KVM
2025-03-18 16:18:44,042 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:[]) (logid:81a64e31) Attempting to convert the OVF 5ea08e49-81b5-4173-80fa-464305af069f of the instance ubuntutest from VMware to KVM
2025-03-18 16:44:42,542 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:4d52eaab) Attempting to convert the instance ubuntutest from VMware to KVM
2025-03-18 16:44:42,912 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-6:[]) (logid:) (export ovf) Opening VI source: vi://t989190@10.63.8.5:443/25%20York/vm/ubuntutest
2025-03-18 16:44:42,918 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-6:[]) (logid:) (export ovf) Error: Locator does not refer to an object: vi://t989190@10.63.8.5:443/25 York/vm/ubuntutest
2025-03-18 16:44:42,922 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:4d52eaab) Execution of process [168049] for command [ovftool --noSSLVerify vi://t989190:D0ntW0rryBH%40ppy%4024@10.63.8.5/25 York/vm/ubuntutest /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/bf09f7f8-be17-4efa-bf40-c0fd2ef33788/ ] failed.
2025-03-18 16:44:42,922 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:4d52eaab) Exception [Stream closed] occurred when attempting to run command [ovftool --noSSLVerify vi://t989190:D0ntW0rryBH%40ppy%4024@10.63.8.5/25 York/vm/ubuntutest /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/bf09f7f8-be17-4efa-bf40-c0fd2ef33788/ ]. java.io.IOException: Stream closed
2025-03-18 16:44:42,923 ERROR [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:4d52eaab) Export OVA for the VM ubuntutest failed

[nvazquez]

Hi @verendrakalsi thanks for the detailed description. From your logs on the KVM host it seems that the root cause is the connectivity between the conversion KVM host and the vCenter and ESXi hosts.

2025-03-18 16:36:23,668 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-5:[]) (logid:) (export ovf) Error: Connection error: Failed to connect to host esx42.pp.emr.local due to Couldn't resolve host name, please see the logs for more details

Can you check the connectivity between them?

[verendrakalsi]

yes we have configured the communication between both the network and now we had moved forwarded but now whichever guest OS VM we try to migrate we get this permission denied error. And all the mount points are with "nobody nogroup" permission

2025-03-28 17:25:07,687 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:ca57fdc5) Cleaning up temporary disk 5cb56dc9-6baf-4bea-af93-94055cfe1e62-sda after conversion from temporary location
2025-03-28 17:25:07,688 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:ca57fdc5) Attempting to remove volume 5cb56dc9-6baf-4bea-af93-94055cfe1e62-sda from pool c4b7aab4-b56d-349c-89d9-8c612bb0731d
2025-03-28 17:25:07,697 ERROR [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:ca57fdc5) Error converting instance ubuntu2204-main-cloudstack from VMware, due to: org.libvirt.LibvirtException: cannot unlink file '/mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/5cb56dc9-6baf-4bea-af93-94055cfe1e62-sda': Permission denied com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: cannot unlink file '/mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/5cb56dc9-6baf-4bea-af93-94055cfe1e62-sda': Permission denied

[verendrakalsi]

Today, again i tried and got this error and the migration failed
2025-04-01 15:57:38,042 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Trying to fetch storage pool 67c1ba98-ecd6-3776-977d-97b72618122b from libvirt
2025-04-01 15:57:38,070 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Attempting to create volume cf275a99-a2d1-475a-b535-38bbcaa7d652 (NetworkFilesystem) in pool 67c1ba98-ecd6-3776-977d-97b72618122b with size (51.00 GB) 54760833024
2025-04-01 15:58:34,691 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:a5187f7a) Cleaning up temporary disk b60aba0b-2e14-47cc-b11f-176139fb7fbe-sda after conversion from temporary location
2025-04-01 15:58:34,692 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Attempting to remove volume b60aba0b-2e14-47cc-b11f-176139fb7fbe-sda from pool c4b7aab4-b56d-349c-89d9-8c612bb0731d
2025-04-01 15:58:34,701 ERROR [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-5:[]) (logid:a5187f7a) Error converting instance ubuntu2204-main-cloudstack from VMware, due to: org.libvirt.LibvirtException: cannot unlink file '/mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/b60aba0b-2e14-47cc-b11f-176139fb7fbe-sda': Permission denied com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: cannot unlink file '/mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d/b60aba0b-2e14-47cc-b11f-176139fb7fbe-sda': Permission denied
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.deletePhysicalDisk(LibvirtStorageAdaptor.java:1143)
at com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.deletePhysicalDisk(LibvirtStoragePool.java:181)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.cleanupDisksAndDomainFromTemporaryLocation(LibvirtConvertInstanceCommandWrapper.java:255)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:152)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:61)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-04-01 15:58:34,865 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Attempting to remove storage pool c4b7aab4-b56d-349c-89d9-8c612bb0731d from libvirt
2025-04-01 15:58:34,885 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Storage pool c4b7aab4-b56d-349c-89d9-8c612bb0731d has no corresponding secret. Not removing any secret.
2025-04-01 15:58:34,897 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) deleteStoragePool removed pool from libvirt, but libvirt had trouble unmounting the pool. Trying umount location /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d again in a few seconds
2025-04-01 15:58:39,907 WARN [utils.script.Script] (agentRequest-Handler-5:[]) (logid:a5187f7a) Execution of process [10952] for command [/bin/bash -c sleep 5 && umount /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d ] failed.
2025-04-01 15:58:39,907 WARN [utils.script.Script] (agentRequest-Handler-5:[]) (logid:a5187f7a) Process [10952] for command [/bin/bash -c sleep 5 && umount /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d ] encountered the error: [umount.nfs4: /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d: device is busy].
2025-04-01 15:58:39,907 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Succeeded in unmounting /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d
2025-04-01 15:58:39,916 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:[]) (logid:a5187f7a) Failed to destroy libvirt pool c4b7aab4-b56d-349c-89d9-8c612bb0731d: org.libvirt.LibvirtException: internal error: Child process (/usr/bin/umount /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d) unexpected exit status 16: umount.nfs4: /mnt/c4b7aab4-b56d-349c-89d9-8c612bb0731d: device is busy