XenServer inter-cluster migration failure due to snapshot records

[nvazquez]

Migration of VMs with removed snapshots fail when the DB record of deleted snapshot is marked as 'removed' (after garbage collection)

ISSUE TYPE

• Bug Report

COMPONENT NAME

API

CLOUDSTACK VERSION

4.13.0

CONFIGURATION

Advanced zone
At least 2 clusters
XCP-ng 7.6 hosts

OS / ENVIRONMENT

SUMMARY

Inter-cluster migration fail under certain conditions. It was found that trying to migrate a VM in which one of their volumes snapshots was removed does not fail when the DB record has:

• state = 'Destroyed'
• removed = NULL

However, after garbage collection, the removed filed gets updated and after that, any migration attempt fails with NullPointerException

STEPS TO REPRODUCE

Deploy VM on cluster 1
Create ROOT volume snapshot
Remove volume snapshot
Wait until DB entry gets the 'removed' not null
Migrate VM to another cluster with storage motion 

EXPECTED RESULTS

Migration succeeds

ACTUAL RESULTS

2020-05-05 04:32:17,770 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-157:ctx-0f4faa96) (logid:8c041fe2) Seq 6-1901081992703924074: Executing request
2020-05-05 04:32:17,808 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-157:ctx-0f4faa96) (logid:7bd1f9b7) Seq 6-1901081992703924074: Response Received: 
2020-05-05 04:32:17,808 DEBUG [c.c.a.t.Request] (DirectAgent-157:ctx-0f4faa96) (logid:7bd1f9b7) Seq 6-1901081992703924074: Processing:  { Ans: , MgmtId: 52243264806, via: 6(dc04-xcpcloud-h47.one.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.MigrateWithStorageCompleteAnsw
er":{"volumeTos":[{"path":"a80b5499-f1c9-40bc-80fa-18c1084a37cd","accountId":0,"id":6},{"path":"4306c091-8df9-4589-be24-1ac4d9314ec7","accountId":0,"id":7}],"result":true,"wait":0}}] }
2020-05-05 04:32:17,808 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-ce0203f2 job-241/job-242 ctx-0fb2548d) (logid:7bd1f9b7) Seq 6-1901081992703924074: Received:  { Ans: , MgmtId: 52243264806, via: 6(dc04-xcpcloud-h47.one.local), Ver: v1, Flags: 10, { MigrateWithStorageC
ompleteAnswer } }
2020-05-05 04:32:17,838 DEBUG [o.a.c.s.s.SnapshotObject] (Work-Job-Executor-25:ctx-ce0203f2 job-241/job-242 ctx-0fb2548d) (logid:7bd1f9b7) Failed to update state:java.lang.NullPointerException
2020-05-05 04:32:17,852 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-25:ctx-ce0203f2 job-241/job-242 ctx-0fb2548d) (logid:7bd1f9b7) Failed to process copy volume callback
2020-05-05 04:32:17,859 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-25:ctx-ce0203f2 job-241/job-242 ctx-0fb2548d) (logid:7bd1f9b7) Failed to migrated vm VM[User|i-17-5-VM] along with its volumes. java.lang.NullPointerException
2020-05-05 04:32:17,859 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-25:ctx-ce0203f2 job-241/job-242 ctx-0fb2548d) (logid:7bd1f9b7) Migration was unsuccessful.  Cleaning up: VM[User|i-17-5-VM]