LINSTOR create template from snapshot not working

[rp-]

problem

Running the Linstor tests against the 4.21.0.0-rc3, I noticed that create template from snapshot is not working anymore.

Here is the log from the error:

2025-08-14 16:08:38,919 DEBUG [c.c.a.ApiServlet] (qtp1438988851-26:[ctx-9798b557]) (logid:f819b910) ===START===  192.168.125.1 -- GET  jobId=36685271-97a9-4b16-882f-bff865b5999d&command=queryAsyncJobResult&response=json&
2025-08-14 16:08:38,919 DEBUG [c.c.a.ApiServlet] (qtp1438988851-26:[ctx-9798b557]) (logid:f819b910) Two factor authentication is already verified for the user 2, so skipping
2025-08-14 16:08:38,921 DEBUG [c.c.a.ApiServer] (qtp1438988851-26:[ctx-9798b557, ctx-8c8c58ff]) (logid:f819b910) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"073aa3df-7923-11f0-9598-525400000078"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-08-14 16:08:38,922 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1438988851-26:[ctx-9798b557, ctx-8c8c58ff]) (logid:f819b910) Account for user id 073ce01a-7923-11f0-9598-525400000078 is Root Admin or Domain Admin, all APIs are allowed.
2025-08-14 16:08:38,922 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1438988851-26:[ctx-9798b557, ctx-8c8c58ff]) (logid:f819b910) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-08-14 16:08:38,922 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1438988851-26:[ctx-9798b557, ctx-8c8c58ff]) (logid:f819b910) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-08-14 16:08:38,932 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Found [0] snapshots [[]] that have checkpoints for volume with id [3].
2025-08-14 16:08:38,936 DEBUG [c.c.a.ApiServlet] (qtp1438988851-26:[ctx-9798b557, ctx-8c8c58ff]) (logid:f819b910) ===END===  192.168.125.1 -- GET  jobId=36685271-97a9-4b16-882f-bff865b5999d&command=queryAsyncJobResult&response=json&
2025-08-14 16:08:38,951 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) template Template {"format":"RAW","id":202,"name":"centos-test","uniqueName":"202-2-1cf8d6c7-4a47-3dcf-b97f-1b461217902d","uuid":"4cc7ad31-624e-49de-b5ca-c071672519f7"} with id 202 is already in store:ImageStore {"id":1,"name":"nfs:\/\/192.168.125.120\/export\/secondary","uuid":"b423238d-1f38-463d-9ec2-5b30a6f05d08"}, type: Image
2025-08-14 16:08:38,962 DEBUG [o.a.c.s.d.d.LinstorPrimaryDataStoreDriverImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) LinstorPrimaryDataStoreDriverImpl.canCopy: SNAPSHOT -> TEMPLATE
2025-08-14 16:08:38,965 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Found [0] snapshots [[]] that have checkpoints for volume with id [3].
2025-08-14 16:08:38,972 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type TEMPLATE
2025-08-14 16:08:38,974 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Found [0] snapshots [[]] that have checkpoints for volume with id [3].
2025-08-14 16:08:38,985 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Found [0] snapshots [[]] that have checkpoints for volume with id [3].
2025-08-14 16:08:38,993 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Wait time setting on org.apache.cloudstack.storage.command.CopyCommand is 10800 seconds
2025-08-14 16:08:38,994 DEBUG [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Seq 5-8009933412255137826: Routed from 90520730730616
2025-08-14 16:08:38,994 DEBUG [c.c.a.t.Request] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Seq 5-8009933412255137826: Sending  { Cmd , MgmtId: 90520730730616, via: 5(s-2-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/mapper/storage-cs--2ab552d2--9ef2--494d--9c1e--df1209ebe970_00000_cs--27ef9239--b408--40a9--9278--85f1dfda4674","volume":{"uuid":"2ab552d2-9ef2-494d-9c1e-df1209ebe970","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3f90e70b-f6af-4ac9-8df4-5792fc477154","name":"Linstor","id":"1","poolType":"Linstor","host":"http://192.168.125.121:3370","path":"cloudstack","port":"3370","url":"Linstor://http://192.168.125.121:3370/cloudstack/?ROLE=Primary&STOREUUID=3f90e70b-f6af-4ac9-8df4-5792fc477154","isManaged":"false"}},"name":"ROOT-3","size":"(8.00 GB) 8589934592","path":"2ab552d2-9ef2-494d-9c1e-df1209ebe970","volumeId":"3","vmName":"i-2-3-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"1","id":"3","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3f90e70b-f6af-4ac9-8df4-5792fc477154","name":"Linstor","id":"1","poolType":"Linstor","host":"http://192.168.125.121:3370","path":"cloudstack","port":"3370","url":"Linstor://http://192.168.125.121:3370/cloudstack/?ROLE=Primary&STOREUUID=3f90e70b-f6af-4ac9-8df4-5792fc477154","isManaged":"false"}},"kvmIncrementalSnapshot":"false","vmName":"i-2-3-VM","name":"test","hypervisorType":"KVM","id":"1","quiescevm":"false","physicalSize":"0","accountId":"2","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/202","uuid":"4cc7ad31-624e-49de-b5ca-c071672519f7","id":"202","format":"RAW","accountId":"2","hvm":"true","displayText":"dfasdf","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.125.120/export/secondary","_role":"Image"}},"name":"202-2-1cf8d6c7-4a47-3dcf-b97f-1b461217902d","hypervisorType":"KVM","bootable":"false","uniqueName":"202-2-1cf8d6c7-4a47-3dcf-b97f-1b461217902d","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"executeInSequence":"false","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2025-08-14 16:08:39,040 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:[]) (logid:) Seq 5-8009933412255137826: Processing:  { Ans: , MgmtId: 90520730730616, via: 5, Ver: v1, Flags: 10, [{"com.cloud.agent.api.UnsupportedAnswer":{"result":"false","details":"Unsupported command issued: org.apache.cloudstack.storage.command.CopyCommand.  Are you sure you got the right type of server?","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-14 16:08:39,040 DEBUG [c.c.a.t.Request] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Seq 5-8009933412255137826: Received:  { Ans: , MgmtId: 90520730730616, via: 5(s-2-VM), Ver: v1, Flags: 10, { UnsupportedAnswer } }
2025-08-14 16:08:39,040 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Unsupported Command: Unsupported command issued: org.apache.cloudstack.storage.command.CopyCommand.  Are you sure you got the right type of server?
2025-08-14 16:08:39,046 DEBUG [c.c.t.TemplateManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand.  Are you sure you got the right type of server?
2025-08-14 16:08:39,051 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Updating resource Type = template count for Account with id = 2 Operation = decreasing Amount = 1
2025-08-14 16:08:39,064 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21, ctx-37aa414a]) (logid:36685271) Updating resource Type = secondary_storage count for Account with id = 2 Operation = decreasing Amount = (8.00 GB) 8589934592
2025-08-14 16:08:39,087 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-11:[ctx-38f6d4e7, job-21]) (logid:36685271) Unexpected exception while executing org.apache.cloudstack.api.command.admin.template.CreateTemplateCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand.  Are you sure you got the right type of server?
	at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1750)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at jdk.proxy3/jdk.proxy3.$Proxy259.createPrivateTemplate(Unknown Source)
	at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:305)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2025-08-14 16:08:39,088 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-38f6d4e7, job-21]) (logid:36685271) Complete async job-21, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create templateUnsupported command issued: org.apache.cloudstack.storage.command.CopyCommand.  Are you sure you got the right type of server?"}

versions

4.21.0.0-rc3, linstor primary storage

The steps to reproduce the bug

1. create/use primary storage Linstor
2. create an instance
3. create a volume snapshot of the instance
4. use create template from snapshot

What to do about it?

No response

[weizhouapache]

I tried to reproduce the issue on NFS storage. it is even worse, I cannot create a volume snapshot ......

[sureshanaparti]

Hi @weizhouapache I'm able to create volume snapshot with NFS as primary and secondary storage and also, able to create template & volume from the volume snapshot. Can you check the primary / secondary storage space available and threshold configured in your environment.

[sureshanaparti]

Hi @rp- Can you verify the fix here: #11452

[weizhouapache]

Hi @weizhouapache I'm able to create volume snapshot with NFS as primary and secondary storage and also, able to create template & volume from the volume snapshot. Can you check the primary / secondary storage space available and threshold configured in your environment.

@sureshanaparti
it is related to disk-only file-based snapshots. I created an issue for follow up: #11453

[sureshanaparti]

Fixed in #11452