Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap · 2024-08-01T13:47:53Z

Description

This PR supports copying a snapshot to StorPool primary storage in different zones.

Added additional API param usestoragereplication for Admin users storageids in CloudStack API calls:

createSnapshot
copySnapshot
createSnapshotPolicy

The option snapshot.backup.to.secondary does not apply to the copy functionality. The snapshots will be copied only to the required primary storage in a different zone.
The user can create volumes/templates from the copied snapshots. The user can make copies only to a primary or to a secondary storage - at the moment, there is no option to do the copy on both.

The destination zone is a mandatory parameter if the users want to copy a snapshot, and the usestoragereplication if the copy has to be on the primary storage

For Admin users:
Those users can define the primary destination storage for the copy or to enable the setting use.storage.replication in the Primary storage settings.

For regular Users:
The Admin should enable the configuration setting use.storage.replication in the Primary storage settings.

For other storage plugins that want to adopt this functionality:

The Primary storage driver should have the capability CAN_COPY_SNAPSHOT_BETWEEN_ZONES
the respective plugin needs to implement the copySnapshot method in their SnapshotStrategy and that the driver can handle the COPY operation

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Manual and smoke tests with StorPool primary storage on multiple zones

codecov · 2024-08-01T13:56:23Z

Codecov Report

Attention: Patch coverage is 7.99508% with 748 lines in your changes missing coverage. Please review.

Project coverage is 16.29%. Comparing base (4d572fa) to head (e93e17d).
Report is 63 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	14.12%	144 Missing and 8 partials ⚠️
...ack/storage/snapshot/StorPoolSnapshotStrategy.java	0.00%	145 Missing ⚠️
...age/collector/StorPoolAbandonObjectsCollector.java	0.00%	63 Missing ⚠️
...tastore/driver/StorPoolPrimaryDataStoreDriver.java	0.00%	58 Missing ⚠️
...org/apache/cloudstack/snapshot/SnapshotHelper.java	16.94%	43 Missing and 6 partials ⚠️
...loudstack/storage/datastore/util/StorPoolUtil.java	0.00%	36 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	20.51%	29 Missing and 2 partials ⚠️
...oudstack/storage/snapshot/SnapshotServiceImpl.java	0.00%	27 Missing ⚠️
...e/wrapper/StorPoolModifyStorageCommandWrapper.java	0.00%	21 Missing ⚠️
...udstack/storage/datastore/util/StorPoolHelper.java	0.00%	21 Missing ⚠️
... and 26 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9478      +/-   ##
============================================
- Coverage     16.30%   16.29%   -0.02%     
- Complexity    13447    13457      +10     
============================================
  Files          5676     5676              
  Lines        499231   499930     +699     
  Branches      60374    60491     +117     
============================================
+ Hits          81415    81440      +25     
- Misses       408745   409408     +663     
- Partials       9071     9082      +11

Flag	Coverage Δ
uitests	`3.98% <ø> (-0.01%)`	⬇️
unittests	`17.15% <7.99%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2024-08-21T08:21:53Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-08-22T13:48:15Z

@blueorangutan package

blueorangutan · 2024-08-22T13:50:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-22T15:24:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10756

DaanHoogland · 2024-08-23T07:43:27Z

@blueorangutan test

blueorangutan · 2024-08-23T07:44:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-08-23T22:40:23Z

[SF] Trillian test result (tid-11155)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51322 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9478-t11155-kvm-ol8.zip
Smoke tests completed. 137 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Failure`	38.32	test_usage.py
test_01_vpc_site2site_vpn	`Failure`	308.08	test_vpc_vpn.py

github-actions · 2024-08-29T08:41:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

github-actions · 2024-09-04T08:40:33Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

DaanHoogland

clgtm, lots of style comments but no blockers

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

DaanHoogland · 2024-10-28T15:12:16Z

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

@@ -177,6 +107,74 @@
 import com.cloud.vm.dao.UserVmCloneSettingDao;
 import com.cloud.vm.dao.UserVmDao;
 import com.cloud.vm.dao.UserVmDetailsDao;
+import org.apache.cloudstack.api.ApiCommandResourceType;


Suggested change

import org.apache.cloudstack.api.ApiCommandResourceType;

import org.apache.cloudstack.api.ApiCommandResourceType;

...chema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImpl.java

DaanHoogland · 2024-10-28T15:13:26Z

...chema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImpl.java

+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine.Event;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine.State;
+import org.apache.commons.collections.CollectionUtils;


Suggested change

import org.apache.commons.collections.CollectionUtils;

import org.apache.commons.collections.CollectionUtils;

DaanHoogland · 2024-10-28T15:14:00Z

...orage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotServiceImpl.java

+import com.cloud.utils.db.TransactionCallbackNoReturn;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;


Suggested change

import com.cloud.utils.fsm.NoTransitionException;

import com.cloud.utils.fsm.NoTransitionException;

server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

DaanHoogland · 2024-10-28T15:38:01Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

@@ -174,6 +108,73 @@
 import com.cloud.vm.snapshot.VMSnapshotVO;
 import com.cloud.vm.snapshot.dao.VMSnapshotDao;
 import com.cloud.vm.snapshot.dao.VMSnapshotDetailsDao;
+import org.apache.cloudstack.acl.SecurityChecker;


Suggested change

import org.apache.cloudstack.acl.SecurityChecker;

import org.apache.cloudstack.acl.SecurityChecker;

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

github-actions · 2024-11-04T14:33:46Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2025-01-08T13:23:21Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2025-01-29T08:06:40Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2025-02-12T20:24:32Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

hide the primary storage from the users in the UI refactor smoke test

github-actions · 2025-05-02T01:39:16Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

boring-cyborg bot added component:api component:ceph component:orchestration component:primary-storage component:storage labels Aug 1, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 8cc7b49 to 0df6764 Compare August 14, 2024 09:14

slavkap marked this pull request as ready for review August 14, 2024 09:51

slavkap changed the title ~~[WIP] Support of snapshot copy to StorPool primary storage in different zones~~ Support of snapshot copy to StorPool primary storage in different zones Aug 14, 2024

github-actions bot added the status:has-conflicts label Aug 21, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 0df6764 to e0f4283 Compare August 22, 2024 12:49

github-actions bot removed the status:has-conflicts label Aug 22, 2024

github-actions bot added the status:has-conflicts label Aug 29, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from e0f4283 to 6f209d8 Compare August 30, 2024 09:08

github-actions bot removed the status:has-conflicts label Aug 30, 2024

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Outdated Show resolved Hide resolved

rohityadavcloud added this to the 4.20.0.0 milestone Sep 4, 2024

github-actions bot added the status:has-conflicts label Sep 4, 2024

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

DaanHoogland approved these changes Oct 28, 2024

View reviewed changes

github-actions bot added the status:has-conflicts label Nov 4, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 88d13d2 to ab01358 Compare December 10, 2024 09:51

github-actions bot removed the status:has-conflicts label Dec 10, 2024

github-actions bot added the status:has-conflicts label Jan 8, 2025

slavkap force-pushed the support-snapshot-copy-on-primary branch from ab01358 to 46695f8 Compare January 17, 2025 12:55

github-actions bot removed the status:has-conflicts label Jan 17, 2025

github-actions bot added the status:has-conflicts label Jan 29, 2025

slavkap force-pushed the support-snapshot-copy-on-primary branch from 46695f8 to 129a1b4 Compare February 4, 2025 12:18

github-actions bot removed the status:has-conflicts label Feb 4, 2025

github-actions bot added the status:has-conflicts label Feb 12, 2025

slavkap force-pushed the support-snapshot-copy-on-primary branch from 4d03837 to 6aa6e65 Compare March 12, 2025 10:12

github-actions bot removed the status:has-conflicts label Mar 12, 2025

slavkap added 10 commits April 15, 2025 09:39

Support of snapshot copy to different StorPool primary storage betwee…

8287dce

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

added option in the UI for the storage pools

6fd7ecc

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

Addressed comments

e9f4199

Address reviews

5532af7

fix snapshot deletion

e0fac8b

Address comments and fix conflicts

1cebfcf

small rearranges

7655edb

removed unused imports

d9bc950

hide the primary storage when a user copies the snapshots

f3a4263

hide the primary storage from users in UI

e93e17d

hide the primary storage from the users in the UI refactor smoke test

slavkap force-pushed the support-snapshot-copy-on-primary branch from 6aa6e65 to e93e17d Compare April 16, 2025 10:23

github-actions bot added the status:has-conflicts label May 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support of snapshot copy to StorPool primary storage in different zones #9478

Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

DaanHoogland left a comment

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 29, 2025

github-actions bot commented Feb 12, 2025

github-actions bot commented May 2, 2025

	import org.apache.cloudstack.api.ApiCommandResourceType;

	import org.apache.cloudstack.api.ApiCommandResourceType;

	import org.apache.commons.collections.CollectionUtils;

	import org.apache.commons.collections.CollectionUtils;

	import com.cloud.utils.fsm.NoTransitionException;
	import com.cloud.utils.fsm.NoTransitionException;

	import org.apache.cloudstack.acl.SecurityChecker;

	import org.apache.cloudstack.acl.SecurityChecker;

Support of snapshot copy to StorPool primary storage in different zones #9478

Are you sure you want to change the base?

Support of snapshot copy to StorPool primary storage in different zones #9478

Conversation

slavkap commented Aug 1, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

DaanHoogland left a comment

Choose a reason for hiding this comment

DaanHoogland Oct 28, 2024

Choose a reason for hiding this comment

DaanHoogland Oct 28, 2024

Choose a reason for hiding this comment

DaanHoogland Oct 28, 2024

Choose a reason for hiding this comment

DaanHoogland Oct 28, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 4, 2024

github-actions bot commented Jan 8, 2025

github-actions bot commented Jan 29, 2025

github-actions bot commented Feb 12, 2025

github-actions bot commented May 2, 2025

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading