Support of snapshot copy to primary storage in different zones.

[slavkap]

Description

This PR allows copying a snapshot from a primary storage pool in one zone to a primary storage pool in different zones without involving secondary storage.
This feature is currently implemented only in the StorPool storage plugin. Other storage plugins can add this feature if the storage pools support direct copies of snapshots from one pool to another.

Added additional API param usestoragereplication and one more for Admin users storageids in CloudStack API calls:

createSnapshot
copySnapshot
createSnapshotPolicy

The option snapshot.backup.to.secondary does not apply to the copy functionality. The snapshots will be copied only to the required primary storage in a different zone.
The user can create volumes/templates from the copied snapshots. The user can make copies only to a primary or to a secondary storage - at the moment, there is no option to do the copy on both.

The destination zone is a mandatory parameter if the users want to copy a snapshot, and the usestoragereplication if the copy has to be on the primary storage

For Admin users:
Those users can define the primary destination storage for the copy or to enable the setting use.storage.replication in the Primary storage settings.

For regular Users:
The Admin should enable the configuration setting use.storage.replication in the Primary storage settings.

For other storage plugins that want to adopt this functionality:

• The Primary storage driver should have the capability CAN_COPY_SNAPSHOT_BETWEEN_ZONES
• the respective plugin needs to implement the copySnapshot method in their SnapshotStrategy and that the driver can handle the COPY operation

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

How Has This Been Tested?

Manual and smoke tests with StorPool primary storage on multiple zones

[codecov]

Codecov Report

❌ Patch coverage is 6.69746% with 808 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.33%. Comparing base (a60c8ca) to head (6668140).
⚠️ Report is 38 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	8.08%	174 Missing and 8 partials ⚠️
...ack/storage/snapshot/StorPoolSnapshotStrategy.java	0.00%	154 Missing ⚠️
...age/collector/StorPoolAbandonObjectsCollector.java	0.00%	69 Missing ⚠️
...org/apache/cloudstack/snapshot/SnapshotHelper.java	14.70%	52 Missing and 6 partials ⚠️
...tastore/driver/StorPoolPrimaryDataStoreDriver.java	0.00%	56 Missing ⚠️
...loudstack/storage/datastore/util/StorPoolUtil.java	0.00%	37 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	24.39%	29 Missing and 2 partials ⚠️
...oudstack/storage/snapshot/SnapshotServiceImpl.java	0.00%	27 Missing ⚠️
...e/wrapper/StorPoolModifyStorageCommandWrapper.java	0.00%	21 Missing ⚠️
...udstack/storage/datastore/util/StorPoolHelper.java	0.00%	21 Missing ⚠️
... and 25 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9478      +/-   ##
============================================
- Coverage     17.35%   17.33%   -0.02%     
- Complexity    15189    15198       +9     
============================================
  Files          5883     5883              
  Lines        524514   525259     +745     
  Branches      64007    64131     +124     
============================================
+ Hits          91013    91042      +29     
- Misses       423216   423924     +708     
- Partials      10285    10293       +8     

Flag	Coverage Δ
uitests	3.63% <ø> (-0.01%)	⬇️
unittests	18.37% <6.69%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10756

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11155)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51322 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9478-t11155-kvm-ol8.zip
Smoke tests completed. 137 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	Failure	38.32	test_usage.py
test_01_vpc_site2site_vpn	Failure	308.08	test_vpc_vpn.py

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[slavkap]

@blueorangutan package

[blueorangutan]

@slavkap a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[slavkap]

Thank you, @rosi-shapeblue, for the detailed information!

The UI issue with the label has been fixed in the latest commit. Additionally, snapshot allocation will no longer occur if the storage doesn't support copying to another zone.

I’ve tested snapshot copying between zones using StorPool, and confirmed that this feature doesn't break existing functionality for copying snapshots between zones on secondary storage. Testing was also done with NFS as the primary storage.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14503

[sureshanaparti]

@rosi-shapeblue can you check if your concerns are addressed?

[rosi-shapeblue]

LGTM. The issues that were discussed have been addressed.

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-14022)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 47587 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9478-t14022-kvm-ol8.zip
Smoke tests completed. 146 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File