[PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs for the storage SDC connections #9903

sureshanaparti · 2024-11-07T13:45:36Z

Description

This PR adds & removes PowerFlex/ScaleIO MDM IP addressess while preparing & unpreparing the storage SDC connections (instead of start & stop scini), when powerflex.connect.on.demand is true. And adds & removes MDM IP addresses during Host connection/disconnection to/from storage pool, when powerflex.connect.on.demand is false.

MDMs details persisted using powerflex.storagepool.mdms detail, in storage_pool_details table:

mysql> SELECT * FROM cloud.storage_pool_details WHERE pool_id = 3 AND name = 'powerflex.storagepool.mdms';
+----+---------+---------------------------------+------------------------------------------------------+---------+
| id | pool_id | name                            | value                                                | display |
+----+---------+---------------------------------+------------------------------------------------------+---------+
|  2 |       3 | powerflex.storagepool.mdms      | 10.0.33.178,10.0.33.87                               |       1 |
+----+---------+---------------------------------+------------------------------------------------------+---------+

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested VM start/stop operations, noticed MDMs added/removed in hosts.

sureshanaparti · 2024-11-07T13:47:04Z

@blueorangutan package

blueorangutan · 2024-11-07T13:48:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-11-07T14:34:38Z

Codecov Report

Attention: Patch coverage is 9.29095% with 371 lines in your changes missing coverage. Please review.

Project coverage is 16.41%. Comparing base (2df1ac5) to head (538d0f1).
Report is 101 commits behind head on main.

Files with missing lines	Patch %	Lines
...orage/datastore/manager/ScaleIOSDCManagerImpl.java	0.00%	74 Missing ⚠️
...torage/datastore/provider/ScaleIOHostListener.java	0.00%	68 Missing ⚠️
.../hypervisor/kvm/storage/ScaleIOStorageAdaptor.java	21.62%	25 Missing and 4 partials ⚠️
...atastore/driver/ScaleIOPrimaryDataStoreDriver.java	10.34%	25 Missing and 1 partial ⚠️
...re/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java	33.33%	17 Missing and 3 partials ⚠️
...ck/storage/datastore/api/StorageConfiguration.java	0.00%	19 Missing ⚠️
.../hypervisor/kvm/storage/KVMStoragePoolManager.java	0.00%	17 Missing ⚠️
...cloudstack/storage/datastore/util/ScaleIOUtil.java	50.00%	14 Missing and 2 partials ⚠️
...ore/client/ScaleIOGatewayClientConnectionPool.java	0.00%	14 Missing ⚠️
...a/com/cloud/storage/StoragePoolAutomationImpl.java	0.00%	11 Missing ⚠️
... and 14 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9903      +/-   ##
============================================
- Coverage     16.42%   16.41%   -0.01%     
+ Complexity    13549    13547       -2     
============================================
  Files          5673     5676       +3     
  Lines        500080   500420     +340     
  Branches      60506    60575      +69     
============================================
+ Hits          82153    82168      +15     
- Misses       408794   409103     +309     
- Partials       9133     9149      +16

Flag	Coverage Δ
uitests	`4.00% <ø> (ø)`
unittests	`17.28% <9.29%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2024-11-07T14:53:46Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11524

sureshanaparti · 2024-11-11T10:06:20Z

@blueorangutan test

blueorangutan · 2024-11-11T10:08:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-12T02:55:25Z

[SF] Trillian test result (tid-11772)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58105 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9903-t11772-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_unsecure_vm_migration	`Error`	503.38	test_vm_life_cycle.py

sureshanaparti · 2024-12-24T08:36:44Z

@blueorangutan package

blueorangutan · 2024-12-24T08:38:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-12-24T08:57:52Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11894

sureshanaparti · 2024-12-26T06:12:10Z

@blueorangutan package

blueorangutan · 2024-12-26T06:14:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-12-26T07:23:15Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11910

sureshanaparti · 2024-12-27T04:18:54Z

@blueorangutan test

blueorangutan · 2024-12-27T04:20:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2024-12-27T14:26:20Z

@blueorangutan package

blueorangutan · 2024-12-27T14:28:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-12-27T15:38:02Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11925

github-actions · 2025-03-27T17:36:04Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

… storage SDC connections (instead of start & stop scini)

…rom storage pool when powerflex.connect.on.demand is false

…er are connected

sureshanaparti · 2025-04-30T10:38:33Z

@blueorangutan package

blueorangutan · 2025-04-30T10:40:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-30T12:10:19Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13227

sureshanaparti · 2025-05-05T07:43:18Z

@blueorangutan test

blueorangutan · 2025-05-05T07:44:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-05-05T23:11:30Z

[SF] Trillian test result (tid-13202)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52889 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9903-t13202-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_restore_vm_strict_tags_failure	`Failure`	53.35	test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failure	`Failure`	56.73	test_vm_strict_host_tags.py
test_06_deploy_vm_on_any_host_with_strict_tags_failure	`Failure`	3.66	test_vm_strict_host_tags.py

harikrishna-patnala

code LGTM

kiranchavala

LGTM

Steps to validata

Add a powerflex storage to the cloudstack setup
Make sure the global setting "powerflex.connect.on.demand " is set to false
Login to the kvm host make sure the scini service is running

yum install EMC-ScaleIO-sdc-3.6-400.107.el8.x86_64.rpm -y
systemctl status scini.service

Stop the service

systemctl status scini.service

Deploy the vm

The vm deployment will fail

Enable the global setting "powerflex.connect.on.demand" to true
Deploy a vm

The vm deployment will succed

Check the scini service

systemctl status scini.service

It should be in active state and started by Cloudstack

sureshanaparti · 2025-05-15T07:11:54Z

Merging this, based on the reviews and tests.

…onnections (apache#9903) * Add & Remove PowerFlex/ScaleIO MDMs while preparing & unpreparing the storage SDC connections (instead of start & stop scini) * Add/Remove MDM IP addresses during Host connection/disconnection to/from storage pool when powerflex.connect.on.demand is false * unit test fixes * Don't remove MDM IPs from SDC when any volumes mapped to SDC * Don't remove MDM IPs when other pools of same ScaleIO/PowerFlex cluster are connected * rebase fixes * update changes, to not remove/disconnect MDMs on maintenance * import fixes after rebase

boring-cyborg bot added component:api component:kvm component:primary-storage component:storage labels Nov 7, 2024

sureshanaparti added this to the 4.21.0.0 milestone Nov 7, 2024

sureshanaparti changed the title ~~[PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs while preparing & unpreparing the storage SDC connections~~ [PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs for the storage SDC connections Dec 24, 2024

sureshanaparti force-pushed the powerflex_scaleio-connection-with-mdms branch from c8603d2 to 86f5f0c Compare December 26, 2024 06:10

apache deleted a comment from blueorangutan Dec 27, 2024

sureshanaparti force-pushed the powerflex_scaleio-connection-with-mdms branch from e7b0700 to b065389 Compare December 27, 2024 14:01

apache deleted a comment from blueorangutan Dec 27, 2024

sureshanaparti and others added 8 commits April 30, 2025 15:32

Add & Remove PowerFlex/ScaleIO MDMs while preparing & unpreparing the…

54a8ac6

… storage SDC connections (instead of start & stop scini)

Add/Remove MDM IP addresses during Host connection/disconnection to/f…

75dd5e5

…rom storage pool when powerflex.connect.on.demand is false

unit test fixes

d2e635a

Don't remove MDM IPs from SDC when any volumes mapped to SDC

93d6a28

Don't remove MDM IPs when other pools of same ScaleIO/PowerFlex clust…

73025da

…er are connected

rebase fixes

d806f1f

update changes, to not remove/disconnect MDMs on maintenance

89a939c

import fixes after rebase

538d0f1

sureshanaparti force-pushed the powerflex_scaleio-connection-with-mdms branch from 8d20c7f to 538d0f1 Compare April 30, 2025 10:35

github-actions bot removed the status:has-conflicts label Apr 30, 2025

sureshanaparti requested review from harikrishna-patnala and kiranchavala May 6, 2025 07:14

harikrishna-patnala approved these changes May 6, 2025

View reviewed changes

sureshanaparti assigned kiranchavala May 9, 2025

kiranchavala approved these changes May 14, 2025

View reviewed changes

sureshanaparti merged commit 572fc11 into apache:main May 15, 2025
17 of 26 checks passed

DaanHoogland deleted the powerflex_scaleio-connection-with-mdms branch May 15, 2025 07:31

sureshanaparti added the type:improvement label Jun 4, 2025

sureshanaparti added this to Apache CloudStack 4.21.0 Jun 5, 2025

sureshanaparti mentioned this pull request Jun 27, 2025

PowerFlex/ScaleIO - MDM and host SDC connection enhancements #11047

Merged

14 tasks

sureshanaparti moved this to Done in Apache CloudStack 4.21.0 Jul 14, 2025

[PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs for the storage SDC connections #9903

[PowerFlex] Add & Remove PowerFlex/ScaleIO MDMs for the storage SDC connections #9903

Uh oh!

Conversation

sureshanaparti commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

sureshanaparti commented Nov 7, 2024

Uh oh!

blueorangutan commented Nov 7, 2024

Uh oh!

codecov bot commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Nov 7, 2024

Uh oh!

sureshanaparti commented Nov 11, 2024

Uh oh!

blueorangutan commented Nov 11, 2024

Uh oh!

blueorangutan commented Nov 12, 2024

Uh oh!

sureshanaparti commented Dec 24, 2024

Uh oh!

blueorangutan commented Dec 24, 2024

Uh oh!

blueorangutan commented Dec 24, 2024

Uh oh!

sureshanaparti commented Dec 26, 2024

Uh oh!

blueorangutan commented Dec 26, 2024

Uh oh!

blueorangutan commented Dec 26, 2024

Uh oh!

sureshanaparti commented Dec 27, 2024

Uh oh!

blueorangutan commented Dec 27, 2024

Uh oh!

sureshanaparti commented Dec 27, 2024

Uh oh!

blueorangutan commented Dec 27, 2024

Uh oh!

blueorangutan commented Dec 27, 2024

Uh oh!

github-actions bot commented Mar 27, 2025

Uh oh!

sureshanaparti commented Apr 30, 2025

Uh oh!

blueorangutan commented Apr 30, 2025

Uh oh!

blueorangutan commented Apr 30, 2025

Uh oh!

sureshanaparti commented May 5, 2025

Uh oh!

blueorangutan commented May 5, 2025

Uh oh!

blueorangutan commented May 5, 2025

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

kiranchavala left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented May 15, 2025

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 7, 2024 •

edited

Loading

kiranchavala left a comment •

edited

Loading