PowerFlex/ScaleIO - MDM and host SDC connection enhancements #11047
Conversation
boring-cyborg
bot
added
component:kvm
component:primary-storage
component:storage
labels
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #11047 +/- ##
==========================================
Coverage 16.57% 16.58%
- Complexity 13971 13988 +17
==========================================
Files 5743 5746 +3
Lines 510648 511103 +455
Branches 62105 62170 +65
==========================================
+ Hits 84641 84754 +113
- Misses 416534 416863 +329
- Partials 9473 9486 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the handling of ScaleIO MDM and host SDC connections by introducing new configuration keys for timeouts, validation, and blocking behavior along with updating the underlying command execution and logging mechanisms.
- Introduced new configuration keys, including MdmsChangeApplyTimeout, ValidateMdmsOnConnect, and BlockSdcUnprepareIfRestartNeededAndVolumesAreAttached.
- Updated MDM add/remove logic to use varargs and improved command execution via Script.executeCommand.
- Adjusted test cases and adapter logic to account for the new ScaleIO configurations.
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| utils/src/main/java/com/cloud/utils/script/Script.java | Added a new executeCommand(String) method for command execution including stdout/stderr handling. |
| plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java | Refactored MDM add/remove methods and updated command templates, patterns, and file read operations. |
| plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManagerImpl.java | Updated configuration details sent to hosts by including new timeout settings. |
| plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManager.java | Introduced new config key definitions and updated the getConfigKeys() return values. |
| plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java | Injected the new configuration details during maintain and cancellation procedures. |
| plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/client/ScaleIOGatewayClient.java | Added a short documentation comment for STORAGE_POOL_MDMS. |
| plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptorTest.java | Updated test mocks to reflect changes in command executions for MDM removal. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptor.java | Introduced validation of MDM state and timeout application after MDM changes. |
| plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtModifyStoragePoolCommandWrapper.java | Wrapped storage pool creation in a try/catch block to better handle CloudRuntimeException. |
Comments suppressed due to low confidence (1)
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java:315
- [nitpick] The ordering of stdout and stderr here is reversed compared to other usages of Script.executeCommand; consider using a consistent ordering (stdout as first, stderr as second) to avoid confusion.
String stdErr = result.first(); String stdOut = result.second();
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 13811 |
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13812 |
|
@blueorangutan test |
|
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian Build Failed (tid-13562) |
|
@blueorangutan test |
|
@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-13586)
|
...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java
Outdated
Show resolved
Hide resolved
...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java
Outdated
Show resolved
Hide resolved
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13907 |
|
@sureshanaparti as these are general bugfix/improvements for powerflex storage, should these be also applicable for 4.20 branch (esp these changes are scaleio/powerflex storage plugin related with no DB changes)? |
sureshanaparti
changed the title
@rohityadavcloud There are no DB changes, but changes in host SDC connection behavior earlier in this PR: #9903 (already part of main), where the SDC connection is controlled using MDMs addition/removal instead of SDC service scini start/stop. This PR changes are on top of these, and introduces some configurations to validate the MDMs and apply timeout after add/remove MDMs. I think, it's better to keep this new behavior with main itself. |
|
@blueorangutan test |
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14072 |
|
@blueorangutan test |
|
@vladimirpetrov a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
[SF] Trillian test result (tid-13713)
|
|
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 14085 |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14093 |
|
@blueorangutan test |
|
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-13730)
|
|
@blueorangutan package |
|
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14160 |
|
@blueorangutan test |
|
@vladimirpetrov a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
There was a problem hiding this comment.
LGTM based on manual testing, tested various scenarios with old/new ScaleIO SDC clients, 2 ScaleIO pools from the same/different cluster, with powerflex.block.sdc.unprepare=true/false. Tested global settings:
powerflex.connect.on.demand
powerflex.block.sdc.unprepare
powerflex.mdm.change.apply.wait
powerflex.sdc.service.wait
powerflex.mdm.validate.on.connect
|
Let's wait for final smoketest run before merging this. |
|
[SF] Trillian test result (tid-13779)
|
github-project-automation
bot
moved this from In Progress
to Done
in Apache CloudStack 4.21.0
…11047) * Cumulative enhancements fix for ScaleIO: MDM add/remove, Host prepare/unprepare, validate Storage Pool can be created in Agent. - Implemented validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted - Implemented Storage Pool validation by checking whether MDM addresses from configuration file and from memory (using CLI) matches, otherwise file ModifyStoragePool command. - Introduced configuration key to apply timeout after making MDM changes for ScaleIO: powerflex.mdm.change.apply.timeout.ms (default 1000ms) - Implemented logic to apply timeout after making MDM changes for ScaleIO in prepare and unprepare logic - Added detection of MDM removal support via CLI - If MDM removal support via CLI supported then use CLI, fall back to edit drv_cfg.txt and restart scini instead Co-authored-by: Suresh Kumar Anaparti <suresh.anaparti@shapeblue.com> Co-authored-by: mprokopchuk <mprokopchuk@apple.com>
Description
This PR enhances the PowerFlex/ScaleIO MDM and host SDC connections, includes the following changes (and some code improvements).
Introduced timeout configuration 'powerflex.mdm.change.apply.wait' (Default value: 1000 ms) at zone scope, to wait after MDM addition, and before & after MDM removal changes made on Host with ScaleIO SDC. Also, Changes to apply the wait time after making MDM changes for ScaleIO in prepare and unprepare logic.
Introduced configuration flag 'powerflex.block.sdc.unprepare' (Default is false) at zone scope, to enable/disable blocking unprepare ScaleIO SDC connection when SDC client restart required (upon PowerFlex MDM removal i.e. no support for --remove_mdm in drv_cfg cmd) and there are volumes attached to the Host. Added validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted.
Introduced configuration flag 'powerflex.mdm.validate.on.connect' (Default is false) at zone scope, to enable/disable validation of MDM addresses on Host, in the Configuration File and in CLI cmd (drv_cfg --query_mdms) output matches or not, during storage pool registration in agent.
Added detection of MDM removal support via CLI. If MDM removal support via CLI supported then use CLI, Otherwise fall back to edit drv_cfg.txt and restart scini as earlier. Tested with /opt/emc/scaleio/sdc/bin/drv_cfg --version: DellEMC PowerFlex Version: R3_6.4000.124, with cmd: /opt/emc/scaleio/sdc/bin/drv_cfg --remove_mdm.
Added agent property 'powerflex.sdc.service.wait' for the time (in secs) to wait after SDC service start/restart/stop, and retries to fetch SDC id/guid.
Updated to allow unprepare SDC when there are no volumes mapped on the host for other connected pools (with same SDC Id, i.e pools of same PowerFlex storage cluster).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Tested the new settings, and PowerFlex SDC connections (MDM add/remove) with VM & Volume operations.
How did you try to break this feature and the system with this change?