Support for Management Server Maintenance Mode

[sureshanaparti]

Description

This PR adds support for Management Server Maintenance Mode. More details below.

• New APIs: prepareForMaintenance and cancelMaintenance, with required parameter - managementserverid.

• New management server states for maintenance: PreparingForMaintenance, Maintenance.

• listHosts API with optional parameter – managementserverid, to list the hosts connected to the management server.

• Support management server maintenance when more than one active management servers available.

• Triggers transfer agents to other available management servers for maintenance, new agent command MigrateAgentConnectionCommand to initiate transfer of indirect agents.

• New global config 'management.server.maintenance.timeout', to set the timeout (in mins) for the management server maintenance window, default: 60 mins.

• UI changes: Prepare and Cancel Maintenance in Management Server section, Connected Agents tab, New fields for hosts and management servers.

This also updates the shutdown plugin introduced here: #6755, to maintenance plugin (as common plugin for MS maintenance and shutdown operations).

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Manually tested prepare and cancel management server maintenance operations.

How did you try to break this feature and the system with this change?

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

Attention: Patch coverage is 8.83095% with 1084 lines in your changes missing coverage. Please review.

Project coverage is 16.18%. Comparing base (98f5663) to head (d60f8d1).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
...enance/ManagementServerMaintenanceManagerImpl.java	14.48%	298 Missing and 3 partials ⚠️
...cloud/agent/manager/ClusteredAgentManagerImpl.java	0.00%	139 Missing ⚠️
...loudstack/agent/lb/IndirectAgentLBServiceImpl.java	12.24%	84 Missing and 2 partials ⚠️
agent/src/main/java/com/cloud/agent/Agent.java	0.00%	64 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	6.66%	56 Missing ⚠️
.../response/ManagementServerMaintenanceResponse.java	0.00%	35 Missing ⚠️
...stack/framework/jobs/impl/AsyncJobManagerImpl.java	6.06%	31 Missing ⚠️
...cloud/agent/api/MigrateAgentConnectionCommand.java	0.00%	24 Missing ⚠️
...cloud/cluster/dao/ManagementServerHostDaoImpl.java	15.38%	22 Missing ⚠️
...d/cluster/dao/ManagementServerHostPeerDaoImpl.java	0.00%	22 Missing ⚠️
... and 47 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9854      +/-   ##
============================================
- Coverage     16.19%   16.18%   -0.01%     
- Complexity    13051    13066      +15     
============================================
  Files          5645     5653       +8     
  Lines        494567   495556     +989     
  Branches      59955    60070     +115     
============================================
+ Hits          80088    80204     +116     
- Misses       405642   406512     +870     
- Partials       8837     8840       +3     

Flag	Coverage Δ
uitests	4.01% <ø> (-0.01%)	⬇️
unittests	17.03% <8.83%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11437

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11701)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52486 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11701-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	Error	0.05	test_ms_maintenance_and_safe_shutdown.py
test_02_unsecure_vm_migration	Error	449.14	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	Error	382.76	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	Error	378.56	test_vm_life_cycle.py

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11509

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11514

[blueorangutan]

[SF] Trillian test result (tid-11748)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51108 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11748-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	Error	0.04	test_ms_maintenance_and_safe_shutdown.py

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11577

[sureshanaparti]

@blueorangutan test

[blueorangutan]

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-11784)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 60592 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t11784-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_prepare_and_cancel_maintenance	Error	0.04	test_ms_maintenance_and_safe_shutdown.py
test_01_secure_vm_migration	Error	408.30	test_vm_life_cycle.py

[sureshanaparti]

@blueorangutan package

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12218

[rohityadavcloud]

@blueorangutan test

[blueorangutan]

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12240

[sureshanaparti]

@blueorangutan package

[blueorangutan]

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12245

[blueorangutan]

[SF] Trillian test result (tid-12210)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54557 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t12210-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	Error	1.26	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	Error	3.40	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	Error	3.40	test_ipv4_routing.py

[sureshanaparti]

[SF] Trillian test result (tid-12210) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 54557 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9854-t12210-kvm-ol8.zip Smoke tests completed. 140 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 1.26 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.40 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 3.40 test_ipv4_routing.py

these failures are not related to the changes in this PR.

[harikrishna-patnala]

code LGTM