marvin: fix test failures when changing service offering of a VM

[Pearl1594]

Description

This PR fixes the test failures noticed on VMware test runs where in change of service offering for a VM fails with the following error:

2021-02-04 11:58:05,241 - CRITICAL - EXCEPTION: test_04_change_offering_small: ['Traceback (most recent call last):\n', '  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run\n    testMethod()\n', '  File "/marvin/tests/smoke/test_service_offerings.py", line 511, in test_04_change_offering_small\n    self.apiclient.changeServiceForVirtualMachine(cmd)\n', '  File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 1086, in changeServiceForVirtualMachine\n    response = self.connection.marvinRequest(command, response_type=response, method=method)\n', '  File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest\n    raise e\n', 'CloudstackAPIException: Execute cmd: changeserviceforvirtualmachine failed, due to: errorCode: 431, errorText:VM current state is : PowerOn. But VM should be in PowerOff state.\n']

This seems to be a consequence of changing the enabling volumeresize with changeServiceForVirtualMachine introduced with : #4491. The failure is seen in test_service_offerings.py -> test_04_change_offering_small and test_vm_snapshots.py -> test_change_service_offering_for_vm_with_snapshots

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

[Pearl1594]

@blueorangutan package

[blueorangutan]

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[shwstppr]

@Pearl1594 does this need to go in 4.14?

[blueorangutan]

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2649

[Pearl1594]

@Pearl1594 does this need to go in 4.14?

@shwstppr I don't see these test failures on 4.14 based PRs

[Pearl1594]

@blueorangutan test centos7 vmware-67u3

[blueorangutan]

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

[yadvr]

We want this in 4.14 branch, @Pearl1594 cc @shwstppr @DaanHoogland

[DaanHoogland]

tests are never critical for release unless they relate to a fix i'd say. In this case it is a corner case, let's include it.

[blueorangutan]

Trillian test result (tid-3496)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41292 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4651-t3496-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_upgrade_kubernetes_cluster	Failure	801.35	test_kubernetes_clusters.py

[DaanHoogland]

code looks good, but one little nasty question: The fix is basically just waiting a bit more because stopping a vm now takes longer. This means that from a user point of view usability has gone down. the VM stopping is no longer as quick an action as it was. Is this easily explained and should we document that somewhere?
@Pearl1594 @shwstppr @rhtyd (cc @slavkap as you created the original fix this test-fix pertains to)

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos7 ✖centos8 ✔debian. JID-2652

[Pearl1594]

@DaanHoogland The time taken to stop the VM hasn't changed, however, earlier with the changeServiceForVirtualMachine cmd, we only updated the DB with the new service offering, and root disk resize wasn't taken care of. With the changes made, resizing of volumes is handled as part of VM upgrade and that enforces that VM's power state be off. Hence, we need to wait until it's off, to proceed further.

[yadvr]

@blueorangutan test centos7 vmware-67u3

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

[yadvr]

I'll merge this based on test from master.

[blueorangutan]

Trillian test result (tid-3498)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42759 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4651-t3498-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_upgrade_kubernetes_cluster	Error	3775.48	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	Error	123.12	test_kubernetes_clusters.py