SSVM cannot reconnect after connection disruption if there is an active event.

[PaulAngus]

ISSUE TYPE

• Bug Report

COMPONENT NAME

System VMs
(Maybe also KVM hosts)

CLOUDSTACK VERSION

4.11.0

CONFIGURATION

OS / ENVIRONMENT

4.11.0 environment with VMware

SUMMARY

If there is an interruption to mgmt server <-> agent communications while an action is taking place (such as the mgmt server restarting when the ssvm is performing a snapshot) the SSVM will not be able to reconnect due to following error:
2018-05-09 11:37:09,403 INFO [cloud.agent.Agent] (Agent-Handler-9:null) Lost connection to host: 10.220.136.127. Dealing with the remaining commands...
2018-05-09 11:37:09,404 INFO [cloud.agent.Agent] (Agent-Handler-9:null) Cannot connect because we still have 1 commands in progress.

STEPS TO REPRODUCE

During a volume snapshot exporting the ovf restart the management server.  

EXPECTED RESULTS

SSVM reconnects.

ACTUAL RESULTS

The storage VM does not reconnect to the management server and has an error such as: 
INFO  [cloud.agent.Agent] (Agent-Handler-9:null) Lost connection to host: 10.220.136.127. Dealing with the remaining commands...
INFO  [cloud.agent.Agent] (Agent-Handler-9:null) Cannot connect because we still have 1 commands in progress.

Once the job had finished it will reconnect but until this point all other jobs failed unless there is another secondary storage vm up and running.
The backup job even though it is forced to complete from secondary storage is still left in the db as state backing up forever so it does not make that it even waiting for it to finish.

[yadvr]

@PaulAngus This is by design of agent (not specific to any cloudstack version), any pending job in Agent's internal queue will block it from reconnecting until the job finishes. /cc @nvazquez @DaanHoogland - any comments on how to deal with this?

[DaanHoogland]

@rhtyd , as discussed on other media: We can make sure there is
0. make sure reconnect is always attempted

1. an other thread guarding there is always exacly one connection open.
2. have a setting that says 'agent.always.reconnect'
3. have a setting 'agent.job.queue.maximum.size' that makes sure only reconnect attempts are done when less than a certain number of jobs are in the queue.

or any combination of the above

[yadvr]

I looked at git history and code, there is no discussion about this design decision. When connection is lost with management server, the agent quickly shutsdown the links and other internal connection related data structure. There is high chance that any pending tasks in its internal queue even if succeeds will fail to send a proper/valid response, in which case they fail. I'll send a PR after some testing and start a discussion on why this historic design was used and if it is applicable for today's version.

[yadvr]

After doing several tests, I've submitted a quick fix that does not block agent from reconnection however other failure cases remain the same - #2638

A much bigger task would be to fix the agent-mgmt server execution design.