Better Handling of First Master Replacement

[benyanke]

Given the list of hosts below, if mgr1 dies, or is removed from the swarm, the current behavior is the creation of a new swarm, despite a swarm quorum still existing with three workers and a quorum of two managers.

However, if mgr2 dies and is replaced with a fresh node, it is properly added to the existing cluster of manager1 and manager3, as expected.

Ideally, the tool should check to see if there is already a quorum among the existing masters before trying to create a new one.

$ cat inventory
[targets]
work[1:3].swarm.domain.com
mgr[1:3].swarm.domain.com


[docker_engine]
work[1:3].swarm.domain.com
mgr[1:3].swarm.domain.com

[docker_swarm_manager]
mgr[1:3].swarm.domain.com swarm_labels='["node_is_master"]'

[docker_swarm_worker]
work[1:3].swarm.domain.com swarm_labels='["node_is_worker"]'