Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When upgrading cluster nodes, determine current Leader and upgrade that one last #11

Open
s4ke opened this issue Jun 19, 2023 · 0 comments

Comments

@s4ke
Copy link
Member

s4ke commented Jun 19, 2023

There seems to be an edge case that might happen when the current leader is upgraded to a newer version first, at least an incident that occured today suggests this.

When upgrading a 3 node cluster from 24.0.0 to 24.0.2 we saw issues where upgrading the leader of the cluster caused a scenario where two members of the cluster had the same name according to the output of docker node ls on the leader at that time.

grafik

When trying to demote the leader at that time, both the leader and the node with the duplicated name got demoted (when doing that by ID instead of name!), causing the swarm to lose quorum.

To prevent issues like this in the future (with the hopes that this can be prevented by proper upgrade order), we should determine the current manager of the swarm and upgrade that node last instead of abusing one of the managers as the "main manager".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant