Possible deadlock in JoinCluster #2286
Comments
|
This bug was introduced after removing timeout in joinCluster call (807976c). I can see the proposal being dropped/ignored silently. When you start three nodes, then if two followers call joinCluster concurrently then one of the entryConfproposal would get dropped because etcd/raft by design allows only one pending entryConfChange. So one of the node was stuck waiting for joinCluster to finish. |
|
Does it return an error when it drops the proposal? |
|
It pushes to a channel and returns. Drop happens later asynchronously. We can probably serialise join cluster calls. |
|
We should talk to Etcd/Raft guys before we make any changes here. I think there has to be a better solution here, than another quick fix. |
This can cause a server to just never join the cluster. |
Cluster converges smoothly with #2467. |
manishrjain
added a commit
that referenced
this issue
Jul 1, 2018
This PR fixes #2286 . - CheckQuorum was causing us multiple issues. When doing a 5-node Zero cluster bootstrap, it would cause a leader to step down when the size of the cluster is 2, then causing all the rest of the joins to be blocked indefinitely. It would also cause leader step down in a seemingly healthy cluster which is processing proposals. CheckQuorum was mandated by raft.ReadOnlyLeaseBased, which is a less safe option to do linearizable reads. Switch ReadOnlyOption back to raft.ReadOnlySafe. Moreover, we don't need to do quorum based lin reads in the Alpha servers, because of the switch to proposing and then applying transaction updates. - raft.ReadIndex is not working for some reason. So, commented out its usage in Zero (and removed it from Alpha permanently). Needs to be fixed when the following issue is resolved. etcd-io/etcd#9893 - The logic to do lin reads was replicated in both Zero and Alpha. Refactor that into one place in conn/node.go. - Retry conf change proposals if they timeout. This mechanism is similar to the one introduced for normal proposals in a previous commit 06ea4c. - Use a lock to only allow one JoinCluster call at a time. Block JoinCluster until node.AddToCluster is successful (or return the error). - Set raft library to 3.2.23. Before upgrade, we were at 3.2.6. Commit log: * Trying to understand why JoinCluster doesn't work properly. * Fucking works. Fucking works. * It all works now. * More Dgraph servers. Found a new issue where requesting read quorum doesn't respond. * Refactor wait lin read code and move it to conn/node.go * Remove lin read wait for server, because txn timestamp should be sufficient for waiting. Also, for the time being, comment out lin read wait from Zero as well.
dna2github
pushed a commit
to dna2fork/dgraph
that referenced
this issue
Jul 19, 2019
This PR fixes dgraph-io#2286 . - CheckQuorum was causing us multiple issues. When doing a 5-node Zero cluster bootstrap, it would cause a leader to step down when the size of the cluster is 2, then causing all the rest of the joins to be blocked indefinitely. It would also cause leader step down in a seemingly healthy cluster which is processing proposals. CheckQuorum was mandated by raft.ReadOnlyLeaseBased, which is a less safe option to do linearizable reads. Switch ReadOnlyOption back to raft.ReadOnlySafe. Moreover, we don't need to do quorum based lin reads in the Alpha servers, because of the switch to proposing and then applying transaction updates. - raft.ReadIndex is not working for some reason. So, commented out its usage in Zero (and removed it from Alpha permanently). Needs to be fixed when the following issue is resolved. etcd-io/etcd#9893 - The logic to do lin reads was replicated in both Zero and Alpha. Refactor that into one place in conn/node.go. - Retry conf change proposals if they timeout. This mechanism is similar to the one introduced for normal proposals in a previous commit 06ea4c. - Use a lock to only allow one JoinCluster call at a time. Block JoinCluster until node.AddToCluster is successful (or return the error). - Set raft library to 3.2.23. Before upgrade, we were at 3.2.6. Commit log: * Trying to understand why JoinCluster doesn't work properly. * Fucking works. Fucking works. * It all works now. * More Dgraph servers. Found a new issue where requesting read quorum doesn't respond. * Refactor wait lin read code and move it to conn/node.go * Remove lin read wait for server, because txn timestamp should be sufficient for waiting. Also, for the time being, comment out lin read wait from Zero as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On recent builds, I've seen new clusters get stuck during concurrent join, where one or two nodes refuse to service requests. It looooks like they might get stuck in JoinCluster for 20+ seconds; after which point my test gives up. This is on
and can be reproduced with jepsen 6b87441dbd1db2aca039226690f74b50ac78ef15 by running any test, e.g.
The text was updated successfully, but these errors were encountered: