[DocDB] system.local is empty on a select count in user table under nemesis

[pilshchikov]

Jira Link: DB-12818

Description

Case:

1. Run xcluster with 3 nodes, RF=3, 2.23.0.0-b710
2. In background running some nemesis - restart master process in a moment of failure
3. Execute select count(*) on some user table

06:55:21.141 [Thread-4] INFO cql: select count(*) as aggr from replication_7ee7_5931 where uuid = 'e71e10c3-3d00-4a5b-ab90-7eaf524d93ac'
06:55:23.056 [Thread-5] DEBUG cql: connecting to 172.151.25.234:9042,172.151.21.139:9042,172.151.30.86:9042
java.lang.IllegalStateException: system.local is empty on ip-172-151-25-234.us-west-2.compute.internal/172.151.25.234:9042, this should not happen
	at com.datastax.driver.core.ControlConnection.refreshNodeListAndTokenMap(ControlConnection.java:794)
	at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:342)
	at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:258)
	at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:132)
	at com.datastax.driver.core.Cluster$Manager.negotiateProtocolVersionAndConnect(Cluster.java:1839)
	at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1752)
	at com.datastax.driver.core.Cluster.init(Cluster.java:216)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:389)
	at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:368)
	at com.datastax.driver.core.Cluster.connect(Cluster.java:313)
	at com.yugabyte.clientserver.Queries.getCassandraSession(Queries.java:147)
	at com.yugabyte.clientserver.Queries.executeYCql(Queries.java:269)
	at com.yugabyte.clientserver.Queries.execute(Queries.java:286)
	at com.yugabyte.clientserver.QueryHandler.runQuery(QueryHandler.java:150)
	at com.yugabyte.clientserver.QueryHandler.lambda$runQueryHandler$2(QueryHandler.java:179)
	at java.base/java.lang.Thread.run(Thread.java:1589)

Issue can be reproduced more easely:

1. Run some workload
2. Start select count in loop
3. Restart master/tserver processes
   Reproduced rate in this case in 100% cases
   All logs can be found in first JIRA comment

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.