Description
Overview of the Issue
A scatter query executed while the set of serving shards does not completely cover the keyspace reports success. It should fail instead since complete results cannot be guaranteed while in this broken state, which can happen due to a topology misconfiguration or bug (e.g. #4969).
Reproduction Steps
-
Deploy a keyspace with 2 shards.
-
Create a table and insert some rows.
-
Execute a scatter query like
select * from table;
and confirm all expected rows are returned. -
Simulate topology misconfiguration or bug by forcing a bad state: remove one shard from the serving set, leaving the set incomplete.
For example:
vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/-80 master vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/-80 replica
-
Execute the scatter query again. It succeeds, but some rows may be silently missing.
-
Restore correct topology:
vtctlclient -server <server> RebuildKeyspaceGraph keyspace
-
Execute the scatter query again. All rows are back.
-
Simulate topology misconfiguration again by removing the other shard:
vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/80- master vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/80- replica
-
Execute the scatter query again. It succeeds, but some rows may be silently missing. The missing rows are exactly those that were returned when the other shard was the one serving.