Skip to content

Scatter query should fail if serving shards incompletely cover the keyspace #4968

Open
@enisoc

Description

Overview of the Issue

A scatter query executed while the set of serving shards does not completely cover the keyspace reports success. It should fail instead since complete results cannot be guaranteed while in this broken state, which can happen due to a topology misconfiguration or bug (e.g. #4969).

Reproduction Steps

  1. Deploy a keyspace with 2 shards.

  2. Create a table and insert some rows.

  3. Execute a scatter query like select * from table; and confirm all expected rows are returned.

  4. Simulate topology misconfiguration or bug by forcing a bad state: remove one shard from the serving set, leaving the set incomplete.

    For example:

    vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/-80 master
    vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/-80 replica
  5. Execute the scatter query again. It succeeds, but some rows may be silently missing.

  6. Restore correct topology:

    vtctlclient -server <server> RebuildKeyspaceGraph keyspace
  7. Execute the scatter query again. All rows are back.

  8. Simulate topology misconfiguration again by removing the other shard:

    vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/80- master
    vtctlclient -server <server> UpdateSrvKeyspacePartition --remove keyspace/80- replica
  9. Execute the scatter query again. It succeeds, but some rows may be silently missing. The missing rows are exactly those that were returned when the other shard was the one serving.

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions