Randomly distribute queries across store-gateways

[pracucci]

What this PR does:
Currently, when the querier needs to query a block it always pick the 1st store-gateway holding that block (1st is based on the tokens in the ring). This may cause to have hot store-gateways instead of balancing the traffic across all store-gateways holding a specific block (eg. 3 when the replication factor is set to 3).

This PR fixes it, randomly selecting a store-gateway.

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[jtlisi]

LGTM

[stevesg]

Looks good, only one question:

Am I right in thinking that whilst two balancing policies are defined, only randomLoadBalancing is used? (though the noLoadBalancing policy is used in the unit test).

I might be missing some context on the issue - is there a plan to allow configuring which policy to use, or add more policies? If it's always going to be randomLoadBalancing, then the configurable policies could be omitted, saving a little glue code.

[pracucci]

Am I right in thinking that whilst two balancing policies are defined, only randomLoadBalancing is used? (though the noLoadBalancing policy is used in the unit test).

Correct. Introducing randomness in unit tests would require to change the unit tests in a way what would lower the confidence on the algorithm, reason why I preferred to not have it.

I might be missing some context on the issue - is there a plan to allow configuring which policy to use, or add more policies? If it's always going to be randomLoadBalancing, then the configurable policies could be omitted, saving a little glue code.

I think we should keep randomLoadBalancing for real usage, without making it configurable. We already did it in blocksStoreBalancedSet (used when store-gateway sharding is disabled) while we missed to do it in blocksStoreReplicationSet (used when store-gateway sharding is enabled).