Fix ring tokens sorting regression introduced in #3601

[pracucci]

What this PR does:
I've introduced a regression in #3601 assuming that tokens are always sorted in the ring. Actually, recent versions of Cortex guarantee it but if your Cortex cluster has been created long time ago and tokens have been transferred from a leaving ingester to a joining one over and over they are still unsorted in the ring.

In this PR:

• Guarantee sorted tokens when a JOINING ingester claim tokems from a LEAVING one
• Guarantee sorted tokens when an ingester read tokens from file
• Rollback tokens sorting in Desc.GetTokens() and Desc.getTokensByZone() which was removed in Optimised Ring.ShuffleShard() and disabled subring cache in store-gateway, ruler and compactor #3601 (required to cover the time range during which ingesters of the new Cortex version are rolling out)
• No CHANGELOG because I expect this to be cherry picked in release-1.7

My plan is:

• Merge this PR
• Cherry pick the fix into release-1.7
• Open another PR to remove the sorting from Desc.GetTokens() and Desc.getTokensByZone() adding a CHANGELOG entry to require upgrade to Cortex 1.7 before upgrading to future version Cortex 1.8

The performance impact for the sorting re-introduced in Desc.GetTokens() and Desc.getTokensByZone() is pretty low:

benchmark                                   old ns/op     new ns/op     delta
BenchmarkRing_ShuffleShard_512Tokens-12     399147        427036        +6.99%
BenchmarkRing_Get-12                        781           787           +0.77%

Ring.Get() is not expected to be impacted, the little diff is just noise.

Which issue(s) this PR fixes:
Fixes #3769

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[bboreham]

After this change, are tokens sorted in disk file and in KV store?
That would be a good thing to ensure, because we could then remove sort-on-load after a couple of versions.

[bboreham]

Looks ok

[pracucci]

After this change, are tokens sorted in disk file and in KV store?

Yes, they should (unless I'm missing anything).

because we could then remove sort-on-load after a couple of versions

My idea (in the PR description) was to remove it from the next version and require to upgrade to Cortex 1.7 before 1.8. What do you think? It would allow us to immediately remove the sort-on-load from master.

[pstibrany]

LGTM.

(Nit: when posting benchmarks, please include full output from benchstat, including statistical significance, otherwise they are not saying much).

[pracucci]

🤞 Let's go and hope (even if hope is never a strategy)