Dynamic copy management to solve the problem of unbalanced load(mem) of Ingester nodes

[wgliang]

Is your feature request related to a problem? Please describe.

The current copy of Cortex is implemented through consistent hashing, and the location of the copy (Inageter node) is determined at the beginning of the metric data writing. In the case that some metric samples are relatively large, the load of different Ingester nodes will vary greatly.

Describe the solution you'd like
Whether we can implement dynamic replicas (shards), we can dynamically schedule between different Ingester nodes based on size, time slice, load, etc.

Describe alternatives you've considered

Additional context

[friedrichg]

I am curious if you are experiencing this with shard-by-all-labels set to true

# Distribute samples based on all labels, as opposed to solely by user and
# metric name.
# CLI flag: -distributor.shard-by-all-labels
[shard_by_all_labels: <boolean> | default = false]

[jeromeinsf]

which configuration are you using for https://cortexmetrics.io/docs/configuration/arguments/#distributor ?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[yeya24]

#5855 could somehow help with the problem with uneven load.
But dynamically schedule between different Ingester nodes is still a useful feature.