Optimize the process of syncing rules in Rulers

[rapphil]

Is your feature request related to a problem? Please describe.
Not really.

Describe the solution you'd like
SyncRules in Rulers is the process of configuring Rulers with the rules that are configured per tenant in cortex so that each Ruler loads the rule groups that they are responsible for. This process can happen periodically (with a configurable interval) or due to ring change events (when Rulers leave or join the ring). During a deployment, several ring change events are expected. We can consider to be a waste of resources to perform a periodic SyncRules right after a ring change has happened.

Therefore this issue proposes that ring change events restart the timer used for periodic SyncRules so that we reduce the number of SyncRules.

Describe alternatives you've considered
Can we aggregate the ring change events in buckets so that we minimize the number of SyncRules that are performed, in case we have a deployment with several instances restarting at the same time? I.e.: we set a upper bound to the number of SyncRules that can happen per period of time.

Additional context
Here is the for loop that we want to optimize:

cortex/pkg/ruler/ruler.go

Lines 670 to 687 in 8a46d20

	for {
	select {
	case <-ctx.Done():
	return nil
	case <-tick.C:
	r.syncRules(ctx, rulerSyncReasonPeriodic)
	case <-ringTickerChan:
	// We ignore the error because in case of error it will return an empty
	// replication set which we use to compare with the previous state.
	currRingState, _ := r.ring.GetAllHealthy(RingOp)
	if ring.HasReplicationSetChanged(ringLastState, currRingState) {
	ringLastState = currRingState
	r.syncRules(ctx, rulerSyncReasonRingChange)
	}
	case err := <-r.subservicesWatcher.Chan():
	return errors.Wrap(err, "ruler subservice failed")
	}

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.