Improve TSDB blocks sync

[pracucci]

What this PR does:
Currently, there's an hardcoded 30s as blocks sync interval (TSDB storage). It's quite a lot. I believe we don't even need such an high-frequency sync interval (at least, not as default) so I'm proposing to allow to configure it and set an higher default (Thanos' default is 3m, I've set 5m which should be more than enough for the Cortex use case).

I've also received an error from the linter because the rununtil.Repeat() error wasn't checked, so I took the change to refactor it (moved to UserStore since it belongs to it), renamed cortex_querier_sync_seconds to cortex_querier_blocks_sync_seconds and customized the buckets: default values are too low for this operation in a medium/large cluster.

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

LGTM

[pracucci]

I've received an error from the linter because the rununtil.Repeat() error wasn't checked, so I took the change to refactor it (moved to UserStore since it belongs to it), renamed cortex_querier_sync_seconds to cortex_querier_blocks_sync_seconds and customized the buckets: default values are too low for this operation in a medium/large cluster.

@pstibrany May you re-review it, please?
@thorfour What's your take?

[pstibrany]

Looks good to me.

Your change made me to look at more code, and one concern I have (but it's not new in this PR) is that when using target=All, initial sync of the querier will block entire startup of the single-binary Cortex. Is that right?

[pracucci]

when using target=All, initial sync of the querier will block entire startup of the single-binary Cortex. Is that right?

@pstibrany That's correct. On the other end, you can't query Cortex until the initial sync is done (but you could ingest), so not sure what's the best way to handle it.

[pstibrany]

@pstibrany That's correct. On the other end, you can't query Cortex until the initial sync is done (but you could ingest), so not sure what's the best way to handle it.

One way to handle it would be to reject query requests until querier is finished with the initial sync, but not to block the startup of Cortex. Let's fix it when we get there.

[pracucci]

@pstibrany @thorfour I've re-iterated on this PR based on your feedback (allow to disable sync, do not synch twice at startup, fix race condition) plus a couple of other changes (max concurrency when synching multi stores, track timing for initial sync too). May you re-review it, please?

[pstibrany]

Looks good, thanks for also fixing concurrency issues!

[pracucci]

@pstibrany Thanks for your review. I've addressed the comments.

[thorfour]

Changes look good to me thanks!

[jtlisi]

LGTM