Add querier.ingester-query-max-attempts to retry on partial data.

[justinjung04]

What this PR does:
In #6526, new configs query_partial_data and rules_partial_data were added which allows tenants to receive 2xx with a warning message when the data accuracy is relatively high in zone-aware setting.

This PR adds retry logic in querier getting data from ingesters, retrying the requests if the response is partial data. The new configuration, querier.ingester-query-max-attempts, allows ingester queries to be retried. Default is set to 1.

Which issue(s) this PR fixes:
n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[danielblando]

LGTM

[SungJin1212]

LGTM!

[yeya24]

Maybe we should document clearly and provide some suggestions on what's the recommended set up for this retry value.

To me this flag is kind of overlapped with partial data and I am not sure how much the retry helps for most of the usecase.
We know it is unlikely to miss series so we return partial data with 4XX. The retry may succeed but given we only wait short period time and Ingester queries are usually within ms, I am unsure if it is worth it to retry more espeically if Ingesters are high load or ongoing deployment

[justinjung04]

Do you think the following description for the config is sufficient?

The maximum number of times we attempt fetching data from ingesters for retryable errors (ex. partial data returned).

What I noticed with partial data is that the status code is 200 (such that the result is actually returned to the customer), but it throws a warning along with the results. So, when there is a transient network issue between a querier and ingesters, the customer could get partial data response instead of the query path retrying to get the full response (current retry logic from query frontend only reties 5xx responses, so partial data response does not get retried at all).

[yeya24]

Thanks. We can merge after fixing the changelog conflict