Skip to content

TSDB: fix user metrics when using blocks storage #1990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jan 20, 2020
Merged

TSDB: fix user metrics when using blocks storage #1990

merged 9 commits into from
Jan 20, 2020

Conversation

pstibrany
Copy link
Contributor

@pstibrany pstibrany commented Jan 16, 2020

When using TSDB, we get per-user created and removed series from TSDB metrics.

This affects cortex_ingester_memory_series_created_total and cortex_ingester_memory_series_removed_total.

This PR also renames shipperMetrics to tsdbMetrics, and registers the same registry to TSDB and TSDB shipper component.

Registration of prometheus registry now happens only after TSDB initialization is finished, in order to fix a panic where metrics were collected while TSDB was still initialized.

This PR builds on PR #1983, so "Files changed" also shows changes from there. I'll rebase once (if) #1983 is merged. Update: now rebased.

@pstibrany pstibrany requested a review from pracucci January 16, 2020 11:14
… metrics.

This affects cortex_ingester_memory_series_created_total and cortex_ingester_memory_series_removed_total.

This change also renames previous shipperMetrics to tsdbMetrics, and registers the same registry
to TSDB and TSDB shipper component.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
This fixes problem with collection occuring when TSDB already registered metrics,
but wasn't otherwise fully initialized yet, and it crashed.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent job @pstibrany! I left few minor comments.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
@pstibrany
Copy link
Contributor Author

Thanks for review @pracucci, good points. I've addressed all your feedback.

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pstibrany! LGTM

Copy link
Contributor

@gouthamve gouthamve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@pracucci pracucci merged commit efb54b3 into cortexproject:master Jan 20, 2020
@pstibrany pstibrany deleted the tsdb-user-metrics branch January 21, 2020 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants