Skip to content

Change 'too many labels' error message to play nicely with Prometheus logs #3718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

pracucci
Copy link
Contributor

@pracucci pracucci commented Jan 20, 2021

What this PR does:
Prometheus truncates remote write errors displayed in logs at 512 bytes (was increased from 256 to 512 in this PR). An user privately reported me that wasn't understanding why some series were rejected and Prometheus was logging the following:

ts=2021-01-19T14:30:48.352Z caller=dedupe.go:112 component=remote level=error remote_name=45f6d9 url=https://prometheus-us-central1.grafana.net/api/prom/push msg="non-recoverable error" count=99 err="server returned HTTP status 400 Bad Request: sample for 'istio_response_bytes_bucket{app_kubernetes_io_instance=\"infrabin\", app_kubernetes_io_name=\"infrabin\", connection_security_policy=\"unknown\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"infra"

Since the log is truncated, it's not clear the problem is "too many labels". In this PR I'm proposing to change the format of the "too many labels" error message to move the series at the end.

I took a quick look at other validation errors and I think this the only one suffering this issue.

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

… logs

Signed-off-by: Marco Pracucci <marco@pracucci.com>
Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@bboreham
Copy link
Contributor

Related: #1582

Copy link
Contributor

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@pracucci
Copy link
Contributor Author

Related: #1582

I'm very bad with the GitHub search apparently. I was looking for that issue and I haven't found it 😞

@pracucci pracucci merged commit e6033ea into cortexproject:master Jan 20, 2021
@pracucci pracucci deleted the fix-error-messages-to-be-more-readable-in-prometheus branch January 20, 2021 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants