-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optionally print OM created lines #1408
base: main
Are you sure you want to change the base?
Conversation
c26d312
to
b7d00b2
Compare
e930252
to
240345e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! One minor nit, otherwise well done!
33ed3e6
to
e9c6f77
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Epic, proposed some improvements, thanks!
prometheus/promhttp/http.go
Outdated
// possible options during content negotiation. Note that Prometheus | ||
// 2.5.0+ will negotiate OpenMetrics as first priority. OpenMetrics is | ||
// the only way to transmit exemplars. However, the move to OpenMetrics | ||
// is not completely transparent. Most notably, the values of "quantile" | ||
// labels of Summaries and "le" labels of Histograms are formatted with | ||
// a trailing ".0" if they would otherwise look like integer numbers | ||
// (which changes the identity of the resulting series on the Prometheus | ||
// server). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// possible options during content negotiation. Note that Prometheus | |
// 2.5.0+ will negotiate OpenMetrics as first priority. OpenMetrics is | |
// the only way to transmit exemplars. However, the move to OpenMetrics | |
// is not completely transparent. Most notably, the values of "quantile" | |
// labels of Summaries and "le" labels of Histograms are formatted with | |
// a trailing ".0" if they would otherwise look like integer numbers | |
// (which changes the identity of the resulting series on the Prometheus | |
// server). | |
// possible options during content negotiation. | |
// | |
// Note that Prometheus 2.5.0+ might negotiate OpenMetrics Text format | |
// as first priority unless user uses custom scrape protocol prioritization or | |
// histograms feature is enabled (then Prometheus proto format is prioritized, | |
// which client_golang supports). | |
// | |
// The main reason for enabling OpenMetrics Text was to enable exemplars, | |
// but Prometheus proto has that as feature as well. | |
// | |
// However, the move to OpenMetrics is not completely transparent. Most notably, | |
// the values of "quantile" labels of Summaries and "le" labels of Histograms are | |
// formatted with a trailing ".0" if they would otherwise look like integer numbers | |
// (which changes the identity of the resulting series on the Prometheus | |
// server). | |
// | |
// See other options in OpenMetricsOptions to learn how to enable some special | |
// features e.g. potentially dangerous created timestamp series. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// The main reason for enabling OpenMetrics Text was to enable exemplars,
// but Prometheus proto has that as feature as well.
this part seems a bit out of place, not adding much to the comment 🤔, is it ok to not add this part?
prometheus/promhttp/http.go
Outdated
// is recommended to enable the feature flag in Prometheus, otherwise enabling | ||
// _created lines will result in increased cardinality and no improvements | ||
// in reset detection. | ||
EnableOpenMetricsCreatedMetrics bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
EnableOpenMetricsCreatedMetrics bool | |
EnableCreatedSeries bool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmmmm, future versions of OM might not expose them as a series. What do you think about EnableCreatedTimestamps
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If they are not expanded as series they are not a problem and will be fine by default (sorry for missing this comment)
e9c6f77
to
81909c5
Compare
We need prometheus/common v0.49.0 for this feature, but this version of prometheus/common requires go 1.21. Therefore, the PR is blocked until we drop support for go 1.20 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
81909c5
to
65df2d7
Compare
After quite some work to unblock this, this is finally ready :) Do you want to review again or should I just merge? |
Now that prometheus/prometheus#14356 was merged, should we revive this? :) |
65df2d7
to
eb0f4ef
Compare
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
eb0f4ef
to
b51b530
Compare
Since we have approvals from the past, I'll take the liberty to merge this PR next week :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's timebox this, but wanted to try last more time to consider an improvement to the config structure, otherwise LGTM!
@@ -418,6 +419,43 @@ type HandlerOpts struct { | |||
ProcessStartTime time.Time | |||
} | |||
|
|||
type OpenMetricsOptions struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm.. want to play with different ideas for this options first 🤔 Even keeping EnableOpenMetrics
and then OpenMetricsOptions.AddV1CreatedSeries
might a better choice? 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm.. want to play with different ideas for this options first 🤔 Even keeping
EnableOpenMetrics
and thenOpenMetricsOptions.AddV1CreatedSeries
might a better choice? 🤔
Interesting! I assume that this suggestion comes from not liking the idea of mentioning that "Created" is a "Series", this is something we'd like to change in the next version to something closer to metadata.
What do you think of EnableCreatedTimestamps
instead?
I think mentioning versions on fields is not the best, in the future I can see a version
field that controls how each feature is exposed instead of AddV1Created..
, AddV2Created...
and then handling mutually exclusiveness
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh lol, I just realized I suggested what's already in the code 🤦
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand the confusion, here is why: #1408 (comment)
We will likely not have "enable X Y Z" unless protocol will make a challenging decision like with CT being a series. So ideally we only have this "unsafe 1.0 Ct as series" flag for now, but do CT by default if they are safe to use (they don't increase 50-30% of Prometheus memory for no reason). (:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might have some more flags in future for units too though, so some plan for those would be nice. For CT as series that an OpenMetrics option, but for units (e.g. AutogenerateUnits)... it's kind of general option? maybe not even in promhttp but in registry or something?
Since we now little and we already make it complex I would vote for keeping EnableOpenMetrics
and adding EmitOpenMetricsTextCreatedSamples bool
would be perfect for now. Samples not series as this is how it's referenced in OpenMetrics https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#counter-1
Simple, and effective, then we probably should fail or ignore this if EnableOpenMetrics is not true.
WDYT
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On a loosely related note, I would agree on a general option for units. Happy to resume the work on that one whenever (if ever) needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we add summaries and counters in the example too?
This PR adds an extra option to print _created lines when OpenMetrics is the chosen exposition format. It's implemented following the conditional option added in prometheus/common#504.