Lifecycle management for Prometheus metrics

[hendrikmakait]

As mentioned in #7374 (comment), we do not have any lifecycle management for Prometheus metrics. On the one hand, most metrics are under active development, so additions or breaking changes are to be expected (e.g., #7368). On the other hand, some metrics have been around for quite a while and might be used in a production setting but might not correspond to guidelines we establish and require changes (e.g., #7374).

This raises the following questions:

• How do we perform lifecycle management for Prometheus metrics?
• Are there any best practices we should follow?
• What are the expectations toward stability?
• Should we version metrics (and how would we)?

[crusaderky]

cc @jacobtomlinson @fjetter

[mrocklin]

Personally I'd be ok with a hard break if we come up with a structure that we're happy living with for a while.

Prometheus usage is, as far as I can tell, sporadic today. If a hard break helps us to make it useful and widespread then I would be ok with that.

cc also @jacobtomlinson

[jacobtomlinson]

I agree that I'm pretty happy to break things now, but I'd like to avoid breaking things again in the future if prometheus usage is going to increase.

Versioning somehow would be nice, but also adds a bunch of overhead. I've not seen this done in practice.

[dchudz]

@gjoseph92 summarizing our in-person discussion:

I think you're planning on working on this soon, with an approach that shouldn't take more than a week or so of engineering effort.

I also think you'll circulate a design/plan this with, in order to get buy-in from Florian etc.

Right?

[gjoseph92]

I'll be working on #7364, not this issue. I won't be changing any metrics in the process.

[dchudz]

oh right, sorry Gabe!