Kubernetes Metrics Overhaul

[brancz]

Enhancement Description

• One-line enhancement description (can be used as a release note): Overhauling Kubernetes' metrics instrumentation to align with the Kubernetes' metrics with the Kubernetes Instrumentation Guidelines.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20181106-kubernetes-metrics-overhaul.md
• Primary contact (assignee): @brancz
• Responsible SIGs: sig-instrumentation
• Enhancement target (which target equals to which milestone):

This is a cleanup so there are no stability milestones involved, however, to not break hard immediately, SIG Instrumentation is doing its best effort to inform about these changes in various ways as follows:

• Metrics that violated the Kubernetes instrumentation guidelines have been marked deprecated in Kubernetes 1.14, and where possible new metrics were added that did follow the guidelines. [DONE]
  • Rename cadvisor metric labels to match instrumentation guidelines kubernetes#69099
  • Change latency bucket size for API server metrics kubernetes#67476
  • Change docker metrics to conform metrics guidelines kubernetes#72323
  • Change kubelet metrics to conform metrics guidelines kubernetes#72470
  • Fit RuntimeClass metrics to prometheus conventions kubernetes#73820
  • Prevent apiserver's metrics from accidental registration. kubernetes#63924
  • Expose kubelet health checks using new prometheus endpoint kubernetes#61369
  • Change scheduler metrics to conform metrics guidelines kubernetes#72332
  • Change proxy metrics to conform metrics guidelines kubernetes#72334
  • Fix admission metrics in true units kubernetes#72343
  • remove the deprecated admission metrics kubernetes#75279
  • Use prometheus conventions for workqueue metrics kubernetes#71300
  • convert latency/latencies in metrics name to duration kubernetes#74418
• Kubernetes 1.16 will remove the duplicate pod_name and container_name metric labels from cAdvisor metrics. For the 1.14 and 1.15 release all pod, pod_name, container and container_name were available as a grace period.
  • Drop deprecated cadvisor metric labels kubernetes#80376
• Kubernetes 1.17 will remove the in 1.14 marked as deprecated metrics. As a stretch goal, if the metrics stability framework is in place, then in Kubernetes 1.17 the metrics will only be turned off by default through the stability framework. Should this not be available, then the metrics will be removed.
  • Currently being discussed with each sig whose metrics are being removed.

1. Alpha release target 1.16

• Stability framework is in place with metric verification/validation running
  in CI.
• Metrics which are deprecated in the metrics overhaul are marked as deprecated,
  which can be overridden in a binary through a command line flag
• No metrics can be marked as stable.

2. Beta release target 1.17

• All previously marked deprecated metrics will be removed from the codebase.
• Metrics can be marked as stable.

3. Stable release target 1.18

• First release cycle in which stable metrics may be deprecated as per the new stability guidelines.

@logicalhan @serathius @piosz @ehashman

[kacole2]

@brancz the graduation criteria in the KEP needs to be more detailed on what make it move between stages. The current is very vague.

@lachie83 @mrbobbytables @rbitia @mariantalla @evillgenius75

[ehashman]

@kacole2 graduation criteria section is vague because this KEP is basically a collection of individual tasks that are graduated once completed.

As @brancz summarized in the first comment, everything has landed except for the full deprecation of the inconsistent labels, which is proposed for the 1.16 release, and removal of deprecated metrics targeted for 1.17. Once those are complete the KEP will be fully implemented. Does it need to be updated to say as much?

[kacole2]

/milestone v1.16
/stage alpha

[kacole2]

@brancz @ehashman is there a certain stage this should be labeled? I added alpha, but not sure if this is beta or moving to GA.

[brancz]

This is “just” a large scale cleanup that spans multiple releases, there’s not really any stability level around it. Not sure how to answer that question.

[brancz]

Talked to some folks, we're merging #1209 into this issue as the umbrella issue (they're actually already related as the current issue description says, we'll be updating the issue and comment here again).

[mrbobbytables]

This is somewhat of a personal opinion but I'd consider this "beta" with the plan to fully deprecate in 1.17 as "stable".

The alpha/beta/stable designation doesn't always align with the effort being done =/ For these I tend to think of it like this:
Alpha == work just beginning, long road ahead -- possibly several releases.
Beta == work in progress, most effort being done here and may span multiple cycles.
Stable == work wrapping up.