Kubernetes Metrics Overhaul

[brancz]

Enhancement Description

• One-line enhancement description (can be used as a release note): Overhauling Kubernetes' metrics instrumentation to align with the Kubernetes' metrics with the Kubernetes Instrumentation Guidelines.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20181106-kubernetes-metrics-overhaul.md
• Primary contact (assignee): @brancz
• Responsible SIGs: sig-instrumentation
• Enhancement target (which target equals to which milestone):

This is a cleanup so there are no stability milestones involved, however, to not break hard immediately, SIG Instrumentation is doing its best effort to inform about these changes in various ways as follows:

• Metrics that violated the Kubernetes instrumentation guidelines have been marked deprecated in Kubernetes 1.14, and where possible new metrics were added that did follow the guidelines. [DONE]
  • Rename cadvisor metric labels to match instrumentation guidelines kubernetes#69099
  • Change latency bucket size for API server metrics kubernetes#67476
  • Change docker metrics to conform metrics guidelines kubernetes#72323
  • Change kubelet metrics to conform metrics guidelines kubernetes#72470
  • Fit RuntimeClass metrics to prometheus conventions kubernetes#73820
  • Prevent apiserver's metrics from accidental registration. kubernetes#63924
  • Expose kubelet health checks using new prometheus endpoint kubernetes#61369
  • Change scheduler metrics to conform metrics guidelines kubernetes#72332
  • Change proxy metrics to conform metrics guidelines kubernetes#72334
  • Fix admission metrics in true units kubernetes#72343
  • remove the deprecated admission metrics kubernetes#75279
  • Use prometheus conventions for workqueue metrics kubernetes#71300
  • convert latency/latencies in metrics name to duration kubernetes#74418
• Kubernetes 1.16 will remove the duplicate pod_name and container_name metric labels from cAdvisor metrics. For the 1.14 and 1.15 release all pod, pod_name, container and container_name were available as a grace period.
  • Drop deprecated cadvisor metric labels kubernetes#80376
• Kubernetes 1.17 will remove the in 1.14 marked as deprecated metrics. As a stretch goal, if the metrics stability framework is in place, then in Kubernetes 1.17 the metrics will only be turned off by default through the stability framework. Should this not be available, then the metrics will be removed.
  • Currently being discussed with each sig whose metrics are being removed.

1. Alpha release target 1.16

• Stability framework is in place with metric verification/validation running
  in CI.
• Metrics which are deprecated in the metrics overhaul are marked as deprecated,
  which can be overridden in a binary through a command line flag
• No metrics can be marked as stable.

2. Beta release target 1.17

• All previously marked deprecated metrics will be removed from the codebase.
• Metrics can be marked as stable.

3. Stable release target 1.18

• First release cycle in which stable metrics may be deprecated as per the new stability guidelines.

@logicalhan @serathius @piosz @ehashman

[kacole2]

@brancz the graduation criteria in the KEP needs to be more detailed on what make it move between stages. The current is very vague.

@lachie83 @mrbobbytables @rbitia @mariantalla @evillgenius75

[ehashman]

@kacole2 graduation criteria section is vague because this KEP is basically a collection of individual tasks that are graduated once completed.

As @brancz summarized in the first comment, everything has landed except for the full deprecation of the inconsistent labels, which is proposed for the 1.16 release, and removal of deprecated metrics targeted for 1.17. Once those are complete the KEP will be fully implemented. Does it need to be updated to say as much?

[kacole2]

/milestone v1.16
/stage alpha

[kacole2]

@brancz @ehashman is there a certain stage this should be labeled? I added alpha, but not sure if this is beta or moving to GA.

[brancz]

This is “just” a large scale cleanup that spans multiple releases, there’s not really any stability level around it. Not sure how to answer that question.

[brancz]

Talked to some folks, we're merging #1209 into this issue as the umbrella issue (they're actually already related as the current issue description says, we'll be updating the issue and comment here again).

[mrbobbytables]

This is somewhat of a personal opinion but I'd consider this "beta" with the plan to fully deprecate in 1.17 as "stable".

The alpha/beta/stable designation doesn't always align with the effort being done =/ For these I tend to think of it like this:
Alpha == work just beginning, long road ahead -- possibly several releases.
Beta == work in progress, most effort being done here and may span multiple cycles.
Stable == work wrapping up.

[kacole2]

@brancz it looks like there is some disconnect over at #1209. Can we find a time to gather all relevant parties to get this sorted out?

@lachie83

[brancz]

Sorry for taking a bit. I edited the issue to reflect alpha/beta/stable timelines and tasks.

[simplytunde]

@brancz I am 1.16 Doc Lead. We need a placeholder PR against k/website(dev-1.16 branch) for this enhancement before Friday, Aug 23rd. Let me know how I can help to make this happen or if doc is not required.

[ehashman]

I'm happy to jump on the doc PR if needed this week, where do the docs need to be updated @simplytunde? Do they just need to reflect the timeline/information included in the description of this issue?

[simplytunde]

@ehashman I do not have enough context on this to make decision on where/what docs needs to be updated. Lets bring it up on sig-instrumentation.

[kacole2]

@ehashman @brancz code freeze for 1.16 is on Thursday 8/29. Looks like all PRs listed have been merged for Alpha. If there are any more that need to be tracked, please let me know!

[ehashman]

@kacole2 we have one more PR coming for showing hidden metrics (as defined in the metrics stability KEP), per

Metrics which are deprecated in the metrics overhaul are marked as deprecated, which can be overridden in a binary through a command line flag

I'm working on that right now, should be able to have it up before code freeze. I think everything else is merged.

Edit: WIP PR link: kubernetes/kubernetes#81970

[logicalhan]

These also need to be merged:

1. Kubelet migration
2. Apiserver migration
3. Controller-manager migration
4. Kube-proxy migration
5. Scheduler migration

[RainbowMango]

@brancz @ehashman @logicalhan
I'd like to join the task (metric overhaul, stability, validation, etc).

Kubernetes 1.17 will remove the in 1.14 marked as deprecated metrics. As a stretch goal, if the metrics stability framework is in place, then in Kubernetes 1.17 the metrics will only be turned off by default through the stability framework. Should this not be available, then the metrics will be removed.

I guess we can start this task after 1.16 release. Where can I find the list of deprecated metrics?

[RainbowMango]

/assign

[brancz]

There won’t be removals as the framework components landed in 1.16 and flags are in progress. That means they’ll just be turned off by default for 1.17 and only truly removed in 1.18.

I would recommend to join the sig instrumentation slack channel and or sig meetings to get involved! :)

[RainbowMango]

Yeah, I know the metrics stability framework is in place now.
Thanks @brancz , I will try to join sig meeting.

[mrbobbytables]

Hey there @brancz @ehashman -- 1.17 Enhancements lead here. I know it's still kind of fuzzy what each stage defines 😬 but I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

Thanks!

/milestone clear

[jeremyrickard]

@RainbowMango @brancz could we please have the KEP for this updated with Test Plan info? It looks like we didn't do that in the 1.17 time frame and we should have. I'm going to remove this from the milestone for now, but you can file an exception request and we can add this back in. The KEP just needs to have test data added to it.

[jeremyrickard]

/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

Hi @RainbowMango -- 1.19 Enhancements Lead here, I wanted to check in if you think this enhancement would graduate in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

---

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

[RainbowMango]

• The KEP PR must be merged in an implementable state
• The KEP must have test plans
• The KEP must have graduation criteria.

@palnabarun Seems the enhancement need to supply test plans. What exactly is it? Is there any document about this?

[palnabarun]

Hi @RainbowMango, thank you for the update.

For the test plans, you can have a look at this KEP template for the exact requirement: https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/NNNN-kep-template/README.md

Also, one quick question, which graduation stage would you be targeting in 1.19?

[palnabarun]

@RainbowMango -- pinging back as a reminder of the above. 🙂

[palnabarun]

Hi @RainbowMango,

Tomorrow, Tuesday May 19 EOD Pacific Time is Enhancements Freeze

Will this enhancement be part of the 1.19 release cycle?

[RainbowMango]

Will this enhancement be part of the 1.19 release cycle?

The legacy changes of this KEP will not introduce a user-facing change, So, I guess you can ignore this KEP.

[palnabarun]

@RainbowMango -- Thanks for the update. I have updated the tracking sheet accordingly. 👍

[ehashman]

/assign

[ehashman]

This work was basically completed in the 1.17 release. I'll update the KEP as needed in order to close out this issue.