Add network stats for Windows nodes and containers #74788

feiskyer · 2019-03-01T06:32:48Z

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Add network stats for Windows nodes and containers.

Which issue(s) this PR fixes:

Fixes #74101

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add network stats for Windows nodes and containers

/sig windows
/sig node
/assign @yujuhong @PatrickLang @michmike

feiskyer · 2019-03-01T06:33:32Z

/milestone v1.14

michmike · 2019-03-01T17:14:01Z

@benmoss, @aserdean can you PTAL

pkg/kubelet/stats/BUILD

pkg/kubelet/stats/cri_stats_provider.go

pkg/kubelet/winstats/network_stats.go

michmike · 2019-03-01T23:53:13Z

looks good overall, even though there is a lot of duplication of code that would have been better in an abstraction. i noted a small bug in there.

have you tested this and how does performance relate to the previous code?
thanks for all the work!

feiskyer · 2019-03-04T07:48:36Z

Fixed the bug now. PTAL.

have you tested this and how does performance relate to the previous code?

Yep, kubelet consumes much less CPU now (kubelet 1-4% from my cluster).

feiskyer · 2019-03-04T08:00:12Z

/retest

pkg/kubelet/stats/cri_stats_provider.go

benmoss · 2019-03-04T17:15:18Z

pkg/kubelet/stats/cri_stats_provider_others.go

+// listContainerNetworkStats returns the network stats of all the running containers.
+func (p *criStatsProvider) listContainerNetworkStats() (map[string]*statsapi.NetworkStats, error) {
+	// Always return nil for unsupported platforms.
+	return nil, nil


This is going to yield a nil pointer exception on unsupported platforms on cri_stats_provider.go:192. Not sure if it matters, but maybe this could return an error like https://github.com/kubernetes/kubernetes/blob/48def74e641f8dff50fcf7c33c2a412593e2fede/pkg/util/mount/exec_mount_unsupported.go ?

In such case, we still want to continue other stats logic in callers. So nil is returned here instead of errors.

The way the code is written right now, it's hard to tell whether nil is expected or not. @feiskyer is there any existing e2e test that can be modified to validate whether networkstats is present? Maybe the summary API test?

For Linx, there always have network stats, but for Windows, there doesn't before this PR. After this, both Windows and Linux containers would have network stats.

pkg/kubelet/winstats/network_stats.go

michmike · 2019-03-06T16:55:51Z

/lgtm

michmike · 2019-03-06T16:55:59Z

/approve

michmike · 2019-03-06T16:56:29Z

/assign @yujuhong

yujuhong · 2019-03-06T19:22:36Z

/hold
Reviewing and I will some questions (will post soon). Holding this PR to avoid it from being merged accidentally.

yujuhong · 2019-03-06T19:25:49Z

/test pull-kubernetes-e2e-containerd-gce

PatrickLang · 2019-03-06T19:46:46Z

I'm only seeing one network now.

    "network": {
      "time": "2019-03-06T19:42:34Z",
      "name": "e76e4a10-eth0",
      "rxBytes": 146563,
      "txBytes": 491768,
      "interfaces": [
        {
          "name": "e76e4a10-eth0",
          "rxBytes": 146563,
          "txBytes": 491768
        }
      ]
    },

That matches the adapter name from the Azure-CNI logs:

azure-vnet.log:704:2019/03/06 19:38:59 [cni-net] Creating endpoint e76e4a10-eth0.
azure-vnet.log:705:2019/03/06 19:38:59 [net] Creating endpoint &{Id:e76e4a10-eth0
ContainerID:e76e4a108a64edbac93f89f286262fea94221c3ecb1f7854909fc7ddd275e96d
NetNsPath:none IfName:eth0 SandboxKey: IfIndex:0 MacAddress:
DNS:{Suffix:default.svc.cluster.local Servers:[10.0.0.10 168.63.129.16]} ...

Looks good!

k8s-ci-robot · 2019-03-06T20:05:16Z

@feiskyer: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-containerd-gce	ee5fd0a9f4d1b797a72c9109b63acef2e1bfbbbe	link	`/test pull-kubernetes-e2e-containerd-gce`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

yujuhong

The CRI stats path is not tested by dockershim+linux AFAIK. I can't get the presubmit containerd job to run right now, so we'll need to check if post-submit CI will be affected by this change.

Other than that, this mostly looks good with questions/comments.

The summary api test is in the node e2e suite (cannot be run by windows yet). I think we should port the test over to a regular e2e, and do some sanity check to see what metrics are present. I don't think it should block this PR, but that should be addressed afterwards.

pkg/kubelet/stats/cri_stats_provider_others.go

pkg/kubelet/stats/cri_stats_provider_windows.go

pkg/kubelet/stats/cri_stats_provider.go

yujuhong · 2019-03-06T19:51:27Z

pkg/kubelet/stats/cri_stats_provider_others.go

+// listContainerNetworkStats returns the network stats of all the running containers.
+func (p *criStatsProvider) listContainerNetworkStats() (map[string]*statsapi.NetworkStats, error) {
+	// Always return nil for unsupported platforms.
+	return nil, nil


The way the code is written right now, it's hard to tell whether nil is expected or not. @feiskyer is there any existing e2e test that can be modified to validate whether networkstats is present? Maybe the summary API test?

pkg/kubelet/stats/helper.go

pkg/kubelet/winstats/network_stats.go

yujuhong · 2019-03-06T20:03:03Z

pkg/kubelet/winstats/perfcounter_nodestats.go

 	go wait.Forever(func() {
-		p.collectMetricsData(cpuCounter, memWorkingSetCounter, memCommittedBytesCounter)
+		p.collectMetricsData(cpuCounter, memWorkingSetCounter, memCommittedBytesCounter, networkAdapterCounter)


Is the perfCounterUpdatePeriod (1s) still reasonable after adding more metrics?

1s should still work. And for network stats, there are some metrics per second. To ensure the cumulative data correctly, 1s should be used.

PatrickLang · 2019-03-07T01:24:58Z

/retest pull-kubernetes-e2e-containerd-gce

yujuhong · 2019-03-07T01:29:44Z

pull-kubernetes-e2e-containerd-gce

We haven't triggered this job for a while. Seems like it doesn't work anymore. We'll watch the post-submit CI instead.

feiskyer · 2019-03-07T01:29:49Z

The summary api test is in the node e2e suite (cannot be run by windows yet). I think we should port the test over to a regular e2e, and do some sanity check to see what metrics are present. I don't think it should block this PR, but that should be addressed afterwards.

@PatrickLang Could you open an issue for tracking this? We should enable node e2e tests for Windows.

PatrickLang · 2019-03-07T06:10:52Z

/LGTM
@yujuhong can you approve?

yujuhong · 2019-03-07T19:14:44Z

/hold cancel
/lgtm
/approve

k8s-ci-robot · 2019-03-07T19:15:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, michmike, yujuhong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/OWNERS~~ [yujuhong]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot assigned michmike Mar 1, 2019

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Mar 1, 2019

k8s-ci-robot assigned PatrickLang and yujuhong Mar 1, 2019

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Mar 1, 2019

feiskyer mentioned this pull request Mar 1, 2019

kubelet on windows consuming 100% cpu running multiple powershell process withGet-NetAdapterStatistics #74101

Closed

feiskyer added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Mar 1, 2019

k8s-ci-robot added this to the v1.14 milestone Mar 1, 2019

k8s-ci-robot added the area/kubelet label Mar 1, 2019

k8s-ci-robot requested review from dims and tallclair March 1, 2019 06:33

michmike reviewed Mar 1, 2019

View reviewed changes

pkg/kubelet/stats/BUILD Outdated Show resolved Hide resolved

michmike reviewed Mar 1, 2019

View reviewed changes

pkg/kubelet/stats/cri_stats_provider.go Show resolved Hide resolved

michmike reviewed Mar 1, 2019

View reviewed changes

pkg/kubelet/winstats/network_stats.go Outdated Show resolved Hide resolved

feiskyer force-pushed the windows-network-stats branch from 575609f to fd5a7a5 Compare March 4, 2019 07:28

benmoss reviewed Mar 4, 2019

View reviewed changes

pkg/kubelet/stats/cri_stats_provider.go Show resolved Hide resolved

benmoss reviewed Mar 4, 2019

View reviewed changes

pkg/kubelet/winstats/network_stats.go Outdated Show resolved Hide resolved

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Mar 6, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 6, 2019

yujuhong reviewed Mar 6, 2019

View reviewed changes

feiskyer added 3 commits March 7, 2019 09:54

Add network stats for Windows containers

84a7f48

Add network stats for node interfaces

1220025

Run hack/update-bazel.sh

d690037

feiskyer force-pushed the windows-network-stats branch from ee5fd0a to d690037 Compare March 7, 2019 01:56

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2019

PatrickLang mentioned this pull request Mar 7, 2019

Enable node e2e tests on Windows #75085

Closed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2019

k8s-ci-robot merged commit fbc5150 into kubernetes:master Mar 8, 2019

feiskyer deleted the windows-network-stats branch March 8, 2019 01:42

feiskyer mentioned this pull request Mar 11, 2019

Clean the workaround of duplicate interface stats for windows containers #75249

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add network stats for Windows nodes and containers #74788

Add network stats for Windows nodes and containers #74788

feiskyer commented Mar 1, 2019

feiskyer commented Mar 1, 2019

michmike commented Mar 1, 2019

michmike commented Mar 1, 2019

feiskyer commented Mar 4, 2019

feiskyer commented Mar 4, 2019

benmoss Mar 4, 2019 •

edited

Loading

feiskyer Mar 5, 2019

yujuhong Mar 6, 2019

feiskyer Mar 7, 2019

michmike commented Mar 6, 2019

michmike commented Mar 6, 2019

michmike commented Mar 6, 2019

yujuhong commented Mar 6, 2019

yujuhong commented Mar 6, 2019

PatrickLang commented Mar 6, 2019 •

edited

Loading

k8s-ci-robot commented Mar 6, 2019

yujuhong left a comment

yujuhong Mar 6, 2019

yujuhong Mar 6, 2019

feiskyer Mar 7, 2019

PatrickLang commented Mar 7, 2019

yujuhong commented Mar 7, 2019

feiskyer commented Mar 7, 2019

PatrickLang commented Mar 7, 2019

yujuhong commented Mar 7, 2019

k8s-ci-robot commented Mar 7, 2019

Add network stats for Windows nodes and containers #74788

Add network stats for Windows nodes and containers #74788

Conversation

feiskyer commented Mar 1, 2019

feiskyer commented Mar 1, 2019

michmike commented Mar 1, 2019

michmike commented Mar 1, 2019

feiskyer commented Mar 4, 2019

feiskyer commented Mar 4, 2019

benmoss Mar 4, 2019 • edited Loading

Choose a reason for hiding this comment

feiskyer Mar 5, 2019

Choose a reason for hiding this comment

yujuhong Mar 6, 2019

Choose a reason for hiding this comment

feiskyer Mar 7, 2019

Choose a reason for hiding this comment

michmike commented Mar 6, 2019

michmike commented Mar 6, 2019

michmike commented Mar 6, 2019

yujuhong commented Mar 6, 2019

yujuhong commented Mar 6, 2019

PatrickLang commented Mar 6, 2019 • edited Loading

k8s-ci-robot commented Mar 6, 2019

yujuhong left a comment

Choose a reason for hiding this comment

yujuhong Mar 6, 2019

Choose a reason for hiding this comment

yujuhong Mar 6, 2019

Choose a reason for hiding this comment

feiskyer Mar 7, 2019

Choose a reason for hiding this comment

PatrickLang commented Mar 7, 2019

yujuhong commented Mar 7, 2019

feiskyer commented Mar 7, 2019

PatrickLang commented Mar 7, 2019

yujuhong commented Mar 7, 2019

k8s-ci-robot commented Mar 7, 2019

benmoss Mar 4, 2019 •

edited

Loading

PatrickLang commented Mar 6, 2019 •

edited

Loading