Design proposal for Windows Container Configuration in CRI #1510

JiangtianLi · 2017-12-14T21:35:50Z

No description provided.

JiangtianLi · 2017-12-14T21:48:42Z

/sig node
/sig windows

JiangtianLi · 2017-12-14T21:50:17Z

/cc @jhowardmsft; @darrenstahlmsft; @jstarks; @PatrickLang; @taylorb-microsoft; @michmike

yujuhong · 2017-12-15T02:28:15Z

contributors/design-proposals/node/cri-windows.md

+[#56734](https://github.com/kubernetes/kubernetes/issues/56734)
+
+## Motivation
+The goal is to fill the gap of platform support in CRI, specifically for Windows platform. For example, currrently in dockershim Windows containers are scheduled using the default resource constraints and does not respect the resource requests and limits specified in POD. With this proposal, Windows containers will be able to leverage POD spec and CRI to allocate compute resource and respect restriction.


It'd be nice to know besides resources, what other parts of the current API is lacking to support Windows containers (e.g., security context?).

@yujuhong According to https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go#L418-L434, I think there are a few other properties such as CredentialSpec or HyperV that could be useful and already implemented in Windows. How does LinuxContainerSecurityContext map to https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go#L132-L164? I saw some similarity such as Seccomp. Selinux and Apparmor is in Process struct though (https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go#L30-L57)

@taylorb-microsoft Please advise what property besides Resources is needed to support Windows containers now (https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go#L418-L434).

@yujuhong @JiangtianLi I have a draft on this that we're discussing in SIG-Windows. @feiskyer also has a draft that we're merging together for the SIG with @michmike. Next meeting is 9 Jan. After that we'll send it up to SIG-Node

@PatrickLang feel free to email me the draft and i can take a look ahead of our jan9 meeting

Next meeting is 9 Jan. After that we'll send it up to SIG-Node

Thanks for the update! I think we can hold on to this PR and wait for the more complete proposal to be presented in both SIGs.

dchen1107 · 2018-01-16T18:23:56Z

contributors/design-proposals/node/cri-windows.md

+	// CPU resource restriction configuration.
+	CPU *WindowsCPUResources `json:"cpu,omitempty"`
+	// Storage restriction configuration.
+	Storage *WindowsStorageResources `json:"storage,omitempty"`


I didn't see anywhere defining WindowsStorageResources. Since Storage is mentioned to support later in this proposal, can we remove this here?

… section Signed-off-by: Patrick Lang <plang@microsoft.com>

feiskyer · 2018-01-23T03:04:16Z

/ok-to-test

yujuhong · 2018-01-23T22:10:06Z

contributors/design-proposals/node/cri-windows.md

+    // Specifies the portion of processor cycles that this container can use as a percentage times 100.
+    int64 cpu_maximum = 3;
+    // Memory limit in bytes. Default: 0 (not specified).
+    int64 memory_limit_in_bytes = 4;


Could you give an example of how kubernetes cpu/memory requests/limit would map to these fields?

Also, are there any significant difference in these configurations and how kernel reacts to the conditions between linux and windows?

example of how kubernetes cpu/memory requests/limit would map

// Refer https://github.com/moby/moby/blob/master/daemon/oci_windows.go#L265 // and https://github.com/Microsoft/hcsshim/blob/master/interface.go#L77. CpuCount = int((container.Resources.Limits.Cpu().MilliValue() + 1000)/1000) // 0 if not set // milliCPUToShares converts milliCPU to 0-10000 CpuShares = milliCPUToShares(container.Resources.Limits.Cpu().MilliValue()) if CpuShares == 0 { CpuShares = milliCPUToShares(container.Resources.Request.Cpu().MilliValue()) } CpuMaximum = container.Resources.Limits.Cpu().MilliValue()/sysinfo.NumCPU()/1000*10000 if isHyperV { CpuMaximum = container.Resources.Limits.Cpu().MilliValue()/CpuCount/1000*10000 } MemoryLimitInBytes = container.Resources.Limits.Memory().Value()

are there any significant difference in these configurations and how kernel reacts to the conditions between linux and windows?

Those CPU parameters are different from Linux, and there is no CFS on windows. this MSDN documentation explains how cpu/memory resource is controlled for Windows containers.

Also to clarify - CpuMaximum is a limits that can't be exceeded. If the node is overprovisioned and under contention for multiple containers operating below their limit, then cycles will be weighted based on shares. If the system isn't overprovisioned then shares has no effect.

I see. Thanks for explaining. Those are the things that I wanted to see in this proposal.

How does CpuCount interact with CpuMaximum?

Thanks for reviewing. I'll add to the proposal.
From https://github.com/docker/docker-ce/blob/master/components/cli/man/docker-run.1.md,
On Windows Server containers, the processor resource controls are mutually exclusive, the order of precedence is CPUCount first, then CPUShares, and CPUPercent last.
CPUMaximum is derived from CPUCount for HyperV container, and from sysinfo.NumCPU() for process container.

Note that for Hyper-V containers these controls are not mutually exclusive. The CPUCount controls the number of virtual processors that the container has access to (and by extension, the host's logical processors available for execution). The CpuMaximum applies to each of these virtual processors independently, for example, CpuCount=2,CpuMaximum=5000 (50%) would limit each CPU to 50%. Running a single threaded application that uses as much CPU as available on the above configuration would be able to use at most 50% of a single host core.

As mentioned above, for Windows Server containers (process isolation) CpuCount would take precedence (and the cpu limit would be simulated based on the provided value compared to the number of host CPUs using JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP), and CpuMaximum would be ignored.

@PatrickLang @darrenstahlmsft Thanks for the input. I have added part of your comment into design proposal. Is there any reference to what you have commented besides:
https://github.com/moby/moby/blob/master/daemon/oci_windows.go
https://github.com/Microsoft/hcsshim/blob/master/interface.go
https://github.com/docker/docker-ce/blob/master/components/cli/man/docker-run.1.md
https://msdn.microsoft.com/en-us/library/windows/desktop/hh448384.aspx

feiskyer · 2018-01-26T00:22:54Z

ping @dchen1107 @derekwaynecarr Would you like to take a look at this?

dchen1107 · 2018-01-30T17:33:58Z

contributors/design-proposals/node/cri-windows.md

+
+// WindowsContainerResources specifies Windows specific configuration for
+// resources.
+message WindowsContainerResources {


I must miss something here. What are the between this: WindowsContainerResources and above API: WindowsResources?

From the explanation given below, what are defined here are the values passing to windows runtime?

WindowsResources is from OCI spec (https://github.com/opencontainers/runtime-spec/blob/master/specs-go/config.go). It is essentially the same as WindowsContainerResources in CRI here.

dchen1107 · 2018-01-30T19:55:47Z

LGTM thanks for updating with the mapping function too.

PatrickLang · 2018-02-01T00:04:36Z

@derekwaynecarr did you have any feedback?

dchen1107 · 2018-02-02T23:20:03Z

/lgtm

k8s-ci-robot · 2018-02-02T23:20:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dchen1107, JiangtianLi

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~contributors/design-proposals/node/OWNERS~~ [dchen1107]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubelet: setup WindowsContainerResources for windows containers **What this PR does / why we need it**: This PR setups WindowsContainerResources for windows containers. It implements proposal here: kubernetes/community#1510. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #56734 **Special notes for your reviewer**: **Release note**: ```release-note WindowsContainerResources is set now for windows containers ```

Design proposal for Windows Container Configuration in CRI

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 14, 2017

k8s-github-robot assigned derekwaynecarr and dchen1107 Dec 14, 2017

k8s-github-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Dec 14, 2017

JiangtianLi mentioned this pull request Dec 14, 2017

Add windows config to Kubelet CRI kubernetes/kubernetes#57076

Merged

JiangtianLi force-pushed the jiangtli-wincri branch from 4cc4909 to 93a929b Compare December 14, 2017 21:41

Design proposal for Windows Container Configuration in CRI

b90f483

JiangtianLi force-pushed the jiangtli-wincri branch from 93a929b to b90f483 Compare December 14, 2017 21:44

k8s-ci-robot added the sig/windows Categorizes an issue or PR as relevant to SIG Windows. label Dec 14, 2017

yujuhong reviewed Dec 15, 2017

View reviewed changes

dchen1107 reviewed Jan 16, 2018

View reviewed changes

Patrick Lang and others added 2 commits January 22, 2018 09:45

Adding more explanation, and moving storage controls to 'future work'…

fd8df07

… section Signed-off-by: Patrick Lang <plang@microsoft.com>

Add security context to future work

66b8409

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 23, 2018

Add Patrick as contributor

bdd5aaf

yujuhong reviewed Jan 23, 2018

View reviewed changes

feiskyer mentioned this pull request Jan 24, 2018

Add support for Windows Container Configuration in CRI kubernetes/enhancements#547

Closed

JiangtianLi force-pushed the jiangtli-wincri branch 5 times, most recently from 6ed4994 to a65926e Compare January 26, 2018 07:11

Add mapping example of kubernetes cpu/memory requests/limit

2421112

JiangtianLi force-pushed the jiangtli-wincri branch from a65926e to 2421112 Compare January 26, 2018 07:16

dchen1107 reviewed Jan 30, 2018

View reviewed changes

Add feature request link

6d08a82

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 2, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2018

k8s-ci-robot merged commit 79082b2 into kubernetes:master Feb 2, 2018

feiskyer mentioned this pull request Feb 5, 2018

kubelet: setup WindowsContainerResources for windows containers kubernetes/kubernetes#59333

Merged

MadhavJivrajani pushed a commit to MadhavJivrajani/community that referenced this pull request Nov 30, 2021

Merge pull request kubernetes#1510 from JiangtianLi/jiangtli-wincri

27cbdcd

Design proposal for Windows Container Configuration in CRI

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Design proposal for Windows Container Configuration in CRI #1510

Design proposal for Windows Container Configuration in CRI #1510

JiangtianLi commented Dec 14, 2017

JiangtianLi commented Dec 14, 2017

JiangtianLi commented Dec 14, 2017

yujuhong Dec 15, 2017

JiangtianLi Dec 27, 2017

PatrickLang Dec 27, 2017 •

edited

Loading

michmike Dec 28, 2017

yujuhong Jan 2, 2018

dchen1107 Jan 16, 2018

feiskyer commented Jan 23, 2018

yujuhong Jan 23, 2018

feiskyer Jan 24, 2018 •

edited

Loading

PatrickLang Jan 25, 2018

yujuhong Jan 25, 2018

JiangtianLi Jan 25, 2018

darstahl Jan 25, 2018 •

edited

Loading

JiangtianLi Jan 26, 2018

feiskyer commented Jan 26, 2018

dchen1107 Jan 30, 2018

JiangtianLi Jan 30, 2018

dchen1107 commented Jan 30, 2018

PatrickLang commented Feb 1, 2018

dchen1107 commented Feb 2, 2018

k8s-ci-robot commented Feb 2, 2018

Design proposal for Windows Container Configuration in CRI #1510

Design proposal for Windows Container Configuration in CRI #1510

Conversation

JiangtianLi commented Dec 14, 2017

JiangtianLi commented Dec 14, 2017

JiangtianLi commented Dec 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang Dec 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feiskyer commented Jan 23, 2018

Choose a reason for hiding this comment

feiskyer Jan 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darstahl Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feiskyer commented Jan 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dchen1107 commented Jan 30, 2018

PatrickLang commented Feb 1, 2018

dchen1107 commented Feb 2, 2018

k8s-ci-robot commented Feb 2, 2018

PatrickLang Dec 27, 2017 •

edited

Loading

feiskyer Jan 24, 2018 •

edited

Loading

darstahl Jan 25, 2018 •

edited

Loading