Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

validation/linux_cgroups_*hugetlb: Use smaller limits #619

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wking
Copy link
Contributor

@wking wking commented Apr 6, 2018

The previous values were giving me:

container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks
process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/.../hugetlb.1GB.limit_in_bytes: permission denied\\\"\""

The previous values are originally from 432615a (#93), which doesn't motivate their choice. The new values are copy/pasted from the spec (which doesn't motivate its choice either ;). I've kept something like @alban's comment from #605 to at least explain how the limit breaks down.

In testing with my local system, the issue seems to be pageSize and not the limit value. That seems to be supported by the kernel docs, which have:

hugepages= [HW,X86-32,IA-64] HugeTLB pages to allocate at boot.
hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages.
On x86-64 and powerpc, this option can be specified
multiple times interleaved with hugepages= to reserve
huge pages of different sizes. Valid pages sizes on
x86-64 are 2M (when the CPU supports "pse") and 1G
(when the CPU supports the "pdpe1gb" cpuinfo flag).

My CPU supports both:

$ cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' \(pse\|pdpe1gb\) '
 pse
 pdpe1gb

but I don't set hugepagesz, and I seem to only get 2M by default. I can get 1GB entries by booting with hugepagesz=1GB. Longer-term, we may want to auto-detect the value(s) currently enabled by the host system, but for this commit I'm hard-coding 2MB.

The previous values were giving me:

  container_linux.go:348: starting container process caused
    "process_linux.go:402: container init caused
    \"process_linux.go:367: setting cgroup config for procHooks
    process caused \\\"failed to write 56892210544640 to
    hugetlb.1GB.limit_in_bytes: open
    /sys/fs/cgroup/hugetlb/.../hugetlb.1GB.limit_in_bytes: permission
    denied\\\"\""

The previous values are originally from 432615a (add cgroup hugetlb
test for runtime, 2017-12-05, opencontainers#93), which doesn't motivate their
choice.  The new values are copy/pasted from the spec [1] (which
doesn't motivate its choice either ;).  I've kept something like
Alban's comment from 984dbc8 (Fix error messages in validation cgroup
tests, 2018-03-14, opencontainers#605) to at least explain how the limit breaks
down.

In testing with my local system, the issue seems to be pageSize and
not the limit value.  That seems to be supported by the kernel docs,
which have [2]:

  hugepages=  [HW,X86-32,IA-64] HugeTLB pages to allocate at boot.
  hugepagesz= [HW,IA-64,PPC,X86-64] The size of the HugeTLB pages.
      On x86-64 and powerpc, this option can be specified
      multiple times interleaved with hugepages= to reserve
      huge pages of different sizes. Valid pages sizes on
      x86-64 are 2M (when the CPU supports "pse") and 1G
      (when the CPU supports the "pdpe1gb" cpuinfo flag).

My CPU supports both:

  $ cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' \(pse\|pdpe1gb\) '
   pse
   pdpe1gb

but I don't set hugepagesz, and I seem to only get 2M by default.  I
can get 1GB entries by booting with hugepagesz=1GB.  Longer-term, we
may want to auto-detect the value(s) currently enabled by the host
system, but for this commit I'm hard-coding 2MB.

[1]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config-linux.md#example-8
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/kernel-parameters.txt?h=v4.16#n1336

Signed-off-by: W. Trevor King <wking@tremily.us>
@dongsupark
Copy link
Contributor

I've just realized, this PR would conflict with #637, which touches hugetlb as well. Though I'm fine with this PR being first merged. I can rebase again.

And it's interesting. On my local machine with Fedora 28, both page sizes 2MB and 1GB are available, so I'm able to run hugetlb tests without any issue. Though there must be other systems where 1GB is not available, so I think it's ok to set page size to 2MB.

@wking
Copy link
Contributor Author

wking commented May 23, 2018

On my local machine with Fedora 28, both page sizes 2MB and 1GB are available...

What do you get from:

$ cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' \(pse\|pdpe1gb\) '  pse
$ cat /proc/cmdline

@dongsupark
Copy link
Contributor

@wking

# cat /proc/cpuinfo | grep '^flags' | head -n1 | grep -o ' \(pse\|pdpe1gb\) '
 pse 
 pdpe1gb 
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.16.9-300.fc28.x86_64 root=UUID=e54de305-64fd-4729-8e65-dd68a8fd36fe ro rootflags=subvol=root rhgb quiet resume=/dev/nvme0n1p4 LANG=en_US.UTF-8

@wking
Copy link
Contributor Author

wking commented May 23, 2018

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.16.9-300.fc28.x86_64 root=UUID=e54de305-64fd-4729-8e65-dd68a8fd36fe ro rootflags=subvol=root rhgb quiet resume=/dev/nvme0n1p4 LANG=en_US.UTF-8 

Hmm, I was expecting a hugepagesz entry. Maybe they've compiled in a default...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants