Skip to content
This repository has been archived by the owner on Oct 30, 2018. It is now read-only.

Reduce maximum memory usage via tempfiles #191

Merged
merged 3 commits into from
Dec 12, 2017
Merged

Reduce maximum memory usage via tempfiles #191

merged 3 commits into from
Dec 12, 2017

Commits on Dec 6, 2017

  1. Add cpu and mem profiling options 

    Following https://golang.org/pkg/runtime/pprof/. When attempting to build
images in https://github.com/linuxkit/kubernetes CI the process is mysteriously
being SIGKILL'd, which I think might be down to OOMing due to the resource
limits placed on the build container.

I haven't done so yet but I'm intending to use these options to investigate and
they seem potentially useful in any case, even if this turns out to be a
red-herring.

Signed-off-by: Ian Campbell <ijc@docker.com>
    Ian Campbell committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    9558740 View commit details
    Browse the repository at this point in the history

  2. Generate intermediate image into a temp file 

    All of the `output*` functions took a `[]byte` and immediately wrapped it in a
`bytes.Buffer` to produce an `io.Reader`. Make them take an `io.Reader` instead
and satisfy this further up the call chain by directing `moby.Build` to output
to a temp file instead of another `bytes.Buffer`.

In my test case (building kube master image) this reduces Maximum RSS (as
measured by time(1)) from 6.7G to 2.8G and overall allocations from 9.7G to
5.3G. When building a tar (output to /dev/null) the Maximum RSS fell slightly
from 2.2G to 2.1G. Overall allocations remained stable at around 5.3G.

Signed-off-by: Ian Campbell <ijc@docker.com>
    Ian Campbell committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    9f44acf View commit details
    Browse the repository at this point in the history

  3. Stream docker export directly to consumer 

    Rather than queueing up into a `bytes.Buffer`.

In my test case (building kube master image) this reduces Maximum RSS (as
measured by time(1)) compared with the previous patch from 2.8G to 110M. The
tar output case goes from 2.1G to 110M also. Overall allocations are ~715M in
both cases.

Signed-off-by: Ian Campbell <ijc@docker.com>
    Ian Campbell committed Dec 6, 2017
    Configuration menu
    Copy the full SHA
    3045a80 View commit details
    Browse the repository at this point in the history