com.docker.osxfs memory leak?

[ilg-ul]

Expected behavior

• process memory consumption to remain more or less constant

Actual behavior

• process memory increases

Information

• Full output of the diagnostics from "Diagnose & Feedback" in the menu: see D71F5326-0D23-4376-8A06-67DBD61AF38C
• A snapshot with the Activity Monitor showing the memory consumption that reaches 8+ GB:
  

Steps to reproduce the behavior

1. run a very long build, which creates tens of thousands of files on the host filesystem; I use a script that builds GCC with multilib
2. the initial memory usage is around 140 MB, but in time it slowly increases; after about a day, the memory usage reached 8+ GB; the system entered 4+ GB in swap, and performace degraded dramatically

This might also explain the slowness reported as #894.

[ilg-ul]

The problem is definitelly there, after restarting Docker and re-running the build, in several hours com.docker.osxfs reached 2.5 GB.

Since my builds consist of a series of runs with different images, could you suggest a scriptable workaround to restart com.docker.osxfs between runs?

[thaJeztah]

ping @dsheets @djs55 PTAL

[ilg-ul]

As for restart, a killall com.docker.osxfs seems to do the job, but perhaps there is a less brutal method.

[djs55]

@yallop: what do you think?

[yallop]

Thanks for the report, @ilg-ul. Could you post the script, along with other details needed to reproduce the problem (e.g. the Dockerfile or details of the image)?

[ilg-ul]

post the script

the script is public, but is quite elaborated:

https://github.com/gnu-mcu-eclipse/riscv-none-gcc-build/blob/master/scripts/build.sh

details of the image

there are several images, each with its docker file, all available from https://github.com/ilg-ul/docker

however the main script is still work in progress, and might need some small adjustments if you want to run it. I'll update the script and let you know.

[ilg-ul]

FYI, one run of the script, started yesterday, just completed and took around 20 hours. given I have 4 such runs, I would say that this solution is mostly useless :-(

[ilg-ul]

I upgraded one macMini to 16 GB and installed Docker stable; the behaviour is the same, the osxfs process slowly grows and occasionally it shrinks; sometimes it reaches several GB, sometimes only a few hundred MB.

the idea to use killall to restart osxfs is not realistic; the process is killed, but it takes forever to restart, and atempts to start docker during this period will fail, crashing the entire script.

I guess the build script should be now functional, you can try it. I suggest you start it with --without-pdf, otherwise you'll need TeX, and the script to install TeX is not longer functional (TeX 2017 was recently released and I need to update the script) .

$ git clone https://github.com/gnu-mcu-eclipse/riscv-none-gcc-build.git ~/Downloads/riscv-none-gcc-build.git
$ caffeinate bash
$ exec bash ~/Downloads/riscv-none-gcc-build.git/scripts/build.sh --without-pdf --deb64 --win64

at first run the script will create a custom homebrew instance (${HOME}/opt/homebrew-gme), then download a large docker image.

the script will create a subfolder in ${HOME}/Work and all builds will happen there. depending on your machine, each platform may take several hours. if the machine has not enough ram, and swap will be used, the script will probably take forever.

---

I started a new build with all libraries; I'll see in a few hours if the machine ran out of memory or not.

[ilg-ul]

about 24 hours later, and almost half into the build, Docker is still running. :-)

it slowly grew to more than 5.5 GB. then I had to manually test some docker commands (like docker run hello-world), and, after a while, when I checked again, the osxfs memory consumption was down to around 600 MB. I have no idea if the commands I used cleared the memory, or the 6 GB limit I set in the Docker preferences page was reached.

as for the slowness, I don't have a way to measure it, but apparently when the osxfs memory consumption is high, even without using any swap, Docker seems to slow down; probably accessing the large cache takes some time.

[ilg-ul]

unfortunately it didn't last that much. at a certain point it reached more than 8 GB, swap was used and the entire machine became almost unresponsive, so I had to restart it. :-(

It seems Docker for macOS is not yet ready for prime time... I'll try to switch running Docker inside an Ubuntu virtual machine...

[thaJeztah]

Out of interest; if I understand your script correctly, all source (and some packages) are installed on the host, but execution is done inside a container. The script is quite complex, so I may not fully grasp it, but is there a reason you're not performing the actual build/execution steps in a Dockerfile?

When building in a Dockerfile, possibly the build cache could help bring down build times, and it would definitely save the time spent on continuously syncing filesystem changes between the Docker VM, and the OS X host (multi-stage builds could help as well, and this proposal may be helpful; moby/moby#32507). Docker 17.07 will also have a new feature for sending the build-context incrementally to the daemon (moby/moby#32677)

This doesn't take away that this should definitely be looked into, just looking at this use case from a slightly different perspective.

Also, if you have thoughts on the proposal I linked, or other changes to the builder that's being worked on (see moby/moby#32925, and the https://github.com/moby/buildkit repository), input is really appreciated 👍

[ilg-ul]

is there a reason you're not performing the actual build/execution steps in a Dockerfile?

yes, because the script is currently under active development, and during development the build crashes quite often and I need to resume it.

but the suggestion is valid, I'll consider adding an option to keep the build folder locally.

[ilg-ul]

as suggested, I changed the script to place the build folder inside the container; this improved things, at least the build script is able to complete (it takes more than 30 hours to build a multilib GCC for 5 platforms!). the install folder, where the build results are stored, remained on a shared folder.

however, the problem might be still there, the com.docker.osxfs memory slowly increases, and sometimes is partly freed; I saw values in the 500-1500 MB range.

with 16 GB of RAM this isn't unbearable, but if I move the build folder on a shared folder, sometimes the memory is no longer freed and the system starts to swap on disk, which is catastrophic in terms of speed.

I'm using 17.06.2-ce-mac27 (19124).

[ilg-ul]

As suggested, I generally run the builds entirely in the Docker image, and only copy the result to the host, and generally the builds are ok.

Today I needed the build in development mode, i.e. run the build inside the container, but with the build folders on the host, so I can restart the build if needed.

The memory leak is still there.

The .osxfs process started with a few tens of MBs, slowly grew at some hundreds of MBs, then something very wong happened and it grew to several GBs. By that time everything became very slow, and I had to cancel the build, restart the Docker process, and resume.

I'm currently using 17.09.0-ce-mac35 (19611).