evaluate "Docker for Mac" as a Dinghy replacement

[codekitchen]

You have to agree to an NDA to get into the private beta, but I've gleaned plenty of information from public sources:

https://blog.docker.com/2016/03/docker-for-mac-windows-beta/
https://news.ycombinator.com/item?id=11352594

I've been saying for 18 months that Docker should solve the OS X dev problem themselves, so I'm a big fan of this. From what I've read so far it seems pretty close to Dinghy in terms of features.

• Mounted volumes are shared from the host using an as-yet-unknown solution that appears to support fsevents. Hopefully its performance is at least close to NFS, if not better.
• Claims that it remaps MacOS X UIDs into Linux ones (no more permissions problems), which could mean a couple different things. Hopefully it is aimed at editing code on a mounted volume.
• DNS resolving, apparently with routing the docker network interfaces directly to the host rather than through any proxy. I'd considered doing this too but never got around to it, it's a nice improvement over Dinghy's current solution. The HTTP proxy would still be nice for making SSL termination easy but that could be layered on top.
  • Not sure how DNS names will be chosen. I've come to really like the docker-compose based names but even if it's just <containername>.docker.local that'd be ok.
• Uses xhyve, which is about 10% slower than virtualbox for typical dev tasks in my testing. However, I've already switched to the dinghy xhyve driver myself because it's so much more pleasant than virtualbox for this use case.

Overall it sounds really nice. The proof will be once we can actually try it out, but if it works as well as they claim I'll probably just deprecate Dinghy once Docker for Mac is publicly released. That's assuming Docker for Mac is a free product, I'd hope it would be but they haven't said, and it sounds like at least parts of it will not be open source.

[codekitchen]

Looks like that's already been clarified, Docker for Mac will be free and open source.

https://news.ycombinator.com/item?id=11353747

[ryansch]

Agreed. This looks to be very exciting. I'm waiting for my invite :-)

[rkazak]

Me too!

[mrmachine]

I've been testing the beta today. The way it maps permissions for mounted volumes is different, and a little weird. It seems to map all bind mounted volumes to whatever UID:GID you are currently running as inside the container. So when you gosu foo, the owner and group of all your bind mounted volumes will change from root to foo.

In theory, this sounds like it should resolve the existing issue where you have to create/modify a user inside the container to match the UID:GID of the bind mounted volumes (which can't be changed). But I was previously working around that by doing gosu "$(stat -c '%u' "${DIR}"):$(stat -c '%g' "${DIR}")" which no longer works as expected.

So at the very least, this different behaviour has broken parity with images that work under regular Docker Engine.

Here's what their docs say. Sounds like this feature might be short-lived:

Presently, all requesting processes are treated as owners and group members on all bind mounted file system resources. This behavior simultaneously solves uid/gid mapping headaches and creates new compatibility (and possibly security) issues. A finer-grained ownership model is in development.

[codekitchen]

I tested out the beta for a bit yesterday, but ran into I/O errors when trying to bundle install onto a host-mounted volume. I haven't had a chance to dig further yet. Interestingly they're almost exactly the same git clone I/O errors I got when trying to run the unfsd NFS daemon as a non-root user. git seems to be especially sensitive to filesystem bugs.

[uberjay]

Unfortunately the Docker for Mac beta is significantly slower than dinghy for my workflow. I use the vmware driver -- between the osxfs driver burning significant CPU and the overhead of xhyve, a build which normally takes a little over 4 minutes ends up taking over 20 minutes.

I currently use vagrant for most of my development environment management, but jumped on the docker for mac beta in hopes it would simplify my workflow. And it does! However, the performance is bad enough that it's unusable for me. I have no doubt that xhyve performance will improve over time... eventually, it may be a good alternative. Dinghy to the rescue! :)

[codekitchen]

Yeah, I believe them when they say the FS performance will improve over time, they should be able to beat NFS performance using their architecture, with some work.

[michaellopez]

I've been playing with Docker for Mac the past few days. What I'm missing is the DNS functionality that Dinghy provides. I'm hoping that they include a custom DNS but looking at this thread it doesn't look like it...anytime soon anyways. Is there anyway to break out the DNS part of Dinghy into its own component that can be reused by Dinghy and standalone by Docker for Mac users?

[codekitchen]

Yep, the HTTP proxy and DNS server have been split into a separate project, install instructions are at https://github.com/codekitchen/dinghy-http-proxy#os-x

[michaellopez]

@codekitchen That's great. Thank you. I will give it a spin....I still find Docker for Mac unstable so I'll stick to Dinghy for the time being. 🍰

[cweagans]

osxfs still can't compete with nfs or even the vbox guest additions. It's so slow. IMO, Docker for Mac is not a complete replacement yet.

[igorvpcleao]

Hey @codekitchen,
Sorry for a silly question. Is it possible to run dinghy with docker-beta?

Thx!

[michaellopez]

@igorvpcleao From what I understand, the only thing you'd want from Dinghy would be its proxy & DNS which are available here https://github.com/codekitchen/dinghy-http-proxy. One major aspect of Docker for mac is to make osxfs performant and stable. So the NFS and fsevents from Dinghy should not really be needed in Docker for mac.

So if/once https://forums.docker.com/t/custom-dns-for-resolving-containers/8361 gets fixed/merged then I think that Dinghy might be up for deprecation all together.

But until Docker for mac is stable (which it wasn't when I tried it last month) I'm sticking to Dinghy. Now I just received e-mail from Docker that Docker for mac/windows is RC. Might be worth testing it out again.

[igorvpcleao]

Thanks @michaellopez!

Good to hear that. I don't know why, but I'm still experiencing problems with shared folders. Solving this DNS issue will make the transference among host and container faster? It's currently taking a long time to render my rails pages when I use shared folders.

[codekitchen]

Yeah that's a great summary, thanks @michaellopez

One twist is that we've got devs using Linux laptops and running Docker directly on the host, and it's nice that the Dinghy HTTP proxy works the same way for them. So we may continue on using it anyway, even if Docker for Mac introduces its own solution there.

[michaellopez]

@igorvpcleao No, proxy & DNS is only for resolving virtual hosts on the host to a container ip and port. It does nothing with shared folders or osxfs in Docker for mac.

[mrmachine]

Until osxfs is as fast as nfs I think there will still be a need for dinghy. It would be good to be able to use docker for Mac with nfs until that happens. Will it be possible?

[codekitchen]

I don't currently have any plans to support that, I haven't even looked into whether it's possible to tell docker for mac to disable its own built-in filesystem so that it could use NFS instead.

[rmtsrc]

Since Mac Docker now has a public beta I thought I'd try rebenchmarking.
Here's my results, I guess I'll stick with docker-osx-dev for now.

Mac benchmark (without Docker)

dd if=/dev/zero of=/tmp/output bs=8k count=100k; rm -f /tmp/output
102400+0 records in
102400+0 records out
838860800 bytes transferred in 1.132447 secs (740750602 bytes/sec)

docker-osx-dev (revision ea0617d)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 0.261691 s, 3.2 GB/s

Dinghy Docker over NFS (4.4.1)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 26.5691 s, 31.6 MB/s

Mac Docker Public Beta (1.12.0-rc2-beta17 build: 9779)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 34.3933 s, 24.4 MB/s

[mickaelperrin]

Thanks for the benchmark.

How would you explain the fact that dockerosx-dev is faster than native ?

If I clearly understood, you ran that commands inside your containers ? and /usr/src/app/output is mounted on the host ? So this is a benchmark only on writing performance ?

Could you provide the same benchmark but for reading performance ?

Thanks

[codekitchen]

As I understand it your docker-osx-dev benchmark is meaningless, because docker-osx-dev only does one-way syncing from the OS X host to the VM, not the other way around. So you're simply measuring the VM write performance. That's the primary reason we chose NFS for Dinghy, rather than rsync, because we have containers that we need to write to shares and persist them back to the OS X host.

[EugenMayer]

little update on this, the current docker-for-mac releases are not longer invite only, you do no longer need to enter a private beta key and therefore no NDA/Terms acceptance needed. This probably helps you.

I am very surprised to see that the rc2 does perform that well against NFS - most of the people in this thread, me included, are not using osxfs since its is absolutely to slow for everything: https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/109

I am here, because i am considering switching from docker for mac with rsync ( custom solution ) to dinghy. Eventhough rsync as the solution lets you run the containers in good speed, the sync is plain PITA in the end. On the paper, its working - but:

• fswatcher are worse then the shares on OSX
• fswacher ( ruby wrapper or the c implementation from brew ) needs about 80% of my CPU when watching my dev folder. This scales with then number of file, feels linear
• due to the way the events are propageted, there is no "bundling or grouping" this means, lets say you trigger rsync on a fswatcher event. No consider you switch a branch in SCM - you happen to have about 400 or more fswatcher events - each tries to sync all the data. This is VERY slow due to the connection overhead
• fswatcher regulary forgets to fire events - it is not relyable or convenient to use

So in the end i am about to give up with rsync ( being using it with vboxn, then fusion now docker for mac ) and trying to think about a NFS solution.

In general, anything except file sharing is pretty cool in docker for mac, so all i want is docker for mac with NFS most probably. But if the performance is as bad as up there ( close to osxfs ) i would rahter not use it.

Any real world examples?

[michaellopez]

@EugenMayer I'm not sure if you are looking for "testimonals" for Dinghy...But we are using Dinghy in our daily workflow at my company. I can't see how working with Docker on a mac can be done without it at the moment. We use modern Macbook Pros and so far no performance issue. Watchers are inside the container, Dinghy forwards file system events on OSX into the container. I can't say that we have very large trees being watched, but I am very meticulous about our builds, just watching what really needs watching. I often see tool.watch('**'), which of course will not perform well with a large tree. I'd be happy to answer any questions, or run something on our setup given that you provide something to run :)

[EugenMayer]

You run the watcher to compile/generate any kind of assets in the container right?
If you lets say change ruby files, you would not need watchers for that, those are synced using NFS, right?

Even without any node_modules/bower/coffescript/scss files, only watching for js/css/code we end up haing arround 12k files. This does, for sure, kill fswatcher + rsync but does totaly defeat osxfs ( we are talking about a factor far above 100 slower then rsync/native)

Am i right that NFS read performance is considerable bette then its write performance? Most of the people in the linked thread, when using synthetic benchmarks, use write benchmarks. I consider those benchmarks useless. No data generated ( written ) in the VM should ever happen to be saved on the code-base. Those data either stays in the container volume, or if to be persisted, on volumes whatever.

osxfs is pretty slow in reading also, so thats were i would need compare it to NFS and then NFS to native ( native is what i have right now ). When running a PHP cli up, parsing about 600 PHP files to bootstrap the app, reading performance does matter a lot :)

In case, when we have them all here: Docker 4 Mac, dinghy - what about dlite. How does dinghy compare to dlite?

[michaellopez]

@EugenMayer Yes, we use watchers inside the container to do things like recompile on save. The files are synced with NFS, but NFS does NOT trigger file system events inside the container. That's why Dinghy uses https://github.com/codekitchen/fsevents_to_vm, it reacts to file system events on OSX and sends a packet to the VM that trigger file system events for the container.

I honestly haven't worked on projects where I need to watch 12k files. I can't understand how it comes to that, but I respect that fact that those kinds of repositories do exist and needs to be worked with.

I do not have any source on NFS performance at hand, but I have a vague memory of reading somewhere that write performance better than read, yes.

If only the "checkout another branch" scenario, or similar, is the only case where the watchers poses a problem (meaning, commands that run seldom) and if you are using Dinghy, then maybe you can kill the fsevents_to_vm process, do the checkout and restart it again with dinghy up (note: untested :)

I have no experience with dlite.

[EugenMayer]

I am not happy with this amount of files - but you cant change it. Though, its at least not only repo :)

dlite 1.x seems to do less, which fits my needs a bit more as its seems. It does not support switching the provider, which is nice on dinghy side, though i use xhyve anyway.

dinghy provides dnsmasq and used with xhyve that colides with my boxen-stack dnsmasq. No needs for proxies (at least i can disable that).

It seems like dlite 2 and dinghy seem to be very similar - but i guess i am hijacking this topic.

Coming back to the performance, both using NFS while dlite uses the OSX build in server. Not sure if this harms or is better, but that could be different in performance terms ( either way )

I guess i try both of them compared against docker4mac+rsync and report back on my "real world test"

[EugenMayer]

Made some tests with dinghy and docker 4 mac:

dinghy

NFS
drush cc all  7.97s user 1.40s system 28% cpu 32.817 total

NO SHARE
drush cc all 7.52s user 0.79s system 75% cpu 10.993 total

Docker for Mac

OSXFS
drush cc all 8.42s user 4.06s system 24% cpu 51.265 total

NO SHARE
drush cc all 7.02s user 0.72s system 77% cpu 10.005 total

---

So NFS vs OSXFS is about 1.5 times slower, but in seconds, thats quiet some bit: 32s vs 51s

Bot compared to no share with 10s is 3-5 times slower. But since adds on each command this is for us, unusable.

Find: The import of the DB dump was a lot faster on Docker 4 mac "no share", so the I/O performance in terms of write is a lot better

---

So overall, the rsync approach is a lot better in general. Syncing all my 12k files takes 3 seconds, differentials sync, if triggered "grouped" takes less then 0.5s - no matter what there is. Running the app i up to 5 times faster.

[codekitchen]

Can we keep this issue discussion to the original purpose please? This side discussion is interesting but would be a lot better in a separate issue.

[EugenMayer]

Created https://github.com/EugenMayer/docker_sync as a result of all the tests, including the tests with dinghy. It has docker-for-mac support ( and docker-machine and all others ). Might be a way for dinghy also ( since dinghy is more, dns..proxy..) .. but it is rsync based, not NFS based, due to the performance issues with NFS.

When you plan to switch to docker-for-mac you might want to look at that gem, probably out-sourcing the sync part and keeping anything else like the http-proxy/dnsmasq in dinghy.

Update: I added unison support as a strategy, so this should be a proper replacement for the NFS based sync. Rsync is supported and the default, people can chose per sync-endpoint

[mickaelperrin]

Sounds interesting and is maybe a better implementation than docker-osx-dev, but if I am not wrong performance comes with some high restrictions compared to dinghy. Mainly there is no sync back, aka files created by processes in the container won't be synced back to the host.

[EugenMayer]

@mickaelperrin docker-sync now supports unison also, which is a two way sync like NFS would be. So also that case is covered, just check the docs ( you can switch between rsync/unison for each sync-endpoint, so pretty flexible )

[mnapoli]

Hi everyone, now that Docker for Mac is out and stable has anyone switched to it? If so, what are the performances like? I wonder if there's a difference between the beta and the stable release.

[EugenMayer]

@mnapoli eventhough there has been quiet some work on osxfs for the last stable release, benachmarks are already telling, nothing changed performance wise. This post is a pretty good statement from the docker for mac developers, what to expect from osxfs, when and what issues there are: https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/158?u=eugenmayer - it has been posted pretty recently

[mnapoli]

@EugenMayer thanks for the link, I'll be staying with Dinghy for now then.

[MrMMorris]

So Docker announced that Docker for Mac and Windows is now "ready for production", but based on the top comment in this HN thread, I am still not in any rush to move over (and they should re-evaluate what "production" means):

https://news.ycombinator.com/item?id=12184567

On top of all those issues, it does not support anything similar to the .docker container DNS which our workflow relies on heavily.

Dinghy fo' lyfe 🤘

[mrmachine]

@MrMMorris The *.docker container DNS is not needed. Docker for Mac/Windows exposes container ports directly on the host OS. So you can access your containerised services directly on localhost. If you were relying on the *.docker DNS to test things that expect a domain name in the host header, or test a site that behaves differently on subdomains, or wanted to keep cookies separate, etc., then you can just use *.lvh.me, which is a wildcard DNS that maps to 127.0.0.1. This is now one more thing (the hostname/port you access a service on) that works the same on Docker for Mac/Win as it does on Docker for Linux, or when running an project natively on the host OS.

[EugenMayer]

@mrmachine he is most probably referencing to dnsmasq, and even if everything is on localhost, you wont be able to use myproject.dev and yourporject.dev without a tool. Installing dnsmasq is as easy as

brew install dnsmasq
cp $(brew list dnsmasq | grep /dnsmasq.conf.example$) /usr/local/etc/dnsmasq.conf
sudo cp $(brew list dnsmasq | grep /homebrew.mxcl.dnsmasq.plist$) /Library/LaunchDaemons/
sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

You can modify its config at /usr/local/etc/dnsmasq.conf and adjust ist, by default it routes all *.dev to localhost, but you can also use *.docker or whatever.

So it is fairly easy, and in addition, also useful for not only docker related things

[mrmachine]

@EugenMayer I know he meant the *.docker wildcard DNS (dnsmasq). But you don't need a tool to access services via a domain name on localhost. As I said, just use *.lvh.me. It's a real wildcard domain that maps to 127.0.0.1, so you don't need dnsmasq. Just access myproject.lvh.me and yourproject.lvh.me on any computer without installing anything.

This won't work with Dinghy because the IP is not 127.0.0.1, which is why you need dnsmasq to access services via hostname with Dinghy.

[codekitchen]

Using *.lvh.me is a good tip, but note it's not quite the same thing -- since there's no HTTP proxy to do vhost resolving, you'd need to run each container on a different port. Doable of course, but not as straightforward as just typing in a domain name, especially when your dev work involves running many HTTP services.

[codekitchen]

Also note that dinghy-http-proxy has been extracted into a separate project that works just fine with Docker for Mac (I have a handful of co-workers doing this), see https://github.com/codekitchen/dinghy-http-proxy#using-outside-of-dinghy

[MrMMorris]

@codekitchen thanks for the tip! although it is still a more manual setup compared to Dinghy

[mrmachine]

@codekitchen unless I'm missing something, there's still only one docker host (VM) with Dinghy, so you can still only ever bind one container to a given port at a time. With Docker for Mac you just also have to avoid conflict with non-container services that are also running on localhost? With Dinghy, I think it's still not possible to say run two containers that both expose port 80 to the host? Or is the Dinghy proxy doing something clever like forwarding port 80 to whatever internally exposed port the containers are running (like dockercloud-haproxy does)? If the container is exposing multiple ports, how does the proxy know which one to proxy?

[codekitchen]

Yes, it proxies HTTP requests to the other running containers based on hostname, without them having to map a port to the host. See the docs https://github.com/codekitchen/dinghy-http-proxy

[codekitchen]

Just as a quick update, we still aren't seeing acceptable volume share performance in Docker for Mac. Some people at my company who only work on small projects have been using it daily without issue, but it's still too slow to use on our larger ruby and nodejs codebases.

[dts]

So the long and the short of it is, if you're comfortable with Dinghy, stick with dinghy because it is faster for large codebases? It'd probably be good to add some notes about this on the main Readme - I just got very confused when I visited the docker website and everything had changed...

[codekitchen]

That's a good call, I'll add something to the README

[MrMMorris]

I am evaluating http://docker-sync.io/ for increasing performance when using large codebases.

It definitely needs some work so if anyone wants to give it a try and contribute anything they can, that would be great!

[xfxf]

I've been recommending developers move away from Docker for Mac to Dinghy as the former seems to be awfully buggy at the moment and is exponentially slower with non-trivial projects using volume sharing. Some encouragement from me to keep working on this project -- it is valuable.

[debovis]

@xfxf
"move away from Docker for Mac to Dinghy as the former seems to be awfully buggy"

Is this still the case? Would love to see this issue be kept up to date. Thanks

[michaellopez]

@debovis You can help keep it up to date. Try out Docker for Mac and report back with your findings. The more data we have the better! 👍

[xfxf]

I swapped back to Docker For Mac about a month ago, and swapped back to dinghy about a week ago. dinghy is still exponentially faster than Docker For Mac.

[rodrigoaguilera]

I changed to docker for mac with http://docker-sync.io/
and this tweak applied
docker/for-mac#668
is really fast

[EugenMayer]

can confirm what @rodrigoaguilera said, using the fix docker/for-mac#668 with d4m gives a huge boost in our setup, its enormous for bigger installations / apps, maybe it does scale over linear. Still have to stick with docker-sync though for the host-mounts / shares, those are still very slow

[robinparisi]

Hi everyone,

I've just used this article after a clean upgrade (OSX 10.13) to install docker for mac.

Personally, I use delegated on volume (more infos) and it seems pretty fast at the moment. For example, before this update, my Laravel application was really slow (it took more than 5 seconds to load a page) but now, it's done in less than 1 second.

Also, I just wanted to say a huge thank you to @codekitchen for this awesome project that helped me a lot to get started on docker!

PS: compared to the article, you don't need the edge version anymore.

[codekitchen]

Using the delegated flag does sound promising, I'll have to try that out. Thanks for the tip @robinparisi

[beardcoder]

Hello :) here a speed Test between dinghy and docker4mac.

Native:

docker run --rm -it -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 0.23s
user	0m 0.03s
sys	0m 0.20s

Dinghy (Dinghy 4.6.5 with xhyve):

docker run --rm -it -v "$(PWD):/pwd" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 0.66s
user	0m 0.02s
sys	0m 0.21s

Dinghy (Dinghy 4.6.5 with virtualbox):

100000+0 records in
100000+0 records out
real    0m 1.34s
user    0m 0.06s
sys    0m 0.36s

docker4mac cached (Version 18.06.0-ce-mac69 (26398)):

docker run --rm -it -v "$(PWD):/pwd:cached" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 17.51s
user	0m 0.31s
sys	0m 1.73s

docker4mac delegated (Version 18.06.0-ce-mac69 (26398)):

docker run --rm -it -v "$(PWD):/pwd:delegated" -w /pwd alpine time dd if=/dev/zero of=speedtest bs=1024 count=100000
100000+0 records in
100000+0 records out
real	0m 19.36s
user	0m 0.19s
sys	0m 2.12s