Support mounting host directories into pods

[vishh]

Having the ability to mount local directories to pods will improve the developer experience a lot.

• Builds can be done in a local kube cluster
• Testing changes to an app will be much faster, without requiring creation of a docker container and a registry.

Given that most hypervisors can already support mapping host directories, adding this feature should be straightforward.

[dlorenc]

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

[rata]

On Mon, May 16, 2016 at 05:31:18PM -0700, dlorenc wrote:

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

NFS persistent volume claim? What? Why?

Why not HostPath, to use a local directory, and that's all regarding the volume?

[dlorenc]

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

For that to work we might need to do dynamic mounting/provisioning from the laptop into the VM.

[rata]

On Mon, May 16, 2016 at 05:42:00PM -0700, dlorenc wrote:

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

Oh, I see what you mean now. But I'm not sure, as @vishh said in the original
report: "most hypervisors can already support mapping host directories". So,
maybe a host dir can be mounted in the VM and then in the container.

NFS is another option, but I'm not sure how that would work. A host dir in the
VM, that dir exported as NFS, and mounted by k8s as an NFS volume? Or what were
you thinking?

[rata]

Also, if docker is used, it can mount from host directly on mac/windows with docker "native" beta. One option is to use that.

Another is to do the same/similar trick that docker is using (probably the one @vishh said and the one I said in my previous post)

[vishh]

If we can have kubelet take in a configurable root directory for hostPath volumes, then we can have the kubelet map in directories from inside the VM that mirror the host root.

[Ehekatl]

this feature is super important for us, is there any way to let me add a shared folder before this come out ?

[dlorenc]

Hey, on a Mac using Virtualbox, the /Users/ directory is automatically mapped into the VM (also under /Users). You could then use a HostPath volume of "/Users/$USER/$DIR". Does that help?

[Ehekatl]

@dlorenc thx, I add mount -t command at boot2docker init script now, but hostpath volume will facing some permission issue (e.g. postgres container), seems nfs is the only workaround

[solsson]

The permission issues that @Ehekatl mentions are quite a blocker. With a PersistentVolume mounted to a pod with a https://hub.docker.com/_/postgres/ container I get:

root@postgres-0:/# ls -la /var/lib/postgresql/data/
drwxr-xr-x 2 root     root       40 Jul 18 15:05 .
drwxr-xr-x 3 postgres postgres 4096 Jul 18 15:09 ..

When postgres tries to create data there it fails with mkdir: cannot create directory ‘/var/lib/postgresql/data/pg_xlog’: Permission denied. No chmod or chown on the host system seems to make any difference. There's plenty of official and non-official Docker images that run a non-privileged user. Postgres prepends its start command with gosu postgres.

root@postgres-0:/# id     
uid=0(root) gid=0(root) groups=0(root)
root@postgres-0:/# gosu postgres id
uid=999(postgres) gid=999(postgres) groups=999(postgres),109(ssl-cert)

I've tested with both virtualbox and xhyve minikube, on Mac.

[rata]

@solsson: maybe it's an option to add permission to the volume API in kubernetes. I'm doing that for others volumes: kubernetes/kubernetes#28733 and kubernetes/kubernetes#28936

[solsson]

I noticed that kubernetes/kubernetes#28733 is merged now. Interesting.

Before Docker for Mac, with Docker Toolbox and boot2docker, I had this kind if issue often with plain docker containers. Hence I'm not so sure the issue is with Kubernetes. Maybe it's inherent with docker + non-privileged user + virtual machine mount.

If I use /tmp/somevolume as hostPath I do get the same issue at postgres start. But the difference is that I can chmod go+ws /tmp/somevolume to solve it (in virtualbox minikube).

This leads me to believe that the problem is with minikube. After all hostPath works fine on for example GKE. Maybe it's with how minikube mounts the shared folder?

[solsson]

This boot2docker issue describes the problem pretty accurately.

I think I found a workaround for virtualbox. After minikube start open the vm in virtualbox (or ssh) and run

umount /Users
mount -t vboxsf -o rw,nodev,relatime,suid,dmode=777,fmode=777 Users /Users

The first three options are from the default mount. Probably only one of the added options are needed.

With postgres howerver the next issue is:

LOG:  could not link file "pg_xlog/xlogtemp.20" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL:  could not open file "pg_xlog/000000010000000000000001": No such file or directory

which I think confims my suspicion that docker and vm shared folders is not a good fit.

[solsson]

In conclusion, an added mount option (see above) might improve minikube. But until native docker is supported, including Docker for Mac, I think that to stay sane one should realize that some services demand a lot from the file system and hostPath should really be the linux host's folder.

With /var/kubedata/pgdata-postgres-0 postgres runs fine. I noticed that /var has more liberal permissions than /tmp that I tested above.

So to access this data directly I'd have to share it from the VM to my physical host.