docker cp to attached volume in container causes an unmount to be issued

[j-griffith]

If you try and use to xfr data from node to a mounted volume in a container the result is an unMount being sent from Docker after the Mount is detected as already existing.

To reproduce:
docker volume create -d netapp --name=foobar
docker run -it --volume-driver netapp --volume foobar:/data --name crumbs bash

Then on the host:
docker cp /var/log/syslog- crumbs:/data/syslog

After running the docker cp command the volume is unmounted by the go-plugin-helper, the logs show this here:

DEBU[2017-08-25T15:21:05Z] Mount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] Mounting volume foo on /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Begin osutils.GetDFOutput DEBU[2017-08-25T15:21:05Z] /var/lib/docker-volumes/solidfire/foo already mounted, returning existing mount 2017/08/25 15:21:05 Entering go-plugins-helpers unmountPath DEBU[2017-08-25T15:21:05Z] Unmount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] SolidfireSANStorageDriver#Detach(foo, /var/lib/docker-volumes/solidfire/foo) DEBU[2017-08-25T15:21:05Z] Begin osutils.Umount: /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Response from umount /var/lib/docker-volumes/solidfire/foo:

I think this might be an upstream issue with the Docker volume API but not sure yet.

[j-griffith]

After further investigation this appears to be a problem with Docker and some assumptions that may be made regarding iSCSI connections.

There are different ways to creat an iSCSI target for a device, one is to use unique target IQN's for each "volume" that exists on a device. This is how SolidFire works.

The other is to used a shared TGT IQN and access individual volumes via iscsi portal/lun assignment. A number of devices use this method including the ONTAP devices.

The docker cp operation in this case does some things with a volume that's attached to a container that doesn't work out well for unique targets (SolidFire for example). As part of the copy process docker will first issue an unmount command to the plugin API, which in our case does an iSCSI logout. This is the first problem, this means we just disconnected and unmounted an in use file system. Next, docker will issue a new Mount command; and copy the data from the local system to the mount point on the local system. After that's complete, it will again Unmount, disconnect and then Mount again back on the container.

This won't work for devices that use individual targets and could potentially cause issues for others. To work around this the best option is to mount the data source in to the container as well. From there use the container FS to copy the data from the source to the attached NDVP volume in the container. So something like this:
docker run -it -v <path-to-source-data>:/src-data --volume-driver ndvp -v my-ndvp-vol:/dest debian cp /src-data/<my-stuff> /dest/

[dutchiechris]

@j-griffith Eeww! Is there an issue open with the Moby project? I searched but couldn't find one.

[j-griffith]

I have a fix using ref counting here, there's definitely some things we should've been doing better here.

[j-griffith]

Preview of fix here: https://github.com/j-griffith/netappdvp/tree/refcount_attachments_in_solidfire

[ntap-rippy]

The change in 29aceaf should work around the problem by delaying the iSCSI session teardown until we delete the volume. The 'docker cp' commands that failed above work for me after this change.