Unable to use existing EFS filesystem

[jarrah42]

Description

I'm trying to start a service using ECS and mount an existing EFS as a volume. When I try, I get the following error:

mount target already exists in this AZ (Service: AmazonElasticFileSystem; Status Code: 409; Error Code: MountTargetConflict; Request ID: ea894cad-f63e-4aeb-8e4f-7711a28102e3; Proxy: null)

Steps to reproduce the issue:

1. Create an EFS filesystem and put data in it.
2. Create an AWS context and use it.
3. Run 'docker compose up'

docker-compose.yml

version: '3'
services:
    coney-neo4j: 
      image: neo4j:3.5
      ports:
        - "127.0.0.1:7474:7474"
        - "127.0.0.1:7687:7687"
      volumes: 
        - coney-neo4j:/data
      networks:
        - coney-net      
networks:
  coney-net:
    driver: bridge

volumes:
  coney-neo4j:
    external: true
    name: fs-21bf435a

Describe the results you received:

mount target already exists in this AZ (Service: AmazonElasticFileSystem; Status Code: 409; Error Code: MountTargetConflict; Request ID: ea894cad-f63e-4aeb-8e4f-7711a28102e3; Proxy: null)

Describe the results you expected:

Filesystem to be mounted.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:57 2021
 OS/Arch:           darwin/amd64
 Context:           ryp
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:
You can also run docker context inspect context-name to give us more details but don't forget to remove sensitive content.

ryp

Output of docker info:

Command "info" not available in current context (ryp), you can use the "default" context to run this comman

Additional environment details (AWS ECS, Azure ACI, local, etc.):

[jarrah42]

As a workaround, I let docker compose create the EFS volume. After that it seemed to mount correctly. There seems to be some issue using an EFS volume I create manually through the console.

[jarrah42]

After I made some changes to the docker-compose.yml file, I am now seeing this message again. My only option seems to be to remove the filesystems and allow them to be created again. So it seems like this is a bug.

[thorfi]

Yeah, bump - I'm seeing this issue as well.

What is happening is that when you have an external EFS, the docker compose generated CloudFormation YAML expects to create the AWS::EFS::MountTarget entries.

e.g.:

volumes:
  my_volume_name:
    external: true
    name: fs-1234567890123

Results in doing this:

  MyvolumenamemountNFSMountTargetOnSubnetNNNNNN:
    Properties:
      FileSystemId: fs-1234567890123
      SecurityGroups:
      - Ref: DefaultNetwork
      SubnetId: subnet-NNNNNN
    Type: AWS::EFS::MountTarget

So if you attempt to use the same FileSystemId as an external volume more than once, and the mount target already exists (either because it was created by a different stack or someone created it manually e.g. during EFS volume creation), then the stack will fail because AWS will not allow a mount target with the same pair of FileSystemId and SubnetId.

What we need here is that the MountTargets should not really be "inside" the stack - they need to be maintained outside of the stack, like the EFS volumes are.

As a minimum solution external: true should probably not really create the MountTargets, or else add external-mount-targets: true or something similar as an additional option?

[thorfi]

FYI my current workaround is to do this ugly hack (which works because I'm hard coding the SubnetIds):

x-aws-cloudformation:
  Resources:
    # We convert external NFSMountTargets into something that we can have lots of and ignore, because
    # external mount targets already exist and docker compose does not understand how to manage that
    MyvolumenameNFSMountTargetOnSubnetN111111N: &my-volume-mount-overlay
      Type: AWS::EFS::AccessPoint
      Properties:
        SecurityGroups:
        SubnetId:
        FileSystemId: fs-1234567890123
        AccessPointTags:
          - Key: Name
            Value: fs-1234567890123
          - Key: com.docker.compose.project
            Value: my-project-name
          - Key: com.docker.compose.volume
            Value: my_volume_name
    MyvolumenameNFSMountTargetOnSubnetN222222N: *my-volume-mount-overlay
    MyvolumenameNFSMountTargetOnSubnetN333333N: *my-volume-mount-overlay

[jhs88]

Has anyone gotten this issue resolved. I'm currently trying to achieve something similar and my volumes will not mount no matter what I do. I tried @thorfi work around, but it still fails.

[jarrah42]

I haven’t had any more issues, other than that first hiccup, after letting docker compose create the filesystems.

[ryands17]

I think this is a sort of a deadlock as docker compose when creating the filesystem doesn't add the inbound NFS rule which causes the stack to fail and it also fails as above if we manually specify a filesystem.

Related issue: #1490

[chingc]

FYI my current workaround is to do this ugly hack (which works because I'm hard coding the SubnetIds):

x-aws-cloudformation:
  Resources:
    # We convert external NFSMountTargets into something that we can have lots of and ignore, because
    # external mount targets already exist and docker compose does not understand how to manage that
    MyvolumenameNFSMountTargetOnSubnetN111111N: &my-volume-mount-overlay
      Type: AWS::EFS::AccessPoint
      Properties:
        SecurityGroups:
        SubnetId:
        FileSystemId: fs-1234567890123
        AccessPointTags:
          - Key: Name
            Value: fs-1234567890123
          - Key: com.docker.compose.project
            Value: my-project-name
          - Key: com.docker.compose.volume
            Value: my_volume_name
    MyvolumenameNFSMountTargetOnSubnetN222222N: *my-volume-mount-overlay
    MyvolumenameNFSMountTargetOnSubnetN333333N: *my-volume-mount-overlay

This fixed a BUNCH of frustrating problems I've been having with EFS volumes. Thank you.

[geocertify]

This thread goes back a full year, highlighting a fundamental problem--actually just one of many issues I've had to deal with setting up a simple Docker/ECS implementation for a website with a backend database. The hack offered by thorfi seems to be working for me. The problem is I'm not hearing any music--I'm looking for signs of activity or progress on the Docker/ECS, but it's just crickets. Is this a safe path, or should I be moving to Kubernetes or some other platform? Like I said, it's just a scalable web frontend with a shared volume for data and node_modules. What's everybody doing?