Docker image is slow to start due to chown command

[scjudd]

Description

When running synapse under docker and setting up the /data directory to be a bind-mount to an AWS EFS file system, the recursive chown command that start.py executes takes between 5-10 minutes to finish. Synapse doesn't start until this is done.

Steps to reproduce

• Create EFS file system, mount it from EC2 instance
• Fill EFS volume with lots of files, ideally copying an existing synapse data directory that has been in use for awhile
• Start docker synapse container with EFS volume bind-mounted to /data

I didn't have any sort of log output until after this command finished, so it could have been much simpler to diagnose with a "changing ownership of synapse data files" message. Ideally this shouldn't block synapse from starting.

Version information

• Homeserver: corporate, internal homeserver
• Version: v0.33.7
• Install method: docker pull matrixdotorg:v0.33.7
• Platform: CoreOS, AWS, EFS

[djmaze]

I am not sure the chown has to be recursive at all.

• When starting the container for the first time, the data volume is empty.
• When starting it again, the data inside the volume has already been saved with the correct owner.

So as far as I can see, a chown is only needed on the first run, and it does not need to be recursive.

[richvdh]

@kaiyou any reason this is recursive?

Otherwise sounds plausible. A PR would be welcome.

[scjudd]

Is there a reason we're using su-exec in start.py as opposed to the USER directive in the Dockerfile? It seems to me that we could remove the chown entirely and just fail to start if we don't have proper permissions to the contents of our data volume, rather than assume we're being started by a user with permissions to run chown in the first place. Thoughts?

[djmaze]

It seems to me that we could remove the chown entirely and just fail to start if we don't have proper permissions to the contents of our data volume

This breaks e.g. docker swarm deployments (like mine) where volumes are automatically created through the stack definition. Manually creating a volume and adjusting its permissions on the server is not a good practice in my opinion.

Alternatively, creating the /data directory in the Dockerfile before declaring it as a volume could probably work! At least when used with named volumes as opposed to bind-mounted volumes.

[scjudd]

After iterating on this idea a few times in my head, I think the best solution is what you've suggested. We create /data with permissions correct in the Dockerfile, then declare it to be a volume. If something out-of-band happens to violate our assumptions about permissions, that should just fail the way it would if we weren't in a container. Does that feel right?

[djmaze]

In that case, we should add proper documentation including the command needed for preexisting data / bind-mounted volumes.

[richvdh]

This has hopefully been fixed by #5566.