Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Complement testing is slow when building all of Synapse #9364

Closed
MadLittleMods opened this issue Feb 9, 2021 · 0 comments · Fixed by #9610
Closed

Complement testing is slow when building all of Synapse #9364

MadLittleMods opened this issue Feb 9, 2021 · 0 comments · Fixed by #9610
Labels
T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

Comments

@MadLittleMods
Copy link
Contributor

Is there a way to speed up this part of the Docker build? Anytime you edit something in Synapse and run the complement script, this part takes 50 seconds to run:

synapse/docker/Dockerfile

Lines 54 to 55 in eec9ab3

RUN pip install --prefix="/install" --no-warn-script-location \
/synapse[all]

Split out from #9247 (comment)
@MadLittleMods MadLittleMods added the T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. label Feb 9, 2021
@MadLittleMods MadLittleMods mentioned this issue Feb 9, 2021
Merged
9 tasks
@MadLittleMods MadLittleMods mentioned this issue Mar 13, 2021
Merged
4 tasks
MadLittleMods added a commit that referenced this issue Mar 13, 2021
@MadLittleMods
Make pip install faster for Complement testing
1846198 
Fix #9364

Using BuildKit caches,
 - https://stackoverflow.com/a/58021389/796832
 - https://gist.github.com/RainJayTsai/16979ee1687546f93827ba78e5657196
anoadragon453 pushed a commit that referenced this issue Mar 26, 2021
@MadLittleMods
Make pip install faster in Docker build for Complement testing (#9610)
0a778c1 
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.

If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘

As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.

Fix #9364
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make pip install faster in Docker build for Complement testing matrix-org/synapse
1 participant
@MadLittleMods