Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "realistic bash user" (remove COPY layer in Dockerfile) #12

Merged
merged 4 commits into from
Mar 29, 2020

Conversation

seriema
Copy link
Owner

@seriema seriema commented Mar 28, 2020

COPY is used for "realistic bash user" (92f5129), but COPY comes with two main issues that made me decide to revert its use: Cache invalidation, and Windows issues. The former being the deciding factor, as the later is fixed. The long build times are not worth it.

Since this COPY is for the user creation it's very early in the Dockerfile, and even with a fat cache layer before it (experimental in docker-multiarch) it still adds a 20min rebuild locally way too often.

Cache invalidation

COPY has a tendency to assume that the file changed and invalidating all the following cache layers. Anything in the build context could invalidate it, even if it wasn't that file. I noticed:

  • .dockerignore was including the docker/ directory, and in the docker-multiarch branch it included build logs.
  • git on Windows keeps touching the line endings, and while docker/ is configured with .gitattributes to always use LF it still ended up with CRLF on multiple occasions.

On Docker Hub it's skipping the cache almost every time. For reference:
✔ a build without the COPY step only took 5 minutes as it ran purely from cache - it usually takes 2-3 hours by always rebuilding from scratch.
❌ the build for this PR ran without cache, twice, which took hours - first the regular build, then the Autotest rebuilt the image from scratch. 🤯

Windows issues

COPY causes two specific issues when building on Windows:

  • It will use the build OS line endings, which is CRLF on Windows.
    • Configured .gitattributes to use LF (2ceb265), despite this the files would sometimes still be in CRLF.
  • Always copies everything with execute permissions.
    • Added a Windows specific layer for removing those permissions. (49c317c)

While those issues are fixed, it's not worth keeping due to the build times.

Conclusion

Note: This will revert the "realistic bash user" improvements introduced in PR #8, e.g. colored prompt, etc, that are found on a real RPi with RetroPie.

It might still be achievable with RetroPie packages such as bashwelcometweak, and there are ways to make COPY more stable. So I might revisit "realistic bash user" at a later point.

Some references:

This PR is just for convenience in the future to see what had to change in the scripts to support the merged feature.

seriema added 4 commits March 28, 2020 11:30
Even without COPY (or ADD) in the Dockerfile it could be added again in
the future. There could also be other currently unknown reasons that
would cause images built on Windows to have the wrong line endings.
@seriema seriema merged commit 3132ccf into develop Mar 29, 2020
@seriema seriema deleted the docker-remove-copy-step branch March 29, 2020 07:59
@seriema seriema mentioned this pull request Apr 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant