Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache volumes created by Pack are not cleaned up by libcnb-test #570

Closed
edmorley opened this issue May 12, 2023 · 1 comment · Fixed by #741
Closed

Cache volumes created by Pack are not cleaned up by libcnb-test #570

edmorley opened this issue May 12, 2023 · 1 comment · Fixed by #741
Assignees
@edmorley
Labels
bug Something isn't working libcnb-test

Comments

@edmorley
Copy link
Member

edmorley commented May 12, 2023 •
edited
Loading

I recently ran into buildpack IO errors during a pack build that turned out to be due to disk space exhaustion on the macOS Docker Desktop created VM - caused by leftover Docker volumes used by Pack for caching.

There were ~800 named Docker volumes that consumed 40GB of disk space, of form:

$ docker volume ls
DRIVER    VOLUME NAME
local     pack-cache-library_libcnbtest_abnaipzkqlhfeopwdzgpcmwnldonle_latest-ce76e11c69c4.build
local     pack-cache-library_libcnbtest_abnaipzkqlhfeopwdzgpcmwnldonle_latest-ce76e11c69c4.launch
local     pack-cache-library_libcnbtest_actqilpclbkigdmjhdrhgnuwgealsm_latest-a44e7557da8c.build
local     pack-cache-library_libcnbtest_actqilpclbkigdmjhdrhgnuwgealsm_latest-a44e7557da8c.launch
local     pack-cache-library_libcnbtest_actyjtmpayftrpdhfehpwefuffycaf_latest-d0ec397b5241.build
local     pack-cache-library_libcnbtest_actyjtmpayftrpdhfehpwefuffycaf_latest-d0ec397b5241.launch
...

For normal (non-integration testing) usage of Pack CLI, one wouldn't end up with hundreds of these volumes, since caches are re-used/replaced for the same output image name. The reason we end up with hundreds when using libcnb-test is that it uses a random image name each time (which is the right thing for it to do), but whilst it deletes the images at the end of each test, it doesn't currently delete the image's associated Pack cache volumes.

I've started a discussion upstream about some potential improvements upstream (for example, adding pack cache prune commands or similar, and also asking why Pack is creating these volumes for layers that also exist in the final app image):
https://cloud-native.slack.com/archives/C0331B61A1Y/p1683817627443339

However, libcnb-test will need to handle this even with those improvements upstream.

To avoid this issue, libcnb-test can use the new --cache argument to pack build, and either:

  1. Continue to use volume-based caching, but pass in an explicit (and random) volume name, that libcnb-test can then remove when TestContext is dropped. (For example: --cache='type=build;format=volume;name=<volume-name>;type=launch;format=volume;name=<volume-name>')
  2. Or, switch to bind mount based caching, and pass in a temporary directory as the cache location, that libcnb-test can then remove when TestContext is dropped. (For example: --cache='type=build;format=bind;source=<a-tmp-dir>;type=launch;format=bind;source=<a-tmp-dir>')

GUS-W-13195631.
@edmorley edmorley self-assigned this Oct 24, 2023
@edmorley edmorley added the bug Something isn't working label Oct 24, 2023
@edmorley
Copy link
Member Author

I have a WIP for this locally - I'll try and find time to clean it up and open a PR soon.

This was referenced Nov 15, 2023
Closed
Merged
edmorley added a commit that referenced this issue Nov 16, 2023
@edmorley
libcnb-test: Test container, volume and image cleanup in CI (#740)
9b105dd 
At the end of each test libcnb-test is meant to clean up any Docker
resources that were created during the test. However, this functionality
currently isn't tested, and we've found several cases where cleanup
wasn't occurring as expected:
- #586
- #570
- #737

We could try and add a Rust integration test or two to check that
cleanup works in the common case, however, as seen by the issues
above, it's often in the edge cases where resource cleanup wasn't
working. In addition, testing cleanup during an individual test is hard,
since other tests will be running at the same time, so we'd have to
check for individual image or container names, which would miss
cases like #570.

As such, it's much easier for us to instead just check that there are
zero unexpected resources left behind at the end of the existing entire
libcnb-test test suite.

For now, two of the checks don't cause CI to fail, since the fixes for
the cleanup implementations have not yet landed.

GUS-W-14502524.
@edmorley edmorley mentioned this issue Nov 16, 2023
Merged
edmorley added a commit that referenced this issue Nov 17, 2023
@edmorley
libcnb-test: Clean up Docker volumes after each test (#741)
50b7a96 
When using `pack build`, by default Pack will create two Docker
volumes for caching - one for the build layer cache, and the other
for the launch layer cache. Pack bases the names for these volumes
on the app image name, so that the next time a build is performed
with the same app image name, the associated cache is used for
that rebuild.

When libcnb-test runs tests, it intentionally uses a random Docker
image name for each new test, and then tears down the image and
any associated containers after the tests.

However, until now the Docker volumes automatically created by
Pack were not being removed, which can cause issues when
running the tests locally, since the Docker VM can run out of disk
space - as seen in #570. The leftover volumes can also be seen
in the tests log output added in #740.

Pack doesn't currently offer a "clean up the volumes associated
with this app name" feature, so we have to remove these volumes
manually. To do so we need the volume name - which we achieve
by generating a name ourselves, and passing it to `pack build`
using the `--cache` options, so Pack uses our volume name instead
of generating its own automatic volume name.

Since we're now cleaning up not only images but volumes too, I've
refactored the way resources are cleaned up, to reduce the amount
of duplication (since otherwise we'd need to handle the "still clean up
for expected failures" case for volumes too).

This new implementation also solves #735 and #736.

Fixes #570.
Fixes #735.
Fixes #736.
GUS-W-13195631.
GUS-W-14503186.
GUS-W-14503192.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edmorley edmorley

Labels
bug Something isn't working libcnb-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

libcnb-test: Clean up Docker volumes after each test heroku/libcnb.rs
1 participant
@edmorley