Docker images contain 20MB of deleted /var/lib/apt/lists/ files

[edmorley]

Hi! Before I dive in, I just want to say thank you for maintaining these images :-)

So I happened to notice this section present in the Dockerfiles (generated here):

# delete all the apt list files since they're big and get stale quickly
RUN rm -rf /var/lib/apt/lists/*
# this forces "apt-get update" in dependent images, which is also good

I agree it's a good idea to remove these files to force later "apt-get update", however the comment about saving space is not correct, since deleting files in a layer after they've already been added won't free up the space. The comment seems to have been copy-pasted from this script (which isn't run across multiple layers so actually does save space).

Rather than just correcting the comment, it would be best to avoid the 20MB wasted space in the first place.

The files in /var/lib/apt/lists/ come from the base image archive from Canonical, which is directly extracted using the ADD command's tar file support. This cannot be switched to the curl/untar/delete pattern used in downstream images, since until the base archive is extracted there are no binaries in the image to use. As such, the removal of /var/lib/apt/lists/ needs to occur prior to the Docker build process.

This example shows the Ubuntu 16.04 image being reduced from 118MB to 97.6MB by doing exactly that...

#!/bin/bash

# Fetch base archive and Dockerfile used for the existing Ubuntu 16.04 image
curl -fLO https://partner-images.canonical.com/core/xenial/current/ubuntu-xenial-core-cloudimg-amd64-root.tar.gz
curl -fLO https://raw.githubusercontent.com/tianon/docker-brew-ubuntu-core/dist-amd64/xenial/Dockerfile

# Prepare a slimmed down version
gzip -dc ubuntu-xenial-core-cloudimg-amd64-root.tar.gz | tar --delete --wildcards 'var/lib/apt/lists/*' | gzip > rootfs-minimised.tar.gz
sed 's/ubuntu-xenial-core-cloudimg-amd64-root\.tar\.gz/rootfs-minimised\.tar\.gz/' Dockerfile > Dockerfile-new

# Compare the before/after
docker build -t ubuntu-16.04-test:before .
docker build -t ubuntu-16.04-test:after -f Dockerfile-new .
docker images ubuntu-16.04-test

Output:

REPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE
ubuntu-16.04-test   after               7b258205a6b1        Less than a second ago   97.6MB
ubuntu-16.04-test   before              65cb86c05710        13 seconds ago           118MB

I guess the question will be whether to store both the original base archive and the processed one in this repo (so people can still use the hashes and compare), or whether to just store the processed one.

Also, I think it's worth pushing the upstream maintainers of these base images to remove the APT lists from them, which will avoid all of this busywork. Perhaps this size-reduction use-case is a more compelling one for them than that outlined here:
https://bugs.launchpad.net/cloud-images/+bug/1685399

Many thanks!

[tianon]

Unfortunately, repacking the tarballs is out of the question, so the best we can do here is fix the comment and file an issue upstream (perhaps linking to the upstream issue in our comment?) If this does get fixed upstream, we can leave our "RUN" line in place as a fallback, since it will then be a no-op zero layer and coalesce just like "ENV", etc metadata layers do. 👍 (especially since the upstream tarballs have flip-flopped on this a few times in the past, IIRC) Thanks for the detailed report!

[edmorley]

Thank you for the reply!

Unfortunately, repacking the tarballs is out of the question

Out of curiosity, could you elaborate? Is it for reproducibility?

the best we can do here is fix the comment and file an issue upstream (perhaps linking to the upstream issue in our comment?)

Sounds good on both counts! Would you mind filing upstream? (I'd have to create a launchpad account just for this)

If this does get fixed upstream, we can leave our "RUN" line in place as a fallback,
... (especially since the upstream tarballs have flip-flopped on this a few times in the past, IIRC)

I think it would be good to make upstream regressions for this more obvious. So perhaps either:

• using rm -rfv /var/lib/apt/lists/* (ie with the verbose flag, which will make it clear from the logs whether anything was deleted)
• replacing the rm entirely with an assertion that the directory is empty. (In this case, if upstream did regress, the workflow would be to file an upstream issue, replace the assertion with the rm again and wait for upstream to fix before reverting back.)

[tianon]

Out of curiosity, could you elaborate? Is it for reproducibility?

Partially for reproducibility, but more because Canonical has asked to be the "source of truth" for the bits that become "Ubuntu" on the Docker Hub (I'm merely a proxy for their builds).

Sounds good on both counts! Would you mind filing upstream? (I'd have to create a launchpad account just for this)

Yeah, I'll go ahead and file a launchpad issue -- having this directory be explicitly empty by design in the tarballs they provide isn't something they'd ever committed to doing, so I didn't want to assume that it was a "bug" (I'm not sure 100% whether we're the only consumers of these particular tarballs), but if we file an issue and they're warm to the idea (and modify their tarballs), then I'd be more amenable to making it an assertion of emptiness instead. 👍

[tianon]

Issue filed: https://bugs.launchpad.net/cloud-images/+bug/1699913

[edmorley]

Thank you for filing that. Since there has been no response to the ticket, I was going to register for a launchpad account and leave another comment explaining further, however my attempts to log in using a newly created Ubuntu One account failed with "Oops! Sorry, something just went wrong in Launchpad." :-(

[edmorley]

Is there anyone we can ping to get some movement on that launchpad ticket?

[Foorack]

@edmorley I was able to login to Launchpad with my Ubuntu One account today. I recommend trying again and see if it works now. 👍

[edmorley]

Ah I forgot to update the above - someone kindly helped fixed the issue with my account since then :-)

[kulprasanna]

Hi All. My client is implementing a product whose images are based on ubunty 17.0.4 docker image. This image has vulnerabilities. We are asked to get rid of all vulnerabilities and have hit a roadblock. I see that the vulnerabilities are related to the packages such as ncurses 6.0 that our application does not require. How could we remove these packages? Anybody trying the same thing?

[tianon]

@kulprasanna please do not post unrelated topics to existing issues 😉

if the relevant packages have not been updated in Ubuntu, there's nothing we can do (which is the case with the current Zesty image)

even "apt-get dist-upgrade" has no changes in our current image (sha256:fe6cc21b9a65b07053cb2614f1ed8217a92cde64d0030d122e19d54881b6cb3e)

See also docker-library/official-images#2740 for an example where @yosifkit did a deep dive into remaining reported CVEs on an image in the past.

[edmorley]

Does anyone have any contacts at Ubuntu who might be able to help get some traction in the launchpad issue?
https://bugs.launchpad.net/cloud-images/+bug/1699913

[pranas]

Looking forward to Ubuntu taking this stuff out of their tarball. Until that happens, a new --squash docker build option would be a good workaround to get rid of it in the meantime. Or the multi stage build..

[mickare]

I support @pranas suggestion. A simple multi stage build does save ~26MB.
No "repacking" needed and removed artifacts are "cleared".

Btw. I don't see any drawbacks of a single-layer base image. 🤔