New image structure based on design document

[humitos]

Dockerfile(s) required to build the new build images (ubuntu20-*) that use asdf to install languages requirements (python, node, rust, go)

Design documents:

• Design document for new Docker images structure readthedocs.org#7566
• Docker build images: update design doc readthedocs.org#8447

Implementation:

• Build: use new Docker images from design document readthedocs.org#8453

How to build this image

1. Clone this PR
2. run docker build -t readthedocs/build:ubuntu-20.04 .

TODO

• update tests to check asdf is installed correctly
• remove tests for Python versions
• add test to check that it's not removing any packages by mistake with apt-get (see libmysqlclient-dev is not installed? #158)
• decide if jsdoc and typedoc are required in the new images and in that case, how to handle them (see Support jsdoc and typedoc doctools #147)

Closes #170
Closes #60
Closes #158
Closes #107
Closes #90
Closes #86
Closes #108
Closes #91
Closes #64
Closes #130
Closes #177
Closes #176
Closes #48

[astrojuanlu]

I wonder if it would be a good moment to try to optimize the Dockerfile: https://www.fromlatest.io/

in particular:

• Couple apt-get update with rm -rf /var/lib/apt/lists/* in the same layer (although I don't think that plays along well with our multi-layered apt-get install - we would need to do update && install && rm -rf /var/lib/... in every layer, which will slow down the builds a bit)
• Use --no-install-recommends to be more explicit about what are we installing and why

[humitos]

Good feedback here!

Couple apt-get update with rm -rf /var/lib/apt/lists/* in the same layer (although I don't think that plays along well with our multi-layered apt-get install - we would need to do update && install && rm -rf /var/lib/... in every layer, which will slow down the builds a bit)

docker run -u root -it readthedocs/build:ubuntu20 /bin/bash
root@42d3d0f8fe1a:/home/docs# du -sh /var/lib/apt/lists/
29M	/var/lib/apt/lists/

I don't think we are spending a lot of disk space. However, I agree it could be a good optimization. As you already noticed, we will need to call apt-get update on each apt-get install command that we run.

On the other hands, packages (.deb files) are cleaned up automatically when using official Ubuntu images because they comes with a post-install hook for this:

docs@927b877e91c3:~$ cat /etc/apt/apt.conf.d/docker-clean
DPkg::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };
APT::Update::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };
Dir::Cache::pkgcache ""; Dir::Cache::srcpkgcache "";
docs@927b877e91c3:~$

Use --no-install-recommends to be more explicit about what are we installing and why

I'm not sure about this one. I think it's a good idea to know exactly what we are installing. However, if add this option now, I wonder if all the PDFs will keep building correctly (e.g. the right font, or stylesheets).

[agjohnson]

So much red!

I don't quite follow the need for additional version information on the build however, it seems like we only wanted a ubuntu20 and ubuntu22 image. The additional granularity probably won't be used anywhere.

[humitos]

@agjohnson I addressed all your comments and made the tests run on CircleCI. Is there any blocker to merge this PR?

[ericholscher]

This looks like a good step forward to me. I didn't look super deeply into the DockerFile packages that are installed, but assume we've tested those.

[ericholscher]

Can we automate this with date or similar? Do we want to?

[humitos]

I haven't try that. Honestly, I don't think LABEL version is strictly needed; but I thought it could be useful to use with docker inspect to know what's the version you have downloaded:

▶ docker inspect c54fcbdfb6c8 | jq '.[0].Config.Labels.version'
"ubuntu-20.04-2021.09.23"

It's a nice to have to me. If we can automate it, it would be 💯

[ericholscher]

Are these not still important to document, since they are still in use?

[humitos]

This was a little (too much) opinionated 😄

• I think we shouldn't explain these images here "to the final user"
• I think users wanting to use these images should be pointed to the Read the Docs documentation instead (config file)
• I would try to keep this repository hidden from the regular users because it only generates confusion
• I wanted to only expose the newer image to avoid people finding this and using the old images

That said, it may make sense to remove the phrase "Available for public usage as build.os: ubuntu-20.04" as well.

[ericholscher]

I would try to keep this repository hidden from the regular users because it only generates confusion

Perhaps we should make it private, then? And move these docs to our public-facing docs & config file?

[humitos]

Make it private may be too much. I think we will lose some issues/prs references linked everywhere.

Definitely, how to use the docker image and what it contains should be documented in the Read the Docs documentation for the config file.

[pradyunsg]

Hurray! 🎉