Why am I experiencing DNS resolution failures only in the final stage of a multi-stage Docker Compose build?

[eljonny]

I am working on a custom deployment of some open source software where some of the code needs to be modified to provide a couple additional features that are not present, so I'm working on making modifications and rebuilding the main-app image.
I would ask this on the main repo (paperless-ngx) but I feel like this is more of a docker build issue than something specific to that project.
This is a project I'm working on in my spare time and I have been searching for a solution for this for the last 3 weeks.

The core problem is this:
When I run the docker multi-stage build, every stage but the final stage (called main-app) does not have DNS issues, but the final stage is not able to resolve deb.debian.org.

I cannot get Docker to build the main-app image on Linux (Xubuntu 24.04) but it works OK in Docker Desktop on Windows, which is bizarre and usually the case is the opposite, works right on Linux and not on Windows.

Docker versions:

• Docker: Docker version 28.2.2, build 28.2.2-0ubuntu1~24.04.1
• Docker Compose: Docker Compose version 2.37.1+ds1-0ubuntu2~24.04.1
• Docker Buildx: github.com/docker/buildx 0.21.3 0.21.3-0ubuntu1~24.04.1

Here is the Dockerfile (unmodified from the 2.17.1 source):

# syntax=docker/dockerfile:1
# https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md

# Stage: compile-frontend
# Purpose: Compiles the frontend
# Notes:
#  - Does PNPM stuff with Typescript and such
FROM --platform=$BUILDPLATFORM docker.io/node:20-bookworm-slim AS compile-frontend

COPY ./src-ui /src/src-ui

WORKDIR /src/src-ui
RUN set -eux \
  && npm update -g pnpm \
  && npm install -g corepack@latest \
  && corepack enable \
  && pnpm install

ARG PNGX_TAG_VERSION=
# Add the tag to the environment file if its a tagged dev build
RUN set -eux && \
case "${PNGX_TAG_VERSION}" in \
  dev|beta|fix*|feature*) \
    sed -i -E "s/tag: '([a-z\.]+)'/tag: '${PNGX_TAG_VERSION}'/g" /src/src-ui/src/environments/environment.prod.ts \
    ;; \
esac

RUN set -eux \
  && ./node_modules/.bin/ng build --configuration production

# Stage: s6-overlay-base
# Purpose: Installs s6-overlay and rootfs
# Comments:
#  - Don't leave anything extra in here either
FROM ghcr.io/astral-sh/uv:0.7.9-python3.12-bookworm-slim AS s6-overlay-base

WORKDIR /usr/src/s6

# https://github.com/just-containers/s6-overlay#customizing-s6-overlay-behaviour
ENV \
    S6_BEHAVIOUR_IF_STAGE2_FAILS=2 \
    S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0 \
    S6_VERBOSITY=1 \
    PATH=/command:$PATH

# Buildx provided, must be defined to use though
ARG TARGETARCH
ARG TARGETVARIANT
# Lock this version
ARG S6_OVERLAY_VERSION=3.2.1.0

ARG S6_BUILD_TIME_PKGS="curl \
                        xz-utils"

RUN set -eux \
    && echo "Installing build time packages" \
      && apt-get update \
      && apt-get install --yes --quiet --no-install-recommends ${S6_BUILD_TIME_PKGS} \
    && echo "Determining arch" \
      && S6_ARCH="" \
      && if [ "${TARGETARCH}${TARGETVARIANT}" = "amd64" ]; then S6_ARCH="x86_64"; \
      elif [ "${TARGETARCH}${TARGETVARIANT}" = "arm64" ]; then S6_ARCH="aarch64"; fi\
      && if [ -z "${S6_ARCH}" ]; then { echo "Error: Not able to determine arch"; exit 1; }; fi \
    && echo "Installing s6-overlay for ${S6_ARCH}" \
      && curl --fail --silent --no-progress-meter --show-error --location --remote-name-all --parallel --parallel-max 4 \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz.sha256" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_ARCH}.tar.xz" \
        "https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-${S6_ARCH}.tar.xz.sha256" \
      && echo "Validating s6-archive checksums" \
        && sha256sum --check ./*.sha256 \
      && echo "Unpacking archives" \
        && tar --directory / -Jxpf s6-overlay-noarch.tar.xz \
        && tar --directory / -Jxpf s6-overlay-${S6_ARCH}.tar.xz \
      && echo "Removing downloaded archives" \
        && rm ./*.tar.xz \
        && rm ./*.sha256 \
    && echo "Cleaning up image" \
      && apt-get --yes purge ${S6_BUILD_TIME_PKGS} \
      && apt-get --yes autoremove --purge \
      && rm -rf /var/lib/apt/lists/*

# Copy our service defs and filesystem
COPY ./docker/rootfs /

# Stage: main-app
# Purpose: The final image
# Comments:
#  - Don't leave anything extra in here
FROM s6-overlay-base AS main-app

LABEL org.opencontainers.image.authors="paperless-ngx team <hello@paperless-ngx.com>"
LABEL org.opencontainers.image.documentation="https://docs.paperless-ngx.com/"
LABEL org.opencontainers.image.source="https://github.com/paperless-ngx/paperless-ngx"
LABEL org.opencontainers.image.url="https://github.com/paperless-ngx/paperless-ngx"
LABEL org.opencontainers.image.licenses="GPL-3.0-only"

ARG DEBIAN_FRONTEND=noninteractive

# Buildx provided, must be defined to use though
ARG TARGETARCH

# Can be workflow provided, defaults set for manual building
ARG JBIG2ENC_VERSION=0.30
ARG QPDF_VERSION=11.9.0
ARG GS_VERSION=10.03.1

# Set Python environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    # Ignore warning from Whitenoise about async iterators
    PYTHONWARNINGS="ignore:::django.http.response:517" \
    PNGX_CONTAINERIZED=1 \
    # https://docs.astral.sh/uv/reference/settings/#link-mode
    UV_LINK_MODE=copy \
    UV_CACHE_DIR=/cache/uv/

#
# Begin installation and configuration
# Order the steps below from least often changed to most
#

# Packages need for running
ARG RUNTIME_PACKAGES="\
  # General utils
  curl \
  # Docker specific
  gosu \
  # Timezones support
  tzdata \
  # fonts for text file thumbnail generation
  fonts-liberation \
  gettext \
  ghostscript \
  gnupg \
  icc-profiles-free \
  imagemagick \
  # PostgreSQL
  postgresql-client \
  # MySQL / MariaDB
  mariadb-client \
  # OCRmyPDF dependencies
  tesseract-ocr \
  tesseract-ocr-eng \
  tesseract-ocr-deu \
  tesseract-ocr-fra \
  tesseract-ocr-ita \
  tesseract-ocr-spa \
  unpaper \
  pngquant \
  jbig2dec \
  # lxml
  libxml2 \
  libxslt1.1 \
  # itself
  qpdf \
  # Mime type detection
  file \
  libmagic1 \
  media-types \
  zlib1g \
  # Barcode splitter
  libzbar0 \
  poppler-utils"

# Install basic runtime packages.
# These change very infrequently
RUN set -eux \
  echo "Installing system packages" \
    && apt-get update \
    && apt-get install --yes --quiet --no-install-recommends ${RUNTIME_PACKAGES} \
    && echo "Installing pre-built updates" \
      && curl --fail --silent --no-progress-meter --show-error --location --remote-name-all --parallel --parallel-max 4 \
        https://github.com/paperless-ngx/builder/releases/download/qpdf-${QPDF_VERSION}/libqpdf29_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/qpdf-${QPDF_VERSION}/qpdf_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/libgs10_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/ghostscript_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        https://github.com/paperless-ngx/builder/releases/download/ghostscript-${GS_VERSION}/libgs10-common_${GS_VERSION}.dfsg-1_all.deb \
        https://github.com/paperless-ngx/builder/releases/download/jbig2enc-${JBIG2ENC_VERSION}/jbig2enc_${JBIG2ENC_VERSION}-1_${TARGETARCH}.deb \
      && echo "Installing qpdf ${QPDF_VERSION}" \
        && dpkg --install ./libqpdf29_${QPDF_VERSION}-1_${TARGETARCH}.deb \
        && dpkg --install ./qpdf_${QPDF_VERSION}-1_${TARGETARCH}.deb \
      && echo "Installing Ghostscript ${GS_VERSION}" \
        && dpkg --install ./libgs10-common_${GS_VERSION}.dfsg-1_all.deb \
        && dpkg --install ./libgs10_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
        && dpkg --install ./ghostscript_${GS_VERSION}.dfsg-1_${TARGETARCH}.deb \
      && echo "Installing jbig2enc" \
        && dpkg --install ./jbig2enc_${JBIG2ENC_VERSION}-1_${TARGETARCH}.deb \
      && echo "Configuring imagemagick" \
        && cp /etc/ImageMagick-6/paperless-policy.xml /etc/ImageMagick-6/policy.xml \
      && echo "Cleaning up image layer" \
        && rm --force --verbose *.deb \
    && rm --recursive --force --verbose /var/lib/apt/lists/*

WORKDIR /usr/src/paperless/src/

# Python dependencies
# Change pretty frequently
COPY --chown=1000:1000 ["pyproject.toml", "uv.lock", "/usr/src/paperless/src/"]

# Packages needed only for building a few quick Python
# dependencies
ARG BUILD_PACKAGES="\
  build-essential \
  # https://github.com/PyMySQL/mysqlclient#linux
  default-libmysqlclient-dev \
  pkg-config"

# hadolint ignore=DL3042
RUN --mount=type=cache,target=${UV_CACHE_DIR},id=python-cache \
  set -eux \
  && echo "Installing build system packages" \
    && apt-get update \
    && apt-get install --yes --quiet --no-install-recommends ${BUILD_PACKAGES} \
  && echo "Installing Python requirements" \
    && uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt \
    && uv pip install --system --no-python-downloads --python-preference system --requirements requirements.txt \
  && echo "Installing NLTK data" \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" snowball_data \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" stopwords \
    && python3 -W ignore::RuntimeWarning -m nltk.downloader -d "/usr/share/nltk_data" punkt_tab \
  && echo "Cleaning up image" \
    && apt-get --yes purge ${BUILD_PACKAGES} \
    && apt-get --yes autoremove --purge \
    && apt-get clean --yes \
    && rm --recursive --force --verbose *.whl \
    && rm --recursive --force --verbose /var/lib/apt/lists/* \
    && rm --recursive --force --verbose /tmp/* \
    && rm --recursive --force --verbose /var/tmp/* \
    && rm --recursive --force --verbose /var/cache/apt/archives/* \
    && truncate --size 0 /var/log/*log

# copy backend
COPY --chown=1000:1000 ./src ./

# copy frontend
COPY --from=compile-frontend --chown=1000:1000 /src/src/documents/static/frontend/ ./documents/static/frontend/

# add users, setup scripts
# Mount the compiled frontend to expected location
RUN set -eux \
  && sed -i '1s|^#!/usr/bin/env python3|#!/command/with-contenv python3|' manage.py \
  && echo "Setting up user/group" \
    && addgroup --gid 1000 paperless \
    && useradd --uid 1000 --gid paperless --home-dir /usr/src/paperless paperless \
  && echo "Creating volume directories" \
    && mkdir --parents --verbose /usr/src/paperless/data \
    && mkdir --parents --verbose /usr/src/paperless/media \
    && mkdir --parents --verbose /usr/src/paperless/consume \
    && mkdir --parents --verbose /usr/src/paperless/export \
  && echo "Creating gnupg directory" \
    && mkdir -m700 --verbose /usr/src/paperless/.gnupg \
  && echo "Adjusting all permissions" \
    && chown --from root:root --changes --recursive paperless:paperless /usr/src/paperless \
  && echo "Collecting static files" \
    && s6-setuidgid paperless python3 manage.py collectstatic --clear --no-input --link \
    && s6-setuidgid paperless python3 manage.py compilemessages

VOLUME ["/usr/src/paperless/data", \
        "/usr/src/paperless/media", \
        "/usr/src/paperless/consume", \
        "/usr/src/paperless/export"]

ENTRYPOINT ["/init"]

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=10s --retries=5 CMD [ "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000" ]

I have tried the following things:

• Disabled AppArmor at user level with the following command: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
• Disabled AppArmor at the kernel level, via grub using this script
• Configured the docker daemon with DNS and private network settings (change the network with the "bip" key)
• Configured the docker daemon with only DNS settings (see daemon config below)
• Created a new builder with the following command to make sure everything is using the docker bridge network: docker buildx create --platform 'linux/amd64' --use --name 'pngx-builder' --driver 'docker-container' --driver-opt 'network=bridge' --buildkitd-flags '--oci-worker-net bridge' --bootstrap
• Created another new builder with the following command to use direct host networking: docker buildx create --platform 'linux/amd64' --use --name 'pngx-builder-hn' --driver 'docker-container' --driver-opt 'network=host' --buildkitd-flags '--allow-insecure-entitlement network.host --oci-worker-net host' --bootstrap
• Run the build with --no-cache and cached mode
• Run the build with the basic command: docker compose build
• Run the build with a more complete command, that uses the non-default builder, to try to help debug and find the root cause of the issue: docker compose --all-resources --progress=plain build --no-cache --builder 'pngx-builder' --with-dependencies
• Run the build with a more complete command, that uses the non-default builder with direct host networking: docker compose --all-resources --progress=plain build --no-cache --builder 'pngx-builder-hn' --with-dependencies
• Run the more complete command on the default builder
• Run the build by itself outside docker compose
• Configure systemd-resolved to create an intermediate DNS resolution server between docker and host network within the private docker network
• Configure systemd-resolved with a set of known-good DNS servers (Cloudflare, Quad9, OpenDNS, Google, my local router which routes to Spectrum DNS)
• Configure systemd-resolved by disabling the stub resolver at 127.0.0.53 so everything just gets routed the way that it would normally without systemd-resolved
• Run docker buildx prune --all --force repeatedly
• Disable Docker Buildx and use the classic builder
  • It seems like DNS might work here but the build is not compatible for other reasons that I don't completely understand
  • I have to modify the Dockerfile here because the classic builder doesn't support multi-platform builds
• Restarting docker daemon (so many times)
• Ran cat /etc/resolv.conf in the Dockerfile RUN command in the main-app image and verified the name servers are present as they should be.
• Confirmed that UFW is disabled with command: sudo ufw status

This is my current daemon config (at /etc/docker/daemon.json):

{
    "dns": [
        "1.1.1.1",
        "1.0.0.1",
        "9.9.9.9",
        "208.67.222.222",
        "208.67.220.220"
    ]
}

This is the current configuration for systemd-resolved that disables the stub resolver (at /etc/systemd/resolved.conf.d/docker.conf):

[Resolve]
DNSStubListener=no

This is the current configuration for ensuring known-good DNS name servers are available through systemd-resolved (at /etc/systemd/resolved.conf.d/userdns.conf):

[Resolve]
#Cloudflare, Quad9, and OpenDNS
DNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001 9.9.9.9 149.112.112.112 2620:fe::fe 2620:fe::9 208.67.222.222 208.67.220.220
#Google, Spectrum
FallbackDNS=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844 192.168.1.1 2600:6c4e:4400:1950::1

Something is blocking DNS resolution in the final image stage in the multi-stage build (main-app), and I cannot for the life of me figure out what it is.
This is what happens for the big RUN command in main-app:

#17 8.967 W: Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease  Temporary failure resolving 'deb.debian.org'
#17 8.968 W: Failed to fetch http://deb.debian.org/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'deb.debian.org'
#17 8.968 W: Failed to fetch http://deb.debian.org/debian-security/dists/bookworm-security/InRelease  Temporary failure resolving 'deb.debian.org'

I have seen many mentions of issues like this on the Docker forums and in limited instances elsewhere, but nowhere have I found a concrete resolution for it.

I feel like I have tried everything, I hope someone here might be able to help. Here is the full build log.

Also posted/asked on SO.

[sunnypatell]

@eljonny this is a known buildkit bug with systemd-resolved: moby/buildkit#5009.

when buildx uses the docker-container driver, the buildkit container tries to read /run/systemd/resolve/resolv.conf for DNS config. that file doesn't exist inside the container's mount namespace, so buildkit falls back to google DNS (8.8.8.8) which may not be reachable on your network. the docker driver doesn't have this problem because it uses the daemon's own DNS resolution.

the reason earlier stages work is almost certainly layer caching from a previous successful build. run docker compose build --no-cache and you'll likely see DNS failures in ALL stages.

fix -- check which driver you're using and switch:

# check current builder
docker buildx ls

# if using docker-container, switch to the default docker driver
docker buildx use default

# rebuild
docker compose build --no-cache

if you need docker-container (e.g., multi-platform builds), mount the host's DNS config:

docker buildx create \
  --name fixed-builder \
  --driver docker-container \
  --driver-opt "network=host" \
  --bootstrap
docker buildx use fixed-builder

then build with host networking so DNS resolves correctly:

# docker-compose.yml
services:
  main-app:
    build:
      context: .
      network: host

nuclear option if nothing else works -- bypass systemd-resolved entirely:

sudo rm /etc/resolv.conf
echo -e "nameserver 1.1.1.1\nnameserver 9.9.9.9" | sudo tee /etc/resolv.conf

this forces real DNS servers instead of the 127.0.0.53 stub resolver that confuses buildkit. you can revert by running sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf.

[eljonny]

I already tried a builder with network=host, as indicated in the following bullet item from the list of things I've tried:

• Created another new builder with the following command to use direct host networking: docker buildx create --platform 'linux/amd64' --use --name 'pngx-builder-hn' --driver 'docker-container' --driver-opt 'network=host' --buildkitd-flags '--allow-insecure-entitlement network.host --oci-worker-net host' --bootstrap

I actually also tried completely disabling the stub resolver:

• Configure systemd-resolved by disabling the stub resolver at 127.0.0.53 so everything just gets routed the way that it would normally without systemd-resolved

So I've actually gone way further than your purported 'nuclear option' to try to resolve this.

[sunnypatell]

@eljonny fair enough, you clearly went deep on the system-level stuff already. let me pivot to something different.

the "only final stage fails" thing is almost certainly a caching illusion

the earlier stages (compile-frontend, s6-overlay-base) likely succeeded from cached layers of a previous build. if you run with --no-cache and watch --progress=plain output carefully, i bet DNS fails in s6-overlay-base too (or you'll see it pulling from cache even with --no-cache if the base image layers are still present). the multi-stage aspect is a red herring here.

the real issue: buildkit has its own DNS resolver that ignores daemon.json

this is the part that trips everyone up. the dns config in /etc/docker/daemon.json does NOT propagate to buildkit workers. buildkit resolves DNS independently using its own resolvconf logic. on systemd-resolved hosts, it tries to read /run/systemd/resolve/resolv.conf, which doesn't exist inside the buildkit container's mount namespace, so it silently falls back to google DNS (8.8.8.8). if google DNS is unreachable or slow on your network, you get the exact failures you're seeing.

this is tracked in moby/buildkit#5009 and related to moby/buildkit#2404.

things to try that you probably haven't:

1. check what DNS buildkit is actually using inside the build. add this temporarily to the failing RUN in main-app:

   RUN cat /etc/resolv.conf && set -eux \

   if it shows 8.8.8.8 / 8.8.4.4 instead of your configured servers, that confirms the buildkit resolver is ignoring your host config entirely.

2. use buildkitd.toml to set DNS at the buildkit level (not daemon.json, which buildkit ignores):

   [dns]
     nameservers = ["1.1.1.1", "9.9.9.9"]

   then create the builder with:

   docker buildx create --use --name pngx-dns \
     --driver docker-container \
     --driver-opt network=host \
     --buildkitd-config ./buildkitd.toml \
     --bootstrap

   this tells buildkit's own resolver what to use, bypassing the broken resolvconf detection logic. ref: buildkitd.toml docs

3. try the docker driver instead of docker-container for this build. the docker driver uses the daemon's built-in buildkit which inherits DNS from the daemon process directly (where your daemon.json dns config actually works). since you're building single-platform linux/amd64 anyway, you don't need the docker-container driver:

   docker buildx use default
   docker compose build --no-cache

   check docker buildx ls to confirm the default builder shows driver docker, not docker-container.

4. if you must use docker-container driver, mount the host's resolv.conf files into the buildkit container. the workaround from buildkit#5009:

   docker buildx create --use --name pngx-mounted \
     --driver docker-container \
     --driver-opt "network=host" \
     --buildkitd-flags "--oci-worker-net host" \
     --node pngx-mounted0
   
   # then manually add the mounts to the running buildkit container:
   docker stop buildx_buildkit_pngx-mounted0
   # recreate with explicit bind mounts for resolv.conf

   ...though honestly option 2 or 3 is cleaner.

5. also worth checking: you're using ubuntu-packaged docker (28.2.2-0ubuntu1~24.04.1) not docker's official packages. the ubuntu-packaged buildkit version bundled in the daemon may have different behavior than what the docker-container driver pulls. might be worth testing with docker's official apt repo to rule out packaging differences.

the tl;dr is that daemon.json dns config is a docker daemon thing, not a buildkit thing. buildkit has its own broken-on-systemd-resolved DNS detection, and the fix is either telling buildkit directly via buildkitd.toml, or using the docker driver where the daemon handles DNS.

[eljonny]

I'll check it out. Thanks for the additional details and suggestions.

[eljonny]

I never figured out what the problem was, but I suspect it's some emergent interaction between docker, compose, buildx, and the awful, debilitating coupling of Ubuntu, Snap, and AppArmor.

I removed Xubuntu 24.04 and installed Arch and the DNS resolution failures no longer occur, for the exact same code.
Docker versions I am now using, and without Snap and AppArmor, and no issues are present:

Docker version 29.2.1, build a5c7197d72
Docker Compose version 5.0.2
github.com/docker/buildx 0.31.1 a2675950d46b2cb171b23c2015ca44fb88607531