Docker latest stuck on RabbitMQ startup, direct rabbitmq-server works

[fayanimous]

Hi,

I think there is an issue with the current ghcr.io/euro-office/documentserver:latest image.

I’m running it with Docker Compose for Nextcloud integration.

The container starts PostgreSQL correctly, then gets stuck here:

Starting PostgreSQL 16 database server
...done.
Postgres schema already present, skipping.
Starting RabbitMQ Messaging Server rabbitmq-server

After that, nothing else starts.

The healthcheck fails with:

curl: (56) Recv failure: Connection reset by peer

I checked the container and start-stop-daemon stays at almost 100% CPU:

start-stop-daemon --quiet --chuid rabbitmq \
  --start --exec /usr/sbin/rabbitmq-server \
  --pidfile /var/run/rabbitmq/pid --background

At this point RabbitMQ is not really started:

rabbitmqctl status

fails, and epmd -names only shows the rabbitmq CLI node.

No listener on 5672.

The interesting part is that RabbitMQ itself works fine if I start it manually inside the container:

su rabbitmq -s /bin/sh -c "/usr/lib/rabbitmq/bin/rabbitmq-server"

or:

su rabbitmq -s /bin/sh -c \
"RABBITMQ_PID_FILE=/var/run/rabbitmq/pid /usr/lib/rabbitmq/bin/rabbitmq-server -detached"

Then:

rabbitmqctl status

works correctly and ports 5672 and 25672 are listening.

I checked /entrypoint.sh and it currently starts RabbitMQ with:

service rabbitmq-server start

This call seems to be the part that hangs.

As a workaround, I built a small custom image where I replaced:

service rabbitmq-server start

with a direct detached RabbitMQ startup, then wait for:

rabbitmqctl status

before continuing.

With this workaround the container starts normally:

Starting RabbitMQ directly
...done.
Starting redis-server: redis-server.
Starting nginx nginx
...done.

and later:

converter entered RUNNING state
docservice entered RUNNING state

The healthcheck also works again:

HTTP/1.1 200 OK

true

Image used:

ghcr.io/euro-office/documentserver:latest

Created:

2026-08-04T08:22:40.568547219Z

Digest:

sha256:6068075cc247f9283d337b2cf9a105d9df49fcd8cd6b5e82fa90640df1552e80

I also noticed RabbitMQ data already present in the image under:

/var/lib/rabbitmq/mnesia/rabbit@buildkitsandbox

I don’t know if it is related, but I thought it was worth mentioning.

Let me know if you want more logs or tests.

[github-actions]

@chrip @DmySyz Please triage this community-created issue within 48 hours.

[chrip]

Thanks for the unusually thorough report — your diagnosis is correct, and the workaround
you built is essentially the right fix.

Confirmed regression

The rabbitmq start line changed in #249 (merged 2026-07-22):

-  runuser -u rabbitmq -- rabbitmq-server -detached
+  service rabbitmq-server start

v9.3.2 (released 2026-06-25) predates that merge; v9.3.3 (2026-08-04) and :latest
contain it. So your bisect holds, and your custom image is effectively a revert of it.

One thing that doesn't add up yet, and where I could use your help

Our CI boots the standalone image with a plain docker run -d and polls /healthcheck
on every push to main, and it has passed on every run since #249 landed — nginx and
docservice start after rabbitmq in the entrypoint, so a hang there would fail the job.
service rabbitmq-server start also runs during the image build itself and succeeds.
So the call is not hanging everywhere; something about the host is triggering it.

start-stop-daemon sitting at ~100% CPU is the strongest clue. With --background it
closes file descriptors up to the process limit, which turns into an effectively infinite
spin when RLIMIT_NOFILE is very large — a common default on some Docker/containerd
hosts. Could you run:

docker exec <container> sh -c 'grep -i "open files" /proc/1/limits; ulimit -n'
docker info --format '{{.ServerVersion}}'
grep -rs LimitNOFILE /lib/systemd/system/docker.service /etc/systemd/system/docker.service.d/

If the nofile limit comes back in the millions, that confirms it. As a quick test you can
also add this to your compose service and check whether stock :latest boots unmodified:

    ulimits:
      nofile:
        soft: 65536
        hard: 65536

We're fixing it either way

Regardless of what triggers it, service rabbitmq-server start is the wrong primitive
here: it gives us no readiness guarantee and it can block the entrypoint indefinitely with
no diagnostic. We'll go back to a direct detached start followed by a bounded
rabbitmqctl await_startup wait that fails loudly on timeout, so this surfaces as an
error instead of silently wedging the container.

Your mnesia observation is a real find too

/var/lib/rabbitmq/mnesia/rabbit@buildkitsandbox is baked into the image because the
Dockerfile starts rabbitmq at build time to create the database. At runtime the node name
follows the container hostname, so that directory is never read — dead weight rather than
a cause, but we'll clean it up.

Note on #314

That one is also a v9.3.2 → v9.3.3 upgrade failure, but its cause is a shifted
postgres UID against a pre-existing data volume, and it needs static UID/GID pinning.
Since rabbitmq starts fine for you when you run it manually as the rabbitmq user,
ownership isn't your problem here. Keeping the two separate so neither fix masks the other.

Labelling as a docker bug and taking this.

Assisted-by: ClaudeCode:claude-opus-5

[fayanimous]

Thanks a lot for the detailed explanation and for looking into this so quickly.

I ran the tests you suggested and your RLIMIT_NOFILE theory seems to be correct.

On the affected host, the default limit is:

Max open files            1073741816           1073741816           files
ulimit -n
1073741816

Docker version:

26.1.5+dfsg1

I also checked for a LimitNOFILE override in the Docker systemd configuration and nothing is configured there.

I then switched back to the stock:

ghcr.io/euro-office/documentserver:latest

and added:

ulimits:
  nofile:
    soft: 65536
    hard: 65536

Inside the container the limit is now correctly set to:

Max open files            65536                65536                files
65536

And with that change, the stock :latest image boots normally again:

Starting PostgreSQL 16 database server
...done.
Postgres schema already present, skipping.
Starting RabbitMQ Messaging Server rabbitmq-server
...done.
Starting redis-server: redis-server.
Starting nginx nginx
...done.
Generating AllFonts.js, please wait...Done

So it really looks like the very high RLIMIT_NOFILE on this host is what triggers the start-stop-daemon --background issue.

Thanks again for the quick response and for confirming the regression. I really appreciate the explanation and the fact that you're planning to improve the RabbitMQ startup logic upstream.

If you need me to test a new image or proposed fix on this host, I'd be happy to do so.

[knarxy]

I can only confirm what @fayanimous has stated. I faced the same issue, confirmed the same high ulimit -n as him and adding

ulimits:
  nofile:
    soft: 65536
    hard: 65536

to the docker-compose.yml fixed it.

Same docker version as well:
26.1.5+dfsg1