Cannot mount redpanda.yaml file in docker

[genzgd]

Version & Environment

Redpanda version: 24.2.1

Please also give versions of other components:

• Operating System: Mac M1 (Sonoma 14.5), Ubuntu (Github Runner)
• Docker Desktop: 4.33.0
• Docker Engine: 27.1.1

What went wrong?

We're using docker to mount a redpanda.yaml file from the local file system into a redpanda container at /etc/redpanda. When we try to start the container, it fails due to lack of permission to write a temp file. Note that our (test) configuration does not use the default 9092 port for Kafka.

unable to chown temp config file: chown /etc/redpanda/redpanda-1003971979532079255

If we use the --config option to point to a different directory (i.e., not /etc/redpanda), everything mostly works, but rpk acl commands fail with a connection refused to 9092, instead of the custom configured port.

What should have happened instead?

Either redpanda should create its temporary configuration "somewhere else" or rpk should be smart enough to read a custom configuration file path to get the correct "Kafka" api port.

How to reproduce the issue?

1. Create a docker configuration and mount a redpanda.yaml file in /etc/redpanda. We do this through a test framework and the GoLang docker client rather than docker compose files.

Essentially, we'd really like to load a redpanda.yaml configuration file into docker/docker compose that actually worked, instead of setting every configuration value as a start flag. If there's some other way to do that I'm missing then I'd be happy to know the workaround.

JIRA Link: CORE-6810

[genzgd]

It also happens in Ubuntu (github workflow runners).