[Bug]: Cannot run kafka container

[ronimizy]

Testcontainers version

4.6.0

Using the latest Testcontainers version?

Yes

Host OS

macOS

Host arch

ARM

.NET version

9.0.203

Docker version

Client:
 Version:           27.4.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        bde2b89
 Built:             Sat Dec  7 10:35:43 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.37.1 (178610)
 Engine:
  Version:          27.4.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       92a8393
  Built:            Sat Dec  7 10:38:33 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/george/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     /Users/george/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     /Users/george/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/george/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/george/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/george/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/george/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/george/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/george/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/george/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/george/.docker/cli-plugins/docker-scout

Server:
 Containers: 8
  Running: 6
  Paused: 0
  Stopped: 2
 Images: 15
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 3
 Total Memory: 3.828GiB
 Name: docker-desktop
 ID: 39675418-d776-4f5a-8775-8a7ab970d5cf
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/george/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

1. .Kafka package uses outdated kafka version (not present in docker registry)
2. when switching to latest, container failes to start zookeeper (can't find script to start it)

Relevant log output

2025-06-24 20:34:36 /testcontainers.sh: line 5: zookeeper-server-start: command not found
2025-06-24 20:34:37 KAFKA_PROCESS_ROLES is required.
2025-06-24 20:34:37 Command [/usr/local/bin/dub ensure KAFKA_PROCESS_ROLES] FAILED !
2025-06-24 20:34:36 ===> User
2025-06-24 20:34:36 uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
2025-06-24 20:34:36 ===> Configuring ...

Additional information

No response

[HofmeisterAn]

1. [...] not present in docker registry

What are you referring to: confluentinc/cp-kafka:6.1.9?

2. when switching to latest, container failes to start zookeeper (can't find script to start it)

7.2.15 still runs with our tests. From version 8 onwards, it looks like configurations are missing and the version is incompatible.

[ronimizy]

running these tests locally produces this output (hence my statement about image not being present in docker registry)

Docker.DotNet.DockerImageNotFoundException: Docker API responded with status code=NotFound, response={"message":"No such image: confluentinc/cp-kafka:...

Docker.DotNet.DockerImageNotFoundException
Docker API responded with status code=NotFound, response={"message":"No such image: confluentinc/cp-kafka:6.1.9"}

   at Docker.DotNet.ContainerOperations.<>c.<.cctor>b__30_1(HttpStatusCode statusCode, String responseBody) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 17
   at Docker.DotNet.DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers) in /_/src/Docker.DotNet/DockerClient.cs:line 485
   at Docker.DotNet.DockerClient.MakeRequestAsync[T](IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token) in /_/src/Docker.DotNet/DockerClient.cs:line 246
   at Docker.DotNet.ContainerOperations.CreateContainerAsync(CreateContainerParameters parameters, CancellationToken cancellationToken) in /_/src/Docker.DotNet/Endpoints/ContainerOperations.cs:line 54
   at DotNet.Testcontainers.Clients.DockerContainerOperations.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /Users/george/Documents/dotnet/testcontainers-dotnet/src/Testcontainers/Clients/DockerContainerOperations.cs:line 225
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct) in /Users/george/Documents/dotnet/testcontainers-dotnet/src/Testcontainers/Clients/TestcontainersClient.cs:line 339
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct) in /Users/george/Documents/dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 462
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct) in /Users/george/Documents/dotnet/testcontainers-dotnet/src/Testcontainers/Containers/DockerContainer.cs:line 306
   at Testcontainers.Kafka.KafkaContainerTest.InitializeAsync() in /Users/george/Documents/dotnet/testcontainers-dotnet/tests/Testcontainers.Kafka.Tests/KafkaContainerTest.cs:line 9
   at Xunit.v3.XunitTestRunnerBase`2.CreateTestClassInstance(TContext ctxt) in /_/src/xunit.v3.core/Runners/XunitTestRunnerBase.cs:line 62
   at Xunit.v3.ExceptionAggregator.RunAsync[T](Func`1 code, T defaultValue) in /_/src/xunit.v3.core/Exceptions/ExceptionAggregator.cs:line 146

seems like it is the issue with macOS and ARM since the image being used does not have a linux/arm64 platform. i previously used latest tag for image (that config part was added long time ago, but i assume that it was in attempt to solve this exact issue).

running tests in my projects with no additional config produces the same error.

*no additional config be like*:

Container = new KafkaBuilder().Build();

[ronimizy]

Container = new KafkaBuilder().WithCreateParameterModifier(x => x.Platform = "linux/amd64").Build();

produces the same output

[HofmeisterAn]

seems like it is the issue with macOS and ARM since the image being used does not have a linux/arm64 platform.

Hm, I'm not sure about the current status, but do you have Rosetta installed? I think in previous versions of Docker Desktop, there was an option to enable x86_64/amd64 support. What happens if you run:

docker pull --platform=linux/amd64 confluentinc/cp-kafka:6.1.9

before?

Container = new KafkaBuilder().WithCreateParameterModifier(x => x.Platform = "linux/amd64").Build();
produces the same output

Yes, that's a clever idea. But unfortunately, we're not passing that option to the pull command. Try pulling the image manually first. If that works, then:

1. We'll need to forward the option to the pull command.
2. We'll need to make sure we support version 8 of the image.

[ronimizy]

Rosetta is installed by default for mac with ARM cpus. i have an Apple Virtualisation Framework selected as VMM and Rosetta enabled for emulation.

manually pulling the image through docker cli work just fine

[ronimizy]

But unfortunately, we're not passing that option to the pull command.

Not sure about that, I mean clearly in the permalink you site there is no platform passed, but I dont think that is the code being executed when I call Container.StartAsync. I have a breakpoint at the line below (in a container client that being called there to be precise), and the exception occurs inside that call to the client. So i believe that there was no separate pulling step.

testcontainers-dotnet/src/Testcontainers/Clients/DockerContainerOperations.cs

Line 225 in 63f67dc

var createContainerResponse = await DockerClient.Containers.CreateContainerAsync(createParameters, ct)

(pulling image prior to running test container solves the problem though, but it is rather a hack then a solution)

[ronimizy]

platform value is being passed to "containers/create" command, at least it is present in the parameters on c# side, so there might be an issue with the way it serialized/passed to the API

[ronimizy]

I am not quite familiar with the internals of the library, but it seems for me that there are a couple of bug (?) present

first (maybe not that important to this exact issue, but nevertheless spotted) the Docker.DotNet library only checks if Name is present when creating the container to build a query string, while Platform value should also go in there

https://github.com/testcontainers/Docker.DotNet/blob/bcf0c7317b6eb7e28fa9dd0afdbbcddc9d5e1f33/src/Docker.DotNet/Endpoints/ContainerOperations.cs#L48-L51

it is not that relevant to the case because (according to docker API docs) when creating the container, docker looks up only the local images with matching platform (and falls back to virtualisation if image with host arch not found):

https://docs.docker.com/reference/api/engine/version/v1.50/#tag/Container/operation/ContainerCreate

When specified, the daemon checks if the requested image is present in the local image cache with the given OS and Architecture, and otherwise returns a 404 status.

And the second bug is probably that the is no proper handling of pull failures (i was incorrect about pull step not being present, looked closely and found it). because docker API returns this when pulling image :

{"status":"Pulling from confluentinc/cp-kafka","id":"6.1.9"}
{"errorDetail":{"message":"no matching manifest for linux/arm64/v8 in the manifest list entries"},"error":"no matching manifest for linux/arm64/v8 in the manifest list entries"}

code 200 though))

[HofmeisterAn]

pulling image prior to running test container solves the problem though, but it is rather a hack then a solution

Thanks for checking! OC this is just a workaround and not a proper fix, but this was the information and feedback I was looking for.

i was incorrect about pull step not being present, looked closely and found it

Neither the Docker Engine API (runtime) nor Docker.DotNet pulls images when creating a container. Testcontainers pulls the images in advance (we pull them here if they don't already exist).

Right now, we're not passing the platform information when pulling the image. That's basically the issue and something we need to improve.

My assumption was that if Rosetta (emulation) is enabled, the runtime would automatically handle it and pull the amd64 image (if arm64 is not available).

And the second bug is probably that the is no proper handling of pull failures

Ah, I see. That's unfortunate. Since the HTTP status code is 200, no exception is thrown. Docker.DotNet probably should throw one in this case.

[ronimizy]

My assumption was that if Rosetta (emulation) is enabled, the runtime would automatically handle it and pull the amd64 image (if arm64 is not available).

Sadly, even the docker CLI does not work that way, you always have to manually specify the platform if there is no native platform for host present

[ronimizy]

seems to me that fixes in Docker.DotNet are rather simple, I wouldn't mind looking into that, if you accept contributions 👀

[HofmeisterAn]

We somehow need to pass the platform information to this method:

testcontainers-dotnet/src/Testcontainers/Clients/TestcontainersClient.cs

Lines 333 to 337 in 63f67dc

	if (configuration.ImagePullPolicy(cachedImage))
	{
	await PullImageAsync(configuration.Image, ct)
	.ConfigureAwait(false);
	}

and set the platform property here:

testcontainers-dotnet/src/Testcontainers/Clients/DockerImageOperations.cs

Lines 60 to 63 in 63f67dc

	var createParameters = new ImagesCreateParameters
	{
	FromImage = image.FullName,
	};

Unfortunately, I'm not yet sure how we can implement this properly. Because of the parameter modifier callback, which gives us access to the low-level Docker Engine APIs, we don't have the platform information "available" while pulling.

seems to me that fixes in Docker.DotNet are rather simple, I wouldn't mind looking into that, if you accept contributions 👀

Yes, please. That would be great and much appreciated! Just a heads-up. I'll be on vacation until next week, so my responses might be a bit slow (be aware and choose the right repo when creating the PR, be default GH suggest the legacy and outdated repo).

For Docker.DotNet, we'll probably need to check the HTTP response somewhere around here.

[ronimizy]

Unfortunately, I'm not yet sure how we can implement this properly. Because of the parameter modifier callback, which gives us access to the low-level Docker Engine APIs, we don't have the platform information "available" while pulling.

I think that the simplest way to approach that is to move platform configuration to the ContainerBuilder API level, as it is pretty major/general option when creating a container. The only question here is, should this value (when specified) override what callbacks will set for container platform. Right now I can’t think of any use case where you would want platform value to differ between image pull and container creation meaning that overriding the value would make the solution more fool-proof, on the other hand such low level API is by definition hacky and not overriding this value would make for more flexible solution (and probably would be fail-fast for invalid configurations)

[HofmeisterAn]

I was thinking about extending the IImage interface. As you mentioned, it probably doesn't make much sense to use a different platform for pulling and creating a container. This way, we have the platform information available wherever we need it, and if it's not set, we can fall back to the default behavior.

I think it's still fine to allow overriding the value using the low-level API. Using this API is always a special situation, and developers step away from the standard approach.