Skip to content

[Bug]: Containers cannot talk to one another with Kernel Version >= 6.19 #3635

Description

@tdellmann

Testcontainers version

1.6.1

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host arch

x86

Go version

1.26.1

Docker version

Client: Docker Engine - Community
 Version:           29.3.1
 API version:       1.51 (downgraded from 1.54)
 Go version:        go1.25.8
 Git commit:        c2be9cc
 Built:             Wed Mar 25 16:17:07 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.5.2
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.25.3
  Git commit:       89c5e8f
  Built:            Wed Nov  5 14:43:24 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.2
  GitCommit:        301b2dac98f15c27117da5c8af12118a041a31d9
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client: Docker Engine - Community
 Version:    29.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 28.5.2
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.18.12-200.fc43.x86_64
 Operating System: Fedora Linux 43 (KDE Plasma Desktop Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.04GiB
 Name: fedora
 ID: cbb7df5c-26a1-4b41-8537-d2bbd6f253e8
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: tdellmann
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables+firewalld

What happened?

When running integration-tests we use the DelayedIntConnection() to give the different containers host and port for talking to each other.

We declare a container as ready when a certain log is printed. On one of our central containers this log did not get displayed because it could not reach a mongo-container.

Relevant log output

[2026-04-07 05:32:26.681 +0000] INFO: 🔑 Using mongodb with {"type":"mongodb","mongodb":{"uri":"mongodb://172.17.0.7:27017/myDB","collection":"myCol","secondCol":"secondCol"}}
    mongowrapper: "circuitbreaker"
[  MongoServerSelectionError: connect ECONNREFUSED 172.17.0.7:27017

  - topology.js:321 Topology.selectServer
    [app]/[mongodb]/lib/sdam/topology.js:321:38

  - topology.js:200 async Topology._connect
    [app]/[mongodb]/lib/sdam/topology.js:200:28

  - topology.js:152 async Topology.connect
    [app]/[mongodb]/lib/sdam/topology.js:152:13

  - mongo_client.js:233 async topologyConnect
    [app]/[mongodb]/lib/mongo_client.js:233:17

  - mongo_client.js:246 async MongoClient._connect
    [app]/[mongodb]/lib/mongo_client.js:246:13

  - mongo_client.js:171 async MongoClient.connect
    [app]/[mongodb]/lib/mongo_client.js:171:13

  - mongo_client.js:360 async MongoClient.connect
    [app]/[mongodb]/lib/mongo_client.js:360:16

  - server.js:65 async start
    file:///dist/cmd/server.js:65:24

] {
  errorLabelSet: Set(0) {},
  reason: TopologyDescription {
    type: 'Unknown',
    servers: Map(1) { '172.17.0.7:27017' => [ServerDescription] },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: null,
    maxElectionId: null,
    maxSetVersion: null,
    commonWireVersion: 0,
    logicalSessionTimeoutMinutes: null
  },
  code: undefined,
  [cause]: [  MongoNetworkError: connect ECONNREFUSED 172.17.0.7:27017

    - connect.js:285 Socket.<anonymous>
      [app]/[mongodb]/lib/cmap/connect.js:285:44

    - node:events:634 Object.onceWrapper
      node:events:634:26

    - node:events:519 Socket.emit
      node:events:519:28

    - node:domain:489 Socket.emit
      node:domain:489:12

    - destroy:170 emitErrorNT
      node:internal/streams/destroy:170:8

    - destroy:129 emitErrorCloseNT
      node:internal/streams/destroy:129:3

    - task_queues:89 process.processTicksAndRejections
      node:internal/process/task_queues:89:21

  ] {
    errorLabelSet: Set(1) { 'ResetPool' },
    beforeHandshake: false,
    [cause]: [  Error: connect ECONNREFUSED 172.17.0.7:27017

      - node:net:1637 TCPConnectWrap.afterConnect [as oncomplete]
        node:net:1637:16

      - async_hooks:130 TCPConnectWrap.callbackTrampoline
        node:internal/async_hooks:130:17

    ] {
      errno: -111,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '172.17.0.7',
      port: 27017
    }
  }
}

Additional information

We had a CI/CD pipeline that could still run our tests. This eventually lead us to the Linux Kernel version. Running our tests inside a VM let us bump the Kernel version gradually. This revealed that 6.18 worked fine.

To be able to work on our machines we pinned the kernel version of all our machines to 6.18.

Activity

  1. mdelapenya commented on Apr 7, 2026

    @mdelapenya
    Member

    Hi @tdellmann is it possible this issue belongs to the testcontainers-node repo? we do not have any DelayedIntConnection API in Go.

  2. tdellmann commented on Apr 8, 2026

    @tdellmann
    Author

    we do not have any DelayedIntConnection API in Go.

    Oh yeah. That is some QoL code we wrapped around a Ready chan struct{} with a timer.
    This resolves the Hostname and Port if the Container is actually started and deemed ready.

    I didn't check one more abstraction layer, sorry.

    is it possible this issue belongs to the testcontainers-node repo

    That container happens to be written in node.
    But our tests are written in go and use this library.

    The log above is by the first container that starts and actually checks if it can talk to other containers.

    Unfortunately the Problem still stands. The mongo container in question is up and deemed ready for over 30 seconds when the container i've pasted the logs from finally starts. The second container then tries for another 30 seconds to reach the mongo container and execute a ping.

    Maybe the problem is actually much deeper - somewhere in the kernel. But I run into a limit of my abilities at this time.

    Since I could only replicate this bug with the go implementation of testcontainers I figured here would be a good place to find people more able to find the actual cause of the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugAn issue with the library

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions