Closed
Description
openedon Jan 30, 2023
Module
Core
Testcontainers version
The one being bundeled with Quarkus 2.15.3.Final
Using the latest Testcontainers version?
Yes
Host OS
Ubuntu 20.04.5 LTS (containerized)
Host Arch
x86_64
Docker version
Client:
Context: default
Debug Mode: false
Server:
Containers: 3
Running: 1
Paused: 0
Stopped: 2
Images: 4
Server Version: 20.10.22
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: none
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d1
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 5.4.226-129.415.amzn2.x86_64
Operating System: Ubuntu 20.04.5 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.29GiB
Name: XXXXXXXXXXXXXXXXXX
ID: JP6C:YJBL:JFY5:YDGK:HAWT:Z6KA:IKSL:JKXP:4ZVG:JUB7:3TSG:GJHL
Docker Root Dir: /home/runner/.local/share/docker
Debug Mode: false
HTTP Proxy: XXXXXXXXXXXXXXXXXXXXX
HTTPS Proxy: XXXXXXXXXXXXXXXXXXXXXXXX
No Proxy: XXXXXXXXXXXXXXXXXXXX
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.What happened?
Starting point
I'm trying to get testcontainers working with Quarkus and docker-rootless on my self-hosted Github Action runner.
The GitHub Action runner is deployed via a k8s pod running on Kubernetes 1.22 & [Docker Engine ](docker://20.10.17). Within that container I've also installed docker-rootless (via the offical documentation), so no sidecar is being used (see Docker version for more details). Furthermore I'm using Quarkus to trigger my application tests which uses testcontainers to spin up required infrastructure services.
Problem description
The problem is that I'm unable to solve connection problems happening for containers managed by testcontainers. As you can see in the logs, the Quarkus test framework can't even communicate with Ryuk:
(testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
The same is true for any other service managed by testcontainers like redis (please see relevant log output).
My guess is it has something to do with my docker-rootless setup but I've no clue what could be the reason. When I start containers manually (e.g. via docker run ..) everything works as expected. So basically the docker-rootless installation works. I've also switched the same setup to an on-prem runner which is not running on Kubernetes but on a VM and uses docker-rootless as well. This setup works fine even when testcontainers comes into play, but obviously it's less complex (less layers of abstraction).
I've also figured out that it's by intention to communicate with the gateway instead of the containers ip directly. So I've tested whetherRyuk can ping 172.17.0.1. That' works. I've also spawned some new containers (busybox) within the same default network bridge and pinged Ryuk, also this works.
I've also tried to set TESTCONTAINERS_HOST_OVERRIDE=localhost as suggested in #4596. Without luck ... Anyway, in my case the containers are running on a container runtime within a single container( withn in a pod). IMHO the fact that containers share the same network within apod is not relevant for my nested setup, since Kubernetes (or in that case the outer docker runtime) is not aware of my nested containers.
Long story short: I'm a little bit frustrated and I don't know how to continue with my investigation. I just want to make sure it's not a testcontainer related issue. Please let me know if you need further logs or something else.
Any help would be very appreciated.
Relevant log output
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running XXXXXXTest
2023-01-30 15:57:17,366 WARN [org.tes.uti.TestcontainersConfiguration] (build-37) Attempted to read Testcontainers configuration file at file:/home/runner/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /home/runner/.testcontainers.properties (No such file or directory)
2023-01-30 15:57:17,843 INFO [org.tes.doc.DockerClientProviderStrategy] (build-37) Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///home/runner/.docker/run/docker.sock
2023-01-30 15:57:17,870 INFO [org.tes.uti.ImageNameSubstitutor] (build-37) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2023-01-30 15:57:17,898 INFO [doc.3.4]] (build-37) Pulling docker image: testcontainers/ryuk:0.3.4. Please be patient; this may take some time but only needs to be done once.
2023-01-30 15:57:19,463 INFO [doc.3.4]] (docker-java-stream-257369379) Starting to pull image
2023-01-30 15:57:19,490 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
2023-01-30 15:57:20,140 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 2 pending, 1 downloaded, 0 extracted, (317 KB/? MB)
2023-01-30 15:57:20,154 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 1 pending, 2 downloaded, 0 extracted, (317 KB/? MB)
2023-01-30 15:57:20,157 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 0 extracted, (317 KB/5 MB)
2023-01-30 15:57:20,303 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 1 extracted, (2 MB/5 MB)
2023-01-30 15:57:20,373 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 2 extracted, (2 MB/5 MB)
2023-01-30 15:57:20,487 INFO [doc.3.4]] (docker-java-stream-257369379) Pulling image layers: 0 pending, 3 downloaded, 3 extracted, (5 MB/5 MB)
2023-01-30 15:57:20,505 INFO [doc.3.4]] (docker-java-stream-257369379) Pull complete. 3 layers, pulled in 1s (downloaded 5 MB at 5 MB/s)
2023-01-30 15:57:20,538 INFO [doc.3.4]] (build-37) Creating container for image: testcontainers/ryuk:0.3.4
2023-01-30 15:57:20,614 INFO [doc.3.4]] (build-37) Container testcontainers/ryuk:0.3.4 is starting: 6d7021822af197cf173d859f70e77b5ea8de34121787b04035c04e162166816e
2023-01-30 15:57:20,942 INFO [doc.3.4]] (build-37) Container testcontainers/ryuk:0.3.4 started in PT3.057842824S
2023-01-30 15:57:20,954 INFO [doc.16]] (build-37) Pulling docker image: alpine:3.16. Please be patient; this may take some time but only needs to be done once.
2023-01-30 15:57:22,191 INFO [doc.16]] (docker-java-stream-1037873389) Starting to pull image
2023-01-30 15:57:22,192 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
2023-01-30 15:57:22,885 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 1 downloaded, 0 extracted, (28 KB/2 MB)
2023-01-30 15:57:22,991 INFO [doc.16]] (docker-java-stream-1037873389) Pulling image layers: 0 pending, 1 downloaded, 1 extracted, (2 MB/2 MB)
2023-01-30 15:57:28,403 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:33,663 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:38,916 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:44,172 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:49,427 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:53,406 ERROR [org.tes.uti.RyukResourceReaper] (build-37) Timed out waiting for Ryuk container to start. Ryuk's logs:
2023/01/30 15:57:20 Pinging Docker...
2023/01/30 15:57:20 Docker daemon is available!
2023/01/30 15:57:20 Starting on port 8080...
2023/01/30 15:57:20 Started!
2023-01-30 15:57:53,417 INFO [doc.io/redis:7-alpine]] (build-37) Pulling docker image: docker.io/redis:7-alpine. Please be patient; this may take some time but only needs to be done once.
2023-01-30 15:57:54,647 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Starting to pull image
2023-01-30 15:57:54,648 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 0 downloaded, 0 extracted, (0 bytes/0 bytes)
2023-01-30 15:57:54,682 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:57:55,322 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 5 pending, 1 downloaded, 0 extracted, (38 KB/? MB)
2023-01-30 15:57:55,324 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 4 pending, 2 downloaded, 0 extracted, (38 KB/? MB)
2023-01-30 15:57:55,344 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 0 extracted, (38 KB/? MB)
2023-01-30 15:57:55,469 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 1 extracted, (3 MB/? MB)
2023-01-30 15:57:55,503 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 2 extracted, (3 MB/? MB)
2023-01-30 15:57:55,606 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 3 pending, 3 downloaded, 3 extracted, (3 MB/? MB)
2023-01-30 15:57:55,665 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 2 pending, 4 downloaded, 3 extracted, (3 MB/? MB)
2023-01-30 15:57:55,671 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 1 pending, 5 downloaded, 3 extracted, (3 MB/? MB)
2023-01-30 15:57:55,831 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 3 extracted, (9 MB/11 MB)
2023-01-30 15:57:56,089 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 4 extracted, (11 MB/11 MB)
2023-01-30 15:57:56,121 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 5 extracted, (11 MB/11 MB)
2023-01-30 15:57:56,160 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pulling image layers: 0 pending, 6 downloaded, 6 extracted, (11 MB/11 MB)
2023-01-30 15:57:56,177 INFO [doc.io/redis:7-alpine]] (docker-java-stream-987104331) Pull complete. 6 layers, pulled in 1s (downloaded 11 MB at 11 MB/s)
2023-01-30 15:57:56,179 INFO [doc.io/redis:7-alpine]] (build-37) Creating container for image: docker.io/redis:7-alpine
2023-01-30 15:57:56,207 INFO [doc.io/redis:7-alpine]] (build-37) Container docker.io/redis:7-alpine is starting: e94b722a880e478b2b07f2e03ac95151438a69cae75970bbd52614de1f03fa18
2023-01-30 15:57:59,936 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:05,191 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:10,446 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:15,701 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:20,955 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:26,210 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:31,465 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:36,720 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:41,975 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:47,227 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:52,482 WARN [org.tes.uti.RyukResourceReaper] (testcontainers-ryuk) Can not connect to Ryuk at 172.17.0.1:32768: java.net.SocketTimeoutException: Connect timed out
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-30 15:58:56,481 ERROR [doc.io/redis:7-alpine]] (build-37) Could not start container: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (172.17.0.1 ports: [32769] should be listening)
at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:102)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:953)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:485)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.lambda$startContainer$1(DevServicesRedisProcessor.java:181)
at java.base/java.util.Optional.orElseGet(Optional.java:364)
at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.startContainer(DevServicesRedisProcessor.java:193)
at io.quarkus.redis.client.deployment.DevServicesRedisProcessor.startRedisContainers(DevServicesRedisProcessor.java:103)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
at io.quarkus.builder.BuildContext.run(BuildContext.java:281)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at java.base/java.lang.Thread.run(Thread.java:833)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
2023-01-30 15:58:56,505 ERROR [doc.io/redis:7-alpine]] (build-37) Log output from the failed container:
1:C 30 Jan 2023 15:57:56.528 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 30 Jan 2023 15:57:56.528 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 30 Jan 2023 15:57:56.528 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 30 Jan 2023 15:57:56.528 * monotonic clock: POSIX clock_gettime
1:M 30 Jan 2023 15:57:56.529 * Running mode=standalone, port=6379.
1:M 30 Jan 2023 15:57:56.529 # Server initialized
1:M 30 Jan 2023 15:57:56.529 * Ready to accept connectionsAdditional Information
docker inspect network bridge
[
{
"Name": "bridge",
"Id": "b244da46a16d6a4d62e550b06684a73d2da4bb95c5572ba5b394a655aa4939c9",
"Created": "2023-01-30T15:44:16.124443221Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ae24baf76f08c5261b92841949ff0cb126d00e368f65082184c1e9dce8a27e6b": {
"Name": "testcontainers-ryuk-464ab1cd-c952-4a10-b0f8-ad9b1ab1a256",
"EndpointID": "bdaeb0e566ec67659e732afc2e1ad50f0b6050894e06af8eca37aff0b07207c0",
"MacAddress": "02:42:ac:18:00:02",
"IPv4Address": "172.24.0.2/16",
"IPv6Address": ""
},
"e94b722a880e478b2b07f2e03ac95151438a69cae75970bbd52614de1f03fa18": {
"Name": "confident_joliot",
"EndpointID": "76b4a0ae3de0d14dc1c5252dae1c81edf7a11410d7100a75e6f8e73d5fd45c00",
"MacAddress": "02:42:ac:11:00:03",
"IPv4Address": "172.17.0.3/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
I haven't figured out why the gateway ip is sometimes missing at the default network config, but testcontainers determines the ip properly as shown in the logs .
Debug output of dockerd-rootless:
+ [ -w /home/runner/.docker/run ]
+ [ -d /home/runner ]
+ rootlesskit=
+ command -v docker-rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ command -v slirp4netns
+ slirp4netns --help
+ grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z ]
+ _DOCKERD_ROOTLESS_CHILD=1
+ export _DOCKERD_ROOTLESS_CHILD
+ id -u
+ [ 1000 = 0 ]
+ command -v selinuxenabled
+ exec rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/runner/bin/dockerd-rootless.sh
WARN[0000] The host root filesystem is mounted as "master:158". Setting child propagation to "rslave" is not supported.
+ [ -w /home/runner/.docker/run ]
+ [ -d /home/runner ]
+ rootlesskit=
+ command -v docker-rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ :
+ :
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z ]
+ command -v slirp4netns
+ slirp4netns --help
+ grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ [ -z 1 ]
+ [ 1 = 1 ]
+ rm -f /run/docker /run/containerd /run/xtables.lock
+ [ -n ]
+ stat -c %T -f /etc
+ [ tmpfs = tmpfs ]
+ [ -L /etc/ssl ]
+ realpath /etc/ssl
+ realpath_etc_ssl=/etc/.ro3276320264/ssl
+ rm -f /etc/ssl
+ mkdir /etc/ssl
+ mount --rbind /etc/.ro3276320264/ssl /etc/ssl
+ exec dockerd
INFO[2023-01-30T16:26:19.427007894Z] Starting up
WARN[2023-01-30T16:26:19.427052714Z] Running in rootless mode. This mode has feature limitations.
INFO[2023-01-30T16:26:19.427059194Z] Running with RootlessKit integration
INFO[2023-01-30T16:26:19.428038072Z] libcontainerd: started new containerd process pid=2067
INFO[2023-01-30T16:26:19.428073583Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.428082773Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.428108174Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.428122444Z] ClientConn switching balancer to "pick_first" module=grpc
WARN[0000] containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header
INFO[2023-01-30T16:26:19.441135860Z] starting containerd revision=78f51771157abb6c9ed224c22013cdf09962315d version=v1.6.13
INFO[2023-01-30T16:26:19.449417591Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
INFO[2023-01-30T16:26:19.449485393Z] loading plugin "io.containerd.snapshotter.v1.aufs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449619075Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"... error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449642695Z] loading plugin "io.containerd.snapshotter.v1.btrfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449876760Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"... error="path /home/runner/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.
containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449897440Z] loading plugin "io.containerd.snapshotter.v1.devmapper"... type=io.containerd.snapshotter.v1
WARN[2023-01-30T16:26:19.449916451Z] failed to load plugin io.containerd.snapshotter.v1.devmapper error="devmapper not configured"
INFO[2023-01-30T16:26:19.449930241Z] loading plugin "io.containerd.snapshotter.v1.native"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.449971081Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.450448240Z] loading plugin "io.containerd.snapshotter.v1.zfs"... type=io.containerd.snapshotter.v1
INFO[2023-01-30T16:26:19.450659404Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"... error="path /home/runner/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapsho
tter.v1
INFO[2023-01-30T16:26:19.450680494Z] loading plugin "io.containerd.metadata.v1.bolt"... type=io.containerd.metadata.v1
WARN[2023-01-30T16:26:19.450706705Z] could not use snapshotter devmapper in metadata plugin error="devmapper not configured"
INFO[2023-01-30T16:26:19.450721295Z] metadata content store policy set policy=shared
INFO[2023-01-30T16:26:19.450814497Z] loading plugin "io.containerd.differ.v1.walking"... type=io.containerd.differ.v1
INFO[2023-01-30T16:26:19.450837027Z] loading plugin "io.containerd.event.v1.exchange"... type=io.containerd.event.v1
INFO[2023-01-30T16:26:19.450858148Z] loading plugin "io.containerd.gc.v1.scheduler"... type=io.containerd.gc.v1
INFO[2023-01-30T16:26:19.450894578Z] loading plugin "io.containerd.service.v1.introspection-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450914639Z] loading plugin "io.containerd.service.v1.containers-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450933729Z] loading plugin "io.containerd.service.v1.content-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450951809Z] loading plugin "io.containerd.service.v1.diff-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.450972330Z] loading plugin "io.containerd.service.v1.images-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451275365Z] loading plugin "io.containerd.service.v1.leases-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451291665Z] loading plugin "io.containerd.service.v1.namespaces-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451326006Z] loading plugin "io.containerd.service.v1.snapshots-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451346797Z] loading plugin "io.containerd.runtime.v1.linux"... type=io.containerd.runtime.v1
INFO[2023-01-30T16:26:19.451415608Z] loading plugin "io.containerd.runtime.v2.task"... type=io.containerd.runtime.v2
INFO[2023-01-30T16:26:19.451509280Z] loading plugin "io.containerd.monitor.v1.cgroups"... type=io.containerd.monitor.v1
INFO[2023-01-30T16:26:19.451828415Z] loading plugin "io.containerd.service.v1.tasks-service"... type=io.containerd.service.v1
INFO[2023-01-30T16:26:19.451856836Z] loading plugin "io.containerd.grpc.v1.introspection"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451868376Z] loading plugin "io.containerd.internal.v1.restart"... type=io.containerd.internal.v1
INFO[2023-01-30T16:26:19.451924057Z] loading plugin "io.containerd.grpc.v1.containers"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451944477Z] loading plugin "io.containerd.grpc.v1.content"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451968248Z] loading plugin "io.containerd.grpc.v1.diff"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451985218Z] loading plugin "io.containerd.grpc.v1.events"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.451998838Z] loading plugin "io.containerd.grpc.v1.healthcheck"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452016569Z] loading plugin "io.containerd.grpc.v1.images"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452038209Z] loading plugin "io.containerd.grpc.v1.leases"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452054899Z] loading plugin "io.containerd.grpc.v1.namespaces"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.452075720Z] loading plugin "io.containerd.internal.v1.opt"... type=io.containerd.internal.v1
WARN[2023-01-30T16:26:19.453204180Z] failed to load plugin io.containerd.internal.v1.opt error="mkdir /opt/containerd: permission denied"
INFO[2023-01-30T16:26:19.453305902Z] loading plugin "io.containerd.grpc.v1.snapshots"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453328433Z] loading plugin "io.containerd.grpc.v1.tasks"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453339983Z] loading plugin "io.containerd.grpc.v1.version"... type=io.containerd.grpc.v1
INFO[2023-01-30T16:26:19.453349313Z] loading plugin "io.containerd.tracing.processor.v1.otlp"... type=io.containerd.tracing.processor.v1
INFO[2023-01-30T16:26:19.453361433Z] skip loading plugin "io.containerd.tracing.processor.v1.otlp"... error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
INFO[2023-01-30T16:26:19.453371783Z] loading plugin "io.containerd.internal.v1.tracing"... type=io.containerd.internal.v1
ERRO[2023-01-30T16:26:19.453392684Z] failed to initialize a tracing processor "otlp" error="no OpenTelemetry endpoint: skip plugin"
INFO[2023-01-30T16:26:19.453554907Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd-debug.sock
INFO[2023-01-30T16:26:19.453640908Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd.sock.ttrpc
INFO[2023-01-30T16:26:19.453710789Z] serving... address=/home/runner/.docker/run/docker/containerd/containerd.sock
INFO[2023-01-30T16:26:19.453740410Z] containerd successfully booted in 0.013205s
INFO[2023-01-30T16:26:19.460450613Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.460473513Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.460500384Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.460513474Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2023-01-30T16:26:19.461329809Z] parsed scheme: "unix" module=grpc
INFO[2023-01-30T16:26:19.461351739Z] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2023-01-30T16:26:19.461366499Z] ccResolverWrapper: sending update to cc: {[{unix:///home/runner/.docker/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>} module=grpc
INFO[2023-01-30T16:26:19.461377109Z] ClientConn switching balancer to "pick_first" module=grpc
INFO[2023-01-30T16:26:19.462047952Z] [graphdriver] using prior storage driver: fuse-overlayfs
WARN[2023-01-30T16:26:19.479733634Z] Your kernel does not support cgroup blkio weight
WARN[2023-01-30T16:26:19.479752304Z] Your kernel does not support cgroup blkio weight_device
INFO[2023-01-30T16:26:19.479885997Z] Loading containers: start.
INFO[2023-01-30T16:26:19.492071069Z] skipping firewalld management for rootless mode
INFO[2023-01-30T16:26:19.546072302Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2023-01-30T16:26:19.576222692Z] Loading containers: done.
INFO[2023-01-30T16:26:19.579787597Z] Docker daemon commit=42c8b31 graphdriver(s)=fuse-overlayfs version=20.10.22
INFO[2023-01-30T16:26:19.579846848Z] Daemon has completed initialization
INFO[2023-01-30T16:26:19.595244968Z] API listen on /home/runner/.docker/run/docker.sock
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment