nats-streaming container won't start as Testcontainer

Hi!

I am using testcontainers for Java in version 1.11.3.

When I use the redis container like this

    @ClassRule
    public static GenericContainer redis =
            new GenericContainer("redis:3.0.2")
                    .withExposedPorts(6379);

it starts without problems.

However when I start a nats-streaming container like this

    @ClassRule
    public static GenericContainer natsStreaming =
            new GenericContainer("nats-streaming:latest")
                    .withExposedPorts(4222);

it fails with this error message:

Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (localhost ports: [32791] should be listening)
	at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:47)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
	at org.testcontainers.containers.wait.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:23)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:582)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:259)
	... 21 more

Starting the nats-streaming container with @Rule results in the same error.

What could I be doing wrong?

And this is the log output from the container:

22:40:20.216 [main] ERROR 🐳 [nats-streaming:latest] - Log output from the failed container:
[1] 2019/08/12 20:39:20.142913 [INF] STREAM: Starting nats-streaming-server[test-cluster] version 0.15.1

[1] 2019/08/12 20:39:20.142985 [INF] STREAM: ServerID: JEamMxWK6kMQe3xmPfHYkx

[1] 2019/08/12 20:39:20.142989 [INF] STREAM: Go version: go1.11.10

[1] 2019/08/12 20:39:20.142991 [INF] STREAM: Git commit: [8e6aa7c]

[1] 2019/08/12 20:39:20.143847 [INF] Starting nats-server version 2.0.0

[1] 2019/08/12 20:39:20.143860 [INF] Git commit [not set]

[1] 2019/08/12 20:39:20.144029 [INF] Starting http monitor on 0.0.0.0:8222

[1] 2019/08/12 20:39:20.144102 [INF] Listening for client connections on 0.0.0.0:4222

[1] 2019/08/12 20:39:20.144108 [INF] Server id is ND5TD2F4YVJYL23KNOR5GSXVGF74OFZCQHIYNXULVBLYLCBVDXONKJPE

[1] 2019/08/12 20:39:20.144109 [INF] Server is ready

[1] 2019/08/12 20:39:20.173231 [INF] STREAM: Recovering the state...

[1] 2019/08/12 20:39:20.173320 [INF] STREAM: No recovered state

[1] 2019/08/12 20:39:20.424165 [INF] STREAM: Message store is MEMORY

[1] 2019/08/12 20:39:20.424199 [INF] STREAM: ---------- Store Limits ----------

[1] 2019/08/12 20:39:20.424202 [INF] STREAM: Channels:                  100 *

[1] 2019/08/12 20:39:20.424205 [INF] STREAM: --------- Channels Limits --------

[1] 2019/08/12 20:39:20.424207 [INF] STREAM:   Subscriptions:          1000 *

[1] 2019/08/12 20:39:20.424209 [INF] STREAM:   Messages     :       1000000 *

[1] 2019/08/12 20:39:20.424211 [INF] STREAM:   Bytes        :     976.56 MB *

[1] 2019/08/12 20:39:20.424213 [INF] STREAM:   Age          :     unlimited *

[1] 2019/08/12 20:39:20.424215 [INF] STREAM:   Inactivity   :     unlimited *

[1] 2019/08/12 20:39:20.424217 [INF] STREAM: ----------------------------------

[rnorth]

Ah, this is interesting - the first time I've seen this occur. I can reproduce.

The reason this is failing is that Testcontainers relies on being able to execute commands inside the container in order to detect when the process starts listening.

It seems that the nats-streaming container image is extremely slimmed down and does not contain any of the programs that Testcontainers typically uses to perform this detection.

With the image slimmed down as it is, there's just going to be no way to do this in-container listening socket detection.

Fortunately, you can change the wait strategy to instead wait for the successful log output, as follows:

@ClassRule
public static GenericContainer natsStreaming =
    new GenericContainer("nats-streaming:latest")
        .withExposedPorts(4222)
        .waitingFor(new LogMessageWaitStrategy().withRegEx(".*Server is ready.*"));

This seems to work for me, and hopefully should also for you.

(You might be wondering why we execute commands inside the container at all to detect listening sockets - why not just check the listening state from the JVM process on the outside? The reason is that Docker for Mac/Windows has a race behaviour between its Mac-side network code and the container itself, meaning that it's possible for the container to look like it's listening from the outside before the process within the container is actually listening. We work around this by checking for the listening socket from both inside and outside the container).

Does this help?

Ah, this is interesting - the first time I've seen this occur. I can reproduce.

The reason this is failing is that Testcontainers relies on being able to execute commands inside the container in order to detect when the process starts listening.

It seems that the nats-streaming container image is extremely slimmed down and does not contain any of the programs that Testcontainers typically uses to perform this detection.

With the image slimmed down as it is, there's just going to be no way to do this in-container listening socket detection.

Fortunately, you can change the wait strategy to instead wait for the successful log output, as follows:

@ClassRule
public static GenericContainer natsStreaming =
    new GenericContainer("nats-streaming:latest")
        .withExposedPorts(4222)
        .waitingFor(new LogMessageWaitStrategy().withRegEx(".*Server is ready.*"));

This seems to work for me, and hopefully should also for you.

(You might be wondering why we execute commands inside the container at all to detect listening sockets - why not just check the listening state from the JVM process on the outside? The reason is that Docker for Mac/Windows has a race behaviour between its Mac-side network code and the container itself, meaning that it's possible for the container to look like it's listening from the outside before the process within the container is actually listening. We work around this by checking for the listening socket from both inside and outside the container).

Does this help?

Hi, @rnorth.

Thank you for your quick reply. Your solution works fine for me.

I have maybe found an even better way:

    @Rule
    public GenericContainer natsStreaming =
            new GenericContainer("nats-streaming:latest")
                    .withExposedPorts(4222, 8222)
                    .waitingFor(Wait.forHttp("/streaming").forStatusCode(200));

Port 8222 exposes NATS Streaming Server's monitoring web interface. I can then wait for the HTTP status code 200 to be returned.

Thank you and best wishes
Carlo

[rnorth]

Even better! That sounds great - I'm happy that you were able to find a solution that works for you.

Ah, this is interesting - the first time I've seen this occur. I can reproduce.

The reason this is failing is that Testcontainers relies on being able to execute commands inside the container in order to detect when the process starts listening.

It seems that the nats-streaming container image is extremely slimmed down and does not contain any of the programs that Testcontainers typically uses to perform this detection.

With the image slimmed down as it is, there's just going to be no way to do this in-container listening socket detection.

Fortunately, you can change the wait strategy to instead wait for the successful log output, as follows:

@ClassRule
public static GenericContainer natsStreaming =
    new GenericContainer("nats-streaming:latest")
        .withExposedPorts(4222)
        .waitingFor(new LogMessageWaitStrategy().withRegEx(".*Server is ready.*"));

This seems to work for me, and hopefully should also for you.

(You might be wondering why we execute commands inside the container at all to detect listening sockets - why not just check the listening state from the JVM process on the outside? The reason is that Docker for Mac/Windows has a race behaviour between its Mac-side network code and the container itself, meaning that it's possible for the container to look like it's listening from the outside before the process within the container is actually listening. We work around this by checking for the listening socket from both inside and outside the container).

Does this help?

This solution works more reliably than mine.